LECTURE 13: ASSESSING RESEARCH II – EXTERNAL, STATISTICAL CONCLUSION AND MEASUREMENT VALIDITY & CONCLUDING RESEARCH

EXTERNAL VALIDITY

EXTERNAL VALIDITY

-     External validity is the degree of generalisability of findings from a piece of research to other situations, events and settings i.e. the degree of generalisability or representativeness

-     It is important because external validity determines the degree to which results can be applied to other contexts and populations.

POPULATION VALIDITY

-     Refers to the characteristics of the sample, as compared to the characteristics of the population from which it is drawn

-     Problematic if the sample is specialized so that conclusions can only be made to a limited population

-     E.g. – students to employees, mice to humans etc…

ECOLOGICAL VALIDITY 

-     The degree to which it is appropriate to generalise from one context to another e.g. geographically

-     E.g. – South Africa to America, lab experiment to naturalistic environment etc…

THREATS TO EXTERNAL VALIDITY

      THREATS TO POPULATION VALIDITY

-     lack of adequate definition of a target population

-     bias in sampling

-     self-selection and volunteer bias

-     non-representative sub-populations or sub-populations reflecting certain characteristics of the population but not others

-     generalisation of results from clinical studies or case studies

-     generalisation of results from animal to human, or across species

      THREATS TO ECOLOGICAL VALIDITY

-     generalisation across geographic areas

-     generalisation from laboratory to field settings

-     generalisation from unique contexts

-     generalisation across experiments or treatments

-     treatment by setting interactions

Threats to external validity are often countered by REPLICATION and/or

      TRIANGULATION

REPLICATION: duplicating findings from a particular study across different contexts and/or sample groups (Leedy & Ormrod, 2005)

TRIANGULATION: using multiple sources of data, methods of data collection, types of analyses or researchers to establish convergences in findings (Leedy & Ormrod, 2005)

  

STATISTICAL CONCLUSION VALIDITY

This involves assessing the use of both descriptive and inferential statistics and fits into the quantitative research process at the point of the analysis of data collected. Assessing statistical conclusion validity is heavily dependent on understanding when it is appropriate to use particular types of statistical analyses, and what decision need to be considered – these issues are addressed in the STATISTICS component of the course.

STATISTICAL CONCLUSION VALIDITY

-    Statistical conclusion validity is about ensuring that the statistics are appropriate for the design used

-    Having strong statistical conclusion validity means that the correct statistical procedure has been chosen to analyse the data, and that the assumptions of the statistical procedures chosen match those applying to the study (All statistical procedures are based on assumptions about the mathematical properties of the numbers being used, and if these are violated both the statistics and the results of the study will be invalid)    

MEASUREMENT VALIDITY

This involves assessing whether the conceptualization and operationalisation (measurement or manipulation) of the variables was appropriate and/or successful within the research. This is heavily dependent on understanding principles of psychological measurement – these issues are addressed in the PSYCHOMETRICS component of the course.

NB:

VALIDITY is important because it is important to make knowledge claims from research that is appropriate and not excessive. Only in designs relatively free from internal, external, statistical conclusion and measurement validity threats, is it possible to make firm and sound knowledge claims. 

ASSESSING QUALITATIVE RESEARCH

With quantitative research we utilise internal, external, statistical conclusion and measurement validity as evaluative tools to assess the rigour of a research project. These tools are not applicable to qualitative research. Separate criteria are used to assess the rigour and utility of qualitative research, for example, those identified by Guba & Lincoln (1983):

·       Credibility: research needs to demonstrate that it was conducted in such a manner so as to ensure that the phenomena were accurately identified and described

·       Transferability: demonstrating the applicability of one set of findings to another context

·       Dependability: the researcher attempts to account for changing conditions to the phenomenon chosen for research as well as changes in the design created by an increasingly refined understanding of the setting

·       Confirmability: is focused on whether the results of the research could be confirmed and places the evaluation on the data themselves

DATA COLLECTION AND ANALYSIS

CONCLUSIONS

Statistical analysis allows one to test whether significant differences exist but the mathematical conclusions obtained from statistical testing need to be translated into ‘English’. In other words the statistical results need to be framed and interpreted within the context of the research study and the relevant correlational and/or causal conclusions need to be drawn. The results must be ‘translated’ in relation to the hypothesis specified.

·       Causal hypotheses should yield causal conclusions (need to assess to what extent the criteria for causality have been met)  

·       Correlational hypotheses should yield correlational conclusions (NB: correlational hypotheses do not usually enable one to make causal conclusions - correlation does not imply causation).   

KNOWLEDGE CLAIMS

-      Knowledge claims involve situating the findings of the study within the broader area or field in which the study is located i.e. situating the findings within available literature and what is already known.

-      Knowledge claims are strongly dependent on the validity of the study