Common threats to internal validity

INSTRUMENTATION

-     Instruments used in research must be reliable, valid and suitable to answering the question.

-     Invalid instruments cannot provide valid measurements of the variable of interest and instruments do not always have the same validity in different contexts or over time.

-     Instruments may also interfere with what they are attempting to measure i.e. react with the phenomenon they are attempting to measure (instrument reactivity) e.g. A subject becoming upset due to traumatic memories while filling out a questionnaire on PTSD

-     use valid, reliable, fair instruments                             

HISTORY

-     Refers to environmental events that are unrelated to the variables of the study which occur during the research and may influence the results

-     E.g. – class attendance and the weather; political situation and media exposure; fuel consumption prices and traffic safety

-     random assignment, matching (personal history)

MATURATION

-     Refers to any systematic changes/ processes in an organism’s biological or psychological condition over time which are not related to or produced by the study i.e. any physical, psychological or emotional changes occurring in subjects over time

-     Particularly problematic in studies involving children or longitudinal studies.

-     E.g. – a two-year case study of a college graduate entering a new job

-     random assignment, shorter duration

TESTING  

-     Refers to the possible effects of having already taken a test on an individual’s score when s/he takes the test a second time i.e. gains in performance on test-retest designs may be at least partially due to testing effects, as opposed to the IV.

-     Problems of familiarity with format, administration and content

-     E.g. – taking the same final year exam twice

-     no pretest, alternate forms of the same test

STATISTICAL REGRESSION

-     In studies involving repeated testing under the same conditions, there is often a trend for extreme scores in a distribution to move or regress towards the mean

-     no pretest, alternate forms of the same test, random assignment (Solomon Four design allowing influence of statistical regression to be estimated)

SELECTION BIAS

-     Occurs when there is a breakdown of random assignment i.e. there is bias (a systematic difference) in the division of the EG and CG

-     NB: selection bias applies to selecting the EG and CG from the sample – it does not apply to selecting the sample itself (issues of bias in selecting the sample from the population apply to threats to population validity – please refer to Lecture 8)

-     Occurs when one group has particular characteristics that the other does not i.e. non-equivalent groups.

-     random assignment, matching

DIFFERENTIAL ATTRITION

-     Attrition refers to the loss of subjects from the study, often caused by withdrawal (refusal to continue) or life events

-     Differential attrition occurs when the numbers of subjects lost are not equal between the groups i.e. differential subject loss.

-     Problematic because it may not be random, thus introducing an extraneous variable, and possibly limiting the generalisability of results

-     quasi-controls, motivational manipulations

DIFFUSION

-     The spread of treatment effects from the EG to the CG

-     May contaminate or confound the effects of the treatment

-     Placebo control method, keep groups apart, shorter duration, deception, disguised experiment

o   COMPENSATORY EQUALISATION 

§  Untreated groups or subjects (CG) learn of the treatment received by the EG and demand the same treatment, thus confounding the effects of the treatment

§  Members of the CG may also attempt to compensate on their own, thus confounding the effects of the treatment

§  E.g. a new teaching method

o   COMPENSATORY RIVALRY (The John Henry Effect)

§  Untreated groups or subjects (CG) learn of the treatment received by the EG and thus work extra hard to compensate and exceed the EG i.e. the superiority of the EG is not demonstrated

o   RESENTFUL DEMORALISATION

§  Opposite effect of compensatory rivalry, and occurs when untreated subjects or groups (CG) learn of the treatment received by the EG and thus become less productive, efficient or motivated as a result

§  Can create appearance of benefit from the treatment, even when this is not the case

HAWTHORNE EFFECTS

-     Distortions in behaviour that occur when people are aware that they are being observed/participating in a study

-     unobtrusive studies/naturalistic observation (ethics?), adaptation period, deception

GOOD SUBJECT / HALO EFFECT / FAKING GOOD

-     Refers to the possibility that subject responses may be distorted by a desire/need to meet social expectations/ please the researcher/appear better than reality

-     deception, disguised experiment, sacrifice group, motivational manipulation, bogus pipeline strategy

FAKING BAD

-     Refers to the possibility that subject responses may be distorted by a desire to thwart the researcher/appear worse than reality

-     Opposite of halo effects/faking good

-     deception, disguised experiment, sacrifice group, motivational manipulation, bogus pipeline strategy

EXPERIMENTER EFFECTS    

2 ways in which experimenter can influence study, viz.    

o   NON – INTERACTIONAL EXPERIMENTER EFFECTS

§  Bias (systematic errors in interpretation/observation)

§  Dishonesty (intentional effect)

§  Sloppiness (intentional effect)

§  ask an objective colleague/expert, professional review panels

o   INTERACTIONAL EXPERIMENTER EFFECTS

§  Biosocial & psychosocial effects (e.g. gender)

§  Experimenter expectancy/ self-fulfilling prophecy

§  quasi-controls, blind techniques, automation