Research Methods I Week 9 Reliability and Validity PDF

**Research Methods I Week 9** **Reliability and Validity** Confounding variables - Uncontrolled 'extraneous' variables that potentially affect the outcome of the study - Major goal of research design is to decrease (or to control) the influence of confounding/ extraneous variables R...

**Research Methods I Week 9** **Reliability and Validity** Confounding variables - Uncontrolled 'extraneous' variables that potentially affect the outcome of the study - Major goal of research design is to decrease (or to control) the influence of confounding/ extraneous variables Reliability - The extent to which your measurement produces consistent and accurate results - An issue for questionnaires or observational designs - Types of reliability: - **Inter-rater reliability**: measures the degree of agreement between different people observing or assessing the same thing - **Test-retest reliability**: measured the consistency of results when you repeat the same test on the same sample at a different point in time - **Internal consistency**: assesses the correlation between multiple items in a test that are intended to measure the same construct Validity - The degree to which a test measures what it intends to measure Construct validity - Is the test measuring what it is supposed to? - 3 different elements: - **Content validity**: does the test measure all aspects of the variable? - **Convergent validity**: does the test give similar results to other measures of the same thing? - **Discriminant validity**: is the test unrelated to measures of other unrelated things? Internal validity - The extent to which the results of an experiment can be attributed to the manipulation of the independent variable (IV) -\> (and not to some uncontrolled variable) - Confounding variables **reduce** internal validity Confounds in between-subjects designs - Different people undergo different conditions of the IV (independent groups) - Problem: non-equivalent groups - E.g. participants choosing particular groups (treatments) based on preference, or other factors - Solution: randomly assign participants to groups - Allows each participant the same chance as all the others of being in a group - Problem with using quasi-independent variables (e.g. age, gender, IQ, ethnicity, handedness, religion) -\> participants may differ in other ways across groups - E.g. compare mood of people who exercise regularly vs people who never exercise -\> groups may also differ in terms of diet, employment, personality traits etc - Solution: **Matching groups** - aim is to match participants between groups on all other relevant variables except the one of interest - often difficult (sometimes impossible): - may not be able to find enough matched participants - may not know what all the relevant variables are Confounds in within-subjects design - each participant takes part in each condition of the IV (repeated measures) - often considered **optimal** because each person is their own control (control for individual differences) - Problem**: order effects** (Carryover effects) - when the order of the tasks affects the DV - practice, fatigue - Solution: **Counterbalancing** - Systematically ordering the conditions for participants - Problem: **maturation effects** - some DVs will change with the passage of time - e.g. study into a new antidepressant: - But mood may have changed overtime anyway, without the antidepressants - Solution: include a **control group** - e.g. give a placebo to some participants Other confounds - **demand characteristics** - when a participant changes their behaviour according to what they think are the expectations of the researchers - e.g. social desirability - solution: deception (ethics?) - unobtrusive manipulation of IV - between-subjects design (so participants are not aware of all levels of IV) Confounds and internal validity - confounding variables can reduce the internal validity of a study - individual differences (between-subjects) - order effects (within-subjects) - demand characteristics (within-subjects) - confounds can be controlled by: - random assignment to groups (between subjects) - matching groups (between subjects, quasi-experimental) - Counterbalancing (within-subjects) Ecological validity - The extent to which the findings of a research study can be generalised to real-life settings External validity - The extent to which the results of a study can be generalised to other situations and to other people - Replication of results by other researchers is a good test of external validity - An important threat to external validity is sampling bias Representativeness - The sample must be representative of the population - Non-representative = biased - Should approximately match the characteristics of the population e.g. gender, age, occupation - Threat to representativeness: - Biased sample = when the characteristics of the sample are different from the population - Response bias = when an unrepresentative section of the population responds to your recruitment advert - Selection bias = when the techniques for selecting the sample lead to over- or under-representation of a proportion of the population Methods of sampling **Probability sampling** - Each member of the population has an equal chance of being selected to participate - Random sampling - Each member of a population is equally likely to be chosen to make up the sample - E.g. if everyone was assigned a number and a random number generator was used to select participants - Systematic sampling - Recruit every Nth person from the population - E.g. recruit every 5^th^ person - Stratified sampling - Ensures that subgroups are represented equally - E.g. 70% female students and 30% male students **Nonprobability sampling** - Each member of the population does not have an equal chance of being selected to take part - Volunteer sampling - The researcher advertises their study and people volunteer to participate - E.g. SONA - Could be biased, as participants are self-selected - Opportunity sampling - Researcher approaches people to participate in their study - E.g. going to a meeting of the Alzheimer's society to recruit people with dementia - Could also be biased - Convenience sampling - People are sampled simply because they are "convenient" sources of data - E.g. collecting data from a research methods class - Could also be biased Sampling in the real world - **Ethical guidelines** state that participants must voluntarily agree to take part - Are volunteers representative of general population? - "volunteerism" may affect external validity - Research has shown that volunteers do not have the same characteristics as the general population (e.g. Rosenthal and Rosnow, 1975) - Very difficult to avoid volunteer bias Sampling and external validity - Sample needs to be representative of the population - A biased sample will reduce the external validity of the study - Random sampling is ideal, but not usually possible - Problems with non-random sampling include volunteer bias