Research Validity PDF

EDWARDS—RESEARCH METHODS LECTURE NOTES PAGE 15 Topic #3 RESEARCH VALIDITY Define validity as—the APPROPRIATENESS of INFERENCES drawn from DATA A key criterion in evaluating...

EDWARDS—RESEARCH METHODS LECTURE NOTES PAGE 15 Topic #3 RESEARCH VALIDITY Define validity as—the APPROPRIATENESS of INFERENCES drawn from DATA A key criterion in evaluating any test, measure, or piece of research is validity. 1. data/observations—not impressions or opinions. 2. drawing inferences 3. appropriateness—implies purposes for which inferences are drawn e.g., polygraph—lying behavior or symptoms of anxiety?? The concept of validity is used in two different ways: 1. Test and measurement validity A. criterion–related B. content–related C. construct–related 2. Research or experimental validity A. internal B. external C. statistical conclusion D. construct EDWARDS—RESEARCH METHODS LECTURE NOTES PAGE 16 Research Validity—A conclusion based on research is valid when it corresponds to the actual state of the world. Four types of research validity are commonly recognized—internal validity, external validity, statistical conclusion validity, and construct validity. 1. Internal Validity—is the extent to which we can infer that a relationship between two variables is causal or that absence of a relationship implies absence of cause. That is, is the system of research internally consistent? Do the relationships obtained follow from the research design? A study has internal validity if a cause–effect relationship actually exists between the independent and dependent variables. The difficulty is determining whether the observed effect is caused only by the IV, since the DV could have been influenced by variables other than the IV. A. Extraneous Variable—any variable other than the IV that influences the DV B. Confounding—when an extraneous variable systematically varies with variations or levels of the IV Internal validity is driven by the quality of the research design, which is defined by control (of confounding variables). Threats A. History (events outside the lab)—the observed effects between the independent and dependent variable might be due to an event which takes place between the pretest and posttest when this event is not the treatment of research interest (e.g., effects of success/failure [IV] on job satisfaction [DV] with success condition run on a sunny day and failure on a gloomy, dark, cold, rainy day [weather=EV]). B. Maturation—a source of error in an experiment related to the amount of time between measurements; concerned with naturally occurring changes (e.g., in employee development research employees tend to get better with more experience - independent of any specific employee development program). C. Testing—effects due to the number of times particular responses are measured—familiarity with the measuring instrument (e.g., increased scores on 2nd test). D. Mortality/Attrition—the dropping out of some participants before a study is completed, causing a threat to validity (e.g., effect of empowerment [IV] on performance [DV]; effect of attrition as a result of feelings of low empowerment would be an EV because only those that felt empowered would be left at the end of the study). EDWARDS—RESEARCH METHODS LECTURE NOTES PAGE 17 E. Selection—many studies compare two or more groups on some dependent variable after the introduction of an IV. "Simpler" studies like surveys just assess attitudes or opinions on an issue. In either case, sampling or selection into the study is critical. Samples must be comparable — in multi–group designs — and/or must represent the population. (e.g., survey of attitudes towards endangered species — one would probably obtain very different results as a function of sampling from individuals in the logging industry vs. members of the Society for the Protection of Baby Seals). F. Regression effects—tendency of participants with extreme scores on first measure to score closer to the mean on a second testing; a statistical threat (e.g., the highest performing firms in a given year tend to do worse in subsequent years). These threats are corrected for by randomization 2. External Validity—is the inference that presumed causal relationships can be generalized to and across alternate measures of cause and effect, and across different types, persons, settings, and times. That is, how generalizable are findings? The concern is whether the results of the research can be generalized to another situation—specifically, participants, settings, and times. Threats A. Other participants (interaction of selection and treatment)—population validity Behavioral and organizational research often uses convenience samples—"the experimentally accessible population"—question is: How representative is the typical sample of the focal group of interest?; participants are often chosen by availability. B. Other settings (interaction of setting and treatment)—ecological validity C. Other times (interaction of history and treatment)—temporal validity External validity may be increased by random sampling for representativeness Importance of trade–off issues between internal and external validity. EDWARDS—RESEARCH METHODS LECTURE NOTES PAGE 18 3. Statistical Conclusion Validity—appropriateness of inferences (or conclusions) made from data as a result (or function) of conclusions drawn from statistical analysis. That is, are the IV and DV statistically related? Threats A. Low statistical power—this is the ability (power) of a statistical test to identify relationships when they are actually present. The larger the sample size, the greater the power. B. Violated assumptions of statistical tests C. Reliability of measures These threats can be addressed by having adequate power, meeting the assumptions of tests, and using reliable measures. 4. Construct Validity—has to do with labels that can be placed on what is being observed and the extent to which said labels are theoretically relevant. Construct validity is a question of whether the research results support the theory underlying the research. That is, is there another theory that could adequately explain the same results? e.g., is emotional intelligence a better label than communication skills for what is being studied? If the labels being used are irrelevant to the theory being researched, then the study can be said to lack construct validity. Threats A. Loose connection between theory and experiment. B. Evaluation apprehension—tendency of research participants to alter their behavior because they are being studied (e.g., Hawthorne Effect). C. Experimenter expectancies ("good–subject response")—tendency of research participants to act according to what they think the researcher wants. EDWARDS—RESEARCH METHODS LECTURE NOTES PAGE 19 "Good–subject response" and to some extent evaluation apprehension can be controlled/minimized by using the following procedures A. Double–blind procedures B. Single–blind procedures C. Deception Need to realize that the four types of research validity are interrelated and NOT independent of one another. For example: statistical conclusion validity is necessary for demonstration of other types of validity. internal validity must be achieved for construct validity to be obtained. and external validity depends in part upon the demonstration of at least statistical conclusion validity and internal validity.