Research Validity PDF

The Research Validity Chapter 7 + pdf slides  VALIDITY in research designs The validity of the design of experimental research studies is a fundamental part of the scien7fic method and a concern of research ethics. Without a valid design, valid scien7...

The Research Validity Chapter 7 + pdf slides  VALIDITY in research designs The validity of the design of experimental research studies is a fundamental part of the scien7fic method and a concern of research ethics. Without a valid design, valid scien7fic conclusions cannot be drawn. 2 Cau$on (Red alert! This is not a drill) Validity in the context of measurement is different from that in the context of research design (e.g. internal validity, external validity) Even graduate students are confused!  Validity in measurement: whether the test can measure what it intends to measure. Frequently paired with Reliability (in test standardizaonship exists between one or more independent variables and one or more dependent variables. Is the extent to which a causal conclusion based on a study is warranted, which is determined by the degree to which a study minimizes systema>c error (or 'bias'). 7 In other words, can we be reasonably sure that the change (or lack of change) was caused by the manipulaBon of the ID? Is the experimental design well designed? Have we constructed the design in a way that controls for sources of threat? Researchers must be aware of aspects that may reduce the internal validity of a study and do whatever they can to control for these threats. These threats, if le@ ignored, can reduce validity to the point that any results are meaningless rendering the en/re study invalid. There are eight major threats to internal validity: 1. history, 5. instrumenta7on, 2. matura7on, 6. sta7s7cal regression 3. tes7ng, 7. experimental bias 4. selec7on, 8. mortality We'll see each of these threats in more detail later in the lesson 10 WHO IS GUILTY? We'll see each of these threats in more detail later in the lesson 11  The only way to try to cotrol the threats is through the formula7on of a GOOD research design a GOOD design is one that manages to control as many threats as possible we now see the most popular one in ABSOLUTE The Solomon four-group design is: *Pretest Sensi+za+on means exposure to an Experimental Design the pretest increases sensi>vity to the introduced during 1949 by experimental treatment, thus preven>ng Solomon generaliza>on of results from the pretested adequate to assess the effect of sample to an unpretested popula>on the treatment and is immune from most threats to internal …in other words is the effect of the pretest validity. with respect to a condi:on that has not been pretested the only true experimental design that assesses the presence of pretest sensiBzaBon* effects 13 The various combina7ons of tested and untested groups with treatment and control groups allows the researcher to ensure that confounding variables and extraneous factors have not influenced the results. 15 Random assignment refers to the use of chance procedures in psychology experiments to ensure that each par:cipant has the same opportunity to be assigned to any given group. Study par>cipants are randomly assigned to different groups, such as the experimental group, or treatment group We now see how Solomon's design is effec7ve in controlling many* of the threats to internal validity. history, matura7on, tes7ng, selec7on, History. History refers to any event outside of the research study that can alter or effect subjects’ performance. Since research does not occur within a vacuum, subjects oOen experience environmental events that are different from one another. These events can play a role in their performance and must therefore be addressed. One way to assure that these events do not impact the study is to control them, or make everyone’s experience idenBcal except for the independent variable(s). Since this is oOen impossible, using randomizaon. Selec%on refers to the manner in which subjects are selected to par%cipate in a study and the manner in which they are assigned to groups. If there are differences between the groups prior to the study taking place, these differences will con%nue throughout the study and may appear as a change in a sta%s%cal analysis. Addressing these differences through subject matching or randomiza%on is highly recommended. R O1 X O2 G1 R O3 O4 G2 R X O5 G3 R O6 G4 22 Back to threats to internal validity: 1. history, 5. instrumentation, ✓ 2. maturation, 6. statistical regression 3. testing, 7. experimental bias 4. selection, 8. mortality 23 Statistical Regression. Statistical regression, or regression to the mean, is a concern especially in studies with extreme scores. It refers to the tendency for subjects who score very high or very low to score more toward the mean on subsequent testing. If you get a 99% on a test, for instance, the odds that your score will be lower the second time are much greater than the odds of increasing your score. 24 Sta2s2cal Regression. Statistical Regression. For example, when children with the worst reading scores are selected to participate in a reading course, improvements at the end of the course might be due to regression toward the mean and not the course's effectiveness. If the children had been tested again before the course started, they would likely have obtained better scores anyway. Likewise, extreme outliers on individual scores are more likely to be captured in one instance of testing but will likely evolve into a more normal distribution with repeated testing. 26 Instrumentation. If the measurement device(s) used in your study changes during the course of the study, changes in scores may be related to the instrument rather than the independent variable. For instance, if your pretest and posttest are different, the change in scores may be a result of the second test being easier than the first rather than the teaching method employed. If any instrumentation changes occur, the internal validity of the main conclusion is affected, as alternative explanations are readily available. tDCS devices 27 Experimenter Bias. We engage in research in order to learn something new or to support a belief or theory. Therefore, we as researchers may be biased toward the results we want. This bias can effect our observations and possibly even result in blatant research errors that skew the study in the direction we want. Using an experimenter who is unaware of the an>cipated results (usually called a double blind study because the tester is blind to the results) works best to control for this bias. 28 Mortality. Mortality, or subject dropout, is always a concern to researchers. They can drastically affect the results when the mortality rate or mortality quality is different between groups. Imagine in the work experience study if many motivated students dropped out of one group due to illness and many low motivated students dropped out of the other group due to personal factors. The result would be a difference in motivation between the two groups at the end and could therefore invalidate the results. 29 Mortality EXAMPLE A comparison of two dieting regimes on weight loss We want to compare the effectiveness of two types of dieting regime (i.e., the independent variable), one decidedly more demanding/aggressive than the other, on weight loss (i.e., the dependent variable). Initially, participants of different levels of health status, ranging from healthy to obese are randomly assigned to the two dieting regimes (i.e., treatments A and B). Let's say that treatment A required more self-discipline because participants had to change their diet without any external help, whilst treatment B provided participants with regular check-ups (e.g., Weight Watchers) and professional counselling Mortality EXAMPLE A comparison of two dieting regimes on weight loss Experimental mortality: Therefore, whilst 96% of participants remained in the treatment B program, only 85% remained in the treatment A program. Note that this may not, in principle, become a significant threat to internal validity. However, if a much higher proportion of those participants that dropped out of treatment B were the more obese participants compared to those dropping out of treatment A (e.g., because they were less motivated, or they required more support/counselling to help them lose weight than the health individuals), the average (i.e., mean score) weight loss of the treatment B group at the end of the study could be lower than would have been expected. As a result, the difference in the scores on the dependent variable (i.e., weight loss) could not be explained solely by the application of the two different treatments (i.e., the independent variable), but also by experimental mortality. This becomes a threat to the internal validity of the results. We now turn to the other major aspect of validity  External Validity. External validity refers to the generalizability of a study. In other words, can we be reasonable sure that the results of our study consisting of a sample of the population truly represents the entire population? Threats to external validity can result in significant results within a sample group but an inability for this to be generalized to the population at large IS THERE ANY EXAMPLE FROM THE AUDIENCE? 33 34 35 36 37 Example 38 39 Example 40 Some researchers believe that a good way to increase external validity is by conducting field experiments. 41 In a field experiment, people's behavior is studied outside the laboratory, in its natural setting. A field experiment is identical in design to a laboratory experiment, except that it is conducted in a real-life setting. The participants in a field experiment are unaware that the events they experience are in fact an experiment. Some claim that the external validity of such an experiment is high because it is taking place in the real world, with real people who are more diverse than a typical university student sample. 42  The basic dilemma of the experimental psychologist When conduc>ng experiments some believe that there is always a trade-off between internal and external Interna l External validity validity—having enough control over validity the situa>on to ensure that no extraneous variables are influencing the results and to randomly assign people to condi>ons, and ensuring that the results can be generalized to everyday life 43  However, as real-world settings differ dramatically, findings in one real world setting may or may not generalize to another real world setting. Neither internal nor external validity are captured in a single experiment. 44 Some psychologists opt first for internal validity, conducting laboratory experiments in which people are randomly assigned to different conditions and all extraneous variables are controlled. Other psychologists (social) prefer external validity to control, conducting some of their research in field studies. And many do both. Taken together, both types of studies meet the requirements of the perfect experiment. Through replication, researchers can study a given research question with maximal internal and external validity.  CONSTRUCT VALIDITY : EXTENT TO WHICH THE RESULTS SUPPORT THE THEORY BEHIND THE RESEARCHJ Is related to concept of operationalization hypothesis of design  Exercise to be done at home alone or in groups: Study the characteris>cs of construct validity and prepare a PPT presenta>on with examples from cogni>ve neuroscience research DEADLINE (classroom presenta>on): Tuesday, April 4 BENEFIT: 1 point to be added to midterms scores