Experimental & Quasi-Experimental Research Lecture Notes PDF

Experimental & Quasi-Experimental Research Chapter 18 Dr. Andrew S. Perrotta University of Windsor Faculty of Human Kinetics | Department of Kinesiology ASP|2023 Learning Objectives I. How t...

Experimental & Quasi-Experimental Research Chapter 18 Dr. Andrew S. Perrotta University of Windsor Faculty of Human Kinetics | Department of Kinesiology ASP|2023 Learning Objectives I. How to select a sample for research II. Controlling for internal & external validity ASP|2023 Ways to Select a Sample ASP|2023 Ways to Select a Sample The sample is the group of participants, treatments, and situations on which the study is conducted. A key issue is how these samples are selected. The ideal ways to select a sample are through random sampling and stratified random sampling. Using these methods our confidence that the sample represents the population is high. Unfortunately, random sampling can be difficult, if not impossible, to use because of the challenges in getting contact information and access to an entire population. ASP|2023 Random Sampling The purpose of random sampling is to select a sample so that it represents the larger population; that is important so that the findings from the sample can be inferred back to the population. The use of random sampling helps to ensure that the sample is representative of the population. Population The entire group that is of interest relative to the research question(s) and from which a sample is taken. ASP|2023 Ex. Protein supplementation immediately after competition maintains fat free mass in female hockey players throughout the season when not performing resistance training. This observation would be expected in other women’s U-Sport hockey programs. ASP|2023 Ex. Random Sampling A.S Perrotta, unpublished data, 2017. ASP|2023 Stratified Random Sampling Stratified random sampling is a method of stratifying a population on some characteristic before random selection of the sample. Stratified Definition = Formed, deposited or arranged in stable layers. Ex. Developing normative data on a physical fitness test for grades 4 through 8 in a school district. Because you would expect performance to be related to age, you should stratify the population by age before randomly selecting the sample on which to collect normative data. ASP|2023 Voluntary Sample Researchers often rely on volunteers to participate in their studies and are enlisted through various recruitment methods I. Flyer postings II. Local media & social media advertisements III. Mailed invitations IV. Face-to-face recruitment One challenge of recruiting is that volunteers are likely to be more interested in the study than people from a random sample. As a result, the researchers must acknowledge the limitation that the results from this sample may not generalize to the population. ASP|2023 Voluntary Sample ASP|2023 Convenience Sampling Convenience sampling involves researchers being interested in specific types of people who would not be present in large numbers if recruited from the general population. Canadian national team sport programs typically carry 18-25 athletes. *They do not represent the typical population & the typical population does not represent them* ASP|2023 Snowball Sampling Snowball sampling is when researchers are recruiting from even more select groups, for example, specific chronic diseases, they may start with convenience sampling but then ask their current participants to recruit other people they know who would be expected to be eligible for the study. Ex. Researchers are examining the effects of chemotherapy on cardiovascular function in breast cancer patients; and they ask their participants to recruit other interested patients using word of mouth (i.e. during cancer survival meetings, treatment periods, etc) ASP|2023 Ways to Assign Participants to Groups ASP|2023 Random Assignment All true experimental designs REQUIRE that the groups within the sample be randomly assigned or randomized. Random assignment has NOTHING to do with selecting the sample, the procedures used for random assignment are the same. ASP|2023 Random Assignment Study Information Procedure for Random Assignment  40 participants I. Each participant in the sample is given a  4 groups of 10 participants numeric ID. If the sample has 40 participants, the numeric IDs range from 1. 1 group who knows they are taking the 01 to 40. drug II. Use a random assignment generator 2. 1 group who is unaware they are taking (Microsoft Excel) you provide the the drug numbers of participants and groups, and the program identifies which group each 3. 1 group who thinks they are taking the participant should be assigned to. drug III. This process allows you to assume that 4. 1 group who knows they are not taking the groups are equivalent at the beginning of the experiment, which is the drug one of several important features of good experimental design that is intended to establish cause and effect. ASP|2023 Random Matched Assignment Random matched assignment is to first match the participants for a characteristic that you want to make sure is represented equally across groups. Ex. To make sure each group has a similar mean age and VO2 Max. ASP|2023 Intact Groups Intact groups Although undesirable, sometimes researchers must assign participants based upon preexisting groups(i.e. CONDITIONS). Not considered as using an experimental design. Necessary if researchers are unable to randomly assign participants to groups. Ex. Comparing patients who have Parkinson’s to people without Parkinson’s, clearly this is an independent variable that cannot be randomly assigned. ASP|2023 Post Hoc Justifications Post hoc justification is also used to compare intact groups, or groups within the sample that are not randomly formed. Post hoc justification An explanation that is provided after the event has occurred. This explanation is meant to demonstrate that the sample is representative of the population. ASP|2023 Sources of Invalidity Within Research Methods Internal validity is the basic minimum without which any experiment is uninterpretable. Did in fact the experimental treatments make a difference in this specific experimental instance? External validity asks the question of generalizability. To what populations, settings, or treatment variables can this effect be generalized? ASP|2023 Internal Validity Gaining internal validity involves controlling all INDEPENDENT variables so that the researcher can eliminate all rival hypotheses as explanations for the observed response in the DEPENDENT variable. Yet in controlling and constraining the research setting to gain internal validity, the researcher places the generalization (external validity) of the findings in jeopardy. *The results are not applicable to society at large!* ASP|2023 Ex. Internal Validity 10.0% d 8.0% d 6.0% c b b 4.0% 2.0% Δ PV% 0.0% -2.0% -4.0% b b -6.0% b -8.0% d -10.0% 1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 12 13 Day Perrotta, A. S., White, M. D., Koehle, M. S., Taunton, J. E., & Warburton, D. E. (2018). Efficacy of hot yoga as a heat stress technique for enhancing plasma volume and cardiovascular performance in elite female field hockey players. The Journal of Strength & Conditioning Research, 32(10), 2878-2887. ASP|2023 Threats to Internal Validity History: Untended events that occur during the treatment that was not part of the treatment. (Ex. Sickness, improvement in fitness, break up from partner) Maturation: The processes in which participants mature during repeated testing over time (Ex. aging, fatigue, hunger) Testing: The effects (learning effect) of one test on subsequent administrations of the same test. (Ex. Wingate test) Instrumentation: Changes in instrument calibration (ie. Accuracy) during the experiment period leads to different results. (VERY COMMONE IN KINESIOLOGY RESEARCH – MUST BE TRAINED IN OPERATING EQUIPMENT) Campbell and Stanley (1963) & Rosenthal (1966) ASP|2023 Threats to Internal Validity Statistical regression: Groups are selected based on extreme scores and are not randomization. (Causes large differences in measures between repeated examination) Selection bias: Occurs when groups are formed on some basis other than random assignment. (When treatments are administered, because the groups were different to begin with, any differences found are due to initial selection biases rather than the treatments) Experimental mortality: The loss of participants from comparison groups for nonrandom reasons. (Ex. Dropout due to sickness, lack of interest) Campbell and Stanley (1963) & Rosenthal (1966) ASP|2023 Threats to Internal Validity Selection-maturation interaction: Occurs only in designs in which one group is selected because of some specific characteristic, whereas the other group lacks this characteristic. (Ex. trained ‘vs’ un-trained individuals' response to 8- week resistance training program) Expectancy: A threat to internal validity in which the researcher anticipates that certain behaviors or results will occur. (Ex. Creatine will enhance muscular performance only in participants consuming it) Campbell and Stanley (1963) & Rosenthal (1966) ASP|2023 Threats to External Validity Reactive or interactive effects of testing: The pretest may make the participant more aware of or sensitive to the upcoming treatment. As a result, the treatment is less effective without the pretest. Ex. A physical fitness test is administered to the sample first. The participants in the experimental group realize that their levels of fitness are low and be particularly motivated to follow the prescribed program closely. Campbell and Stanley (1963) ASP|2023 Threats to External Validity Interaction of selection bias and experimental treatment: When a group is selected on some characteristic, the treatment may work only on groups possessing that characteristic. (Ex. Concussion prevention techniques in youth ‘vs’ NFL athletes) Campbell and Stanley (1963) ASP|2023 Threats to External Validity Reactive effects of experimental treatment: Treatments that are effective in constrained situations (i.e. laboratories) may not be effective in less constrained settings (i.e. real world) *The difference in conducting APPLIED RESEARCH* Hawthorne effect: A phenomenon in which participants’ performances change when attention is paid to them, which is likely to reduce the ability to generalize the results. Campbell and Stanley (1963) (Brown, 1954) ASP|2023 Threats to External Validity Multiple-treatment interference: When participants receive more than one treatment, the effects of previous treatments may influence subsequent ones. Ex. We want to examine if participants can increase their 1RM barbell flat bench press in 8-weeks. Participants perform both flat and decline bench press exercises in their weekly routine. Performing one might interfere with (or enhance) learning the other and effect their post-study 1RM test. Campbell and Stanley (1963) ASP|2023 Controlling Threats to Internal Validity 1. Randomization: controls for… I. History up to the point of the experiment; that is, the researcher can assume that past events are equally distributed among groups. II. Maturation because the passage of time is equivalent in all groups. III. Statistical regression, selection biases, and selection–maturation interaction are controlled because they occur only when groups are not randomly formed. ASP|2023 Controlling Threats to Internal Validity 2. Placebo: A control group receives a false treatment while the experimental group receives the real treatment. Ex. plain olive oil ‘vs’ olive oil mixed with CBD “vs ” ASP|2023 Controlling Threats to Internal Validity 3. Blind setup: A method of controlling a threat to internal validity in which participants do not know whether they are receiving the experimental or control treatment. Ex. When the taste is identical, both groups are blinded to the treatment! “vs ” ASP|2023 Controlling Threats to Internal Validity 4. Double-blind setup: A method of controlling a threat to internal validity in which neither the participant nor the experimenter knows which treatment the participant is receiving. ASP|2023 Controlling Threats to Internal Validity *** Randominzed double blind control studies are the “Gold Standard” in intervention-based research *** “The major advantage DBRCT is the ability to demonstrate causality (i.e. cause-effect relationship). When DBRCT is compared with other research designs, the level of evidence given by DBRCT is nearly 100% and hence it is considered “gold standards” for comparison.” Misra S. (2012). Randomized double blind placebo control studies, the "Gold Standard" in intervention-based studies. Indian journal of sexually transmitted diseases and AIDS, 33(2), 131–134. ASP|2023