Experimental Research Design

Experimental Research

• Researchers manipulate one or more independent variables, administering treatments.
• Researchers measure one or more dependent variables, observing behaviors affected by the treatment and looking for change.

Research Design

• A typical research design includes a control group not receiving the experimental treatment.
• A typical research design also includes one or more experimental groups receiving the experimental treatment.
• Control groups are not always possible, but they lead to the strongest research designs.

Experimental Control

• Strong experiment controls over experimental conditions increase the degree of internal validity.
• Representative samples are similar to the population of interest.
• Inclusion and exclusion criteria define the sample.
• Randomization of equivalent groups avoids selection bias.
• During study, maximum control of variables occurs.

Confounding Variables

• Extraneous variables MAY/POTENTIALLY affect the dependent variable.
• Confounder variables DO/WILL influence the dependent and/or independent variable if not controlled for.
• Always be aware of extraneous variables.

Extraneous vs. Confounder Example

• In study, does studying for 5 hours affect HPED 2030 final exam tests in comparison to studying 1 hour per day.
• Extraneous variables that MAY affect, if not controlled for are time of day, test anxiety, student's interest in statistics, and major.
• Confounding variables that WILL affect, if not controlled for are cognitive impairment and learning disability.

Additional Extraneous vs. Confounder Example

• In study, does caffeine improve performance outcomes among soccer athletes.
• Extraneous variables that MAY affect if not controlled for are time of day, how the athlete is feeling, and skill level.
• Confounder variables that WILL affect if not controlled for are genetic predispositions relating to caffeine metabolism.

Key Experimental Design Concepts

• Control of experimental variables is critical in creating a valid research study.
• Uncontrolled experimental variables creates confound, reducing the experiment's value.
• Internal and external validity are critical to experimental design.

Internal Validity

• Internal validity measures the extent to which research conditions are controlled.
• Internal validity means changes in the dependent variable can be attributed to the independent variable.
• High internal validity means any change in outcome is a result of the intervention.
• Internal validity has no statistical value associated with it; it's assessed by examining the research design.
• Threats to internal validity compromise confidence in findings.
• Internal validity refers to the extent that differences can be attributed to independent variable.
• Research with strong internal validity characteristics contains no confounding variables.
• Research with strong internal validity characteristics provides only 1 explanation for the results.

Threats to Internal Validity - Bias

• Bias means colouring of observations.
• Bias means "See what you want to see and hear what you want to hear."

Threats to Internal Validity - Participants

• Maturation
• History
• Statistical Regression
• Selection bias
• Experimental Mortality (Attrition Bias)
• Procedure/Testing
• Independent Variable
• Dissemination/Translation
• Testing
• Instrumentation
• Design Contamination
• Compensatory Rivalry
• Confirmation Bias (Placebo Effect/Hawthorn Effect)
• Participation Bias (Volunteer Bias)

Threats to Internal Validity - Maturation

• Normal developmental processes that occur as a function of time impacts the study.
• Outcomes may vary based on natural result of time.
• Growth in classroom interventions in 1st graders is an example.

Threats to Internal Validity - History

• "Something" happens during the study that affects the dependent variable.
• An unrelated event influences the outcome.
• A study investigating morale, layoffs at a company impact results.
• To solve issue, use two or more groups.

Threats to Internal Validity - Statistical Regression Towards the Mean

• Statistical tendency for people who score extremely low or high on a test to score closer to the middle next time, regression occurs.
• Lower subjects will improve more than the average or above-average subjects.

Threats to Internal Validity - Selection Bias

• Groups are not comparable at the beginning of the study.
• Random sampling and establishing similar baseline values at pre-test helps mitigate selection bias.

Threats to Internal Validity - Experimental Mortality (Attrition Bias)

• Dropout from participants affects the study.
• Did some participants drop out more than others?

Threats to Internal Validity - Procedure/Testing

• Familiarity or learning effect can threaten internal validity.
• Did the pre-test affect the scores of the post-test?
• Athletes can 'learn' how to do the study.

Threats to Internal Validity - Instrumentation

• Accuracy and calibration errors can affect results.
• Was the dependent variable measured differently over the course of the study?

Threats to Internal Validity - Design Contamination

• Design contamination occurs when the control group finds out about the experimental group
• Design contamination may lead to John Henry Effect or Independent Variable threats.
• Design contamination can happen if one group preferentially wants the study to succeed/fail.

Threats to Internal Validity - Compensatory Rivalry

• "John Henry Effect" can occur.
• In the "John Henry Effect", a control, aware of their status and comparing their performance with a treatment group, actively works harder to overcome perceived disadvantage.
• Control group will feel 'disadvantaged', so they work harder

Threats to Internal Validity - Dissemination/Translation

• Funding bias can skew study results
• Publication bias can skew study results

Threats to Internal Validity - Independent Variable

• Diffusion occurs when participants in the control group adopt or mimic the experimental group.
• Independent Variable diffusion: Leads to underestimation of treatment effect

Threats to Internal Validity - Confirmation Bias

• Confirmation bias is people's tendency to process information by looking for or interpreting information that is consistent with their beliefs.
• In research, confirmation bias means a researcher stops gathering information when retrieved evidence confirms researcher viewpoints.

Threats to Internal Validity - Halo Effect

• Halo effect is a cognitive bias where our overall impression of a person or company influences how we feel.

Threats to Internal Validity - Placebo Effect

• Placebo effect is when a beneficial effect is seen when the subject thinks they received the treatment

Threats to Internal Validity - Hawthorne Effect

• Hawthorne effect, or observer effect, is where participants modify behaviour as a result of knowing they are being observed.

External Validity

• External validity means the ability to apply the results of a study in other conditions or settings.

External Validity Example

• The results from an aerobic training study with only males participants cannot be applied to aerobic training in females.

Generalizability

• Generalizability means ability to extend results to other situations.
• The greater the generalizability, the greater the degree of external validity.
• There is no statistical tool to determine generalizability; it depends on readers judgment.

Threats to External Validity - Participants

• Subject and treatment interaction
• Artificial nature of the experimental condition (Lab vs Real World)
• Field
• Lab

Threats to External Validity - Subject and Treatment Interaction

• Characteristic of a subject affects how they respond to treatment
• If sample properly represents the population.
• Solution: Subjects must be representative of the population to be generalized to.

Threats to External Validity - Artificial Nature of the Experimental Condition

• Does the experimental situation reflect reality?
• Generalizing is difficult given the tight controls.
• "In vitro" drug interventions may not translate to human subjects.
• Solution: Minimize artificial and constrained research setting

Lab vs Field Research

• Field research means researchers observe, describe, analyze what exists in a certain study (Descriptive or Correlational)
• No manipulation of a variable
• Participants may or may not know they are under study
• Examples include asking individuals to record their food intakes for 7 days.
• Lab research involves tight controls on a study
• Lab research means subjects are carefully included or excluded
• Participants usually know they are being studied
• Examples include asking individuals to follow a specific diet for 7 days.