Research Design: Validity Threats & One-Group Designs

• Research design is the outline, plan, or strategy used to answer a research question.
• Purpose of research design is to control for unwanted variation.
• Research design suggests/shows how data are statistically analyzed.

Designs with Threats to Internal Validity: One-Group Posttest-Only Design

• Treatment (X) is an IV/experimental intervention.
• Posttest measure (O) is a measurement of DV.
• Rarely helpful because there is no pretest or control group.
• Almost all threats to internal validity apply.
• It's helpful only when specific background information exists.
• Example: An institution assesses knowledge, attitudes, and behavioral outcomes of program participants upon completion of the program to evaluate its effectiveness.
• Positive DV measures may lead administrators to believe the program worked.

One-Group Pretest-Posttest Design

• Pretest measure is taken, followed by treatment (X), and then a posttest measure.
• Compare measures.
• Most Threats to internal validity exist (no control group).
• Causality can be inferred through identification and demonstration that internal validity threats do not exist.
• Example: A campaign distributes educational material and pooper-scoopers with plastic bags reduces dog litter.

Posttest-Only Design with Nonequivalent Groups

• Compare an experimental group (receives treatment X) with a control group via posttest measures.
• No assurance of equality of groups because random assignment isn't used.
• Control and experimental groups are not equal.
• Selection may confound the treatment effect.
• Comparisons of depressive vs. HCs on memory performance, for example.
• The dashed line indicates nonequivalent groups.

Strong Experimental Research Designs

• Answer a research question or tests a hypothesis.
• Controls for the effect of extraneous variables with control techniques.
• Has a control group: A group that does not get the IV or gets some standard value.
• Serves as a source of comparison to an experimental group.
• Should allow generalizability of the results.

Strong Experimental Research Designs: Requirements

• Manipulation of variables
• Random assignment
• Control of confounding variables
• Control and experimental groups
• Operational definitions
• Standardized procedures
• Replication
• Ethical considerations
• Blinding techniques (if applicable)
• Reliable and valid measurements
• Adequate sample size.

Pretesting Participants

• Several reasons for including pretesting in the experimental design:
  • Establishing baseline measures
  • Ensure group equivalence.
  • Identifying individual differences
  • Improving statistical power
  • Detecting attrition bias
  • Identifying ceiling or floor effects
  • Testing instrument reliability
  • Clarifying research hypotheses
  • Facilitating exploratory analysis
  • Enhancing interpretation of results
• Increases sensitivity through matching by taking pretest.
• Tests initial position to know pre performance on DV.
• Insures initial comparability should randomization not be feasible, especially for field experiments.
• Obtains evidence of change by measure differences before and after treatment

Difficulties in Pretesting

• Increases the amount of time or budget required for pre-investigation.
• Sensitizes research participants to the experimental treatment condition.
• Possibility of maturation.
• Possibility of a carry-over effect.
• Recall of previously learned material.

Between-Participants Designs: Posttest-Only Control-Group Design

• Experimental group is given Treatment X
• Posttest measure is taken for both the experimental and control groups.
• Random assignment creates equivalence.
• Use of control group eliminates most threats to internal validity.
  • Weaknesses of this design:
  • It does not guarantee equivalence of groups, particularly with small sample sizes.
  • There is no pretest to assess equivalence.

Pretest-Posttest Control-Group Design

• Pretest measures taken.
• Randomly assigned to either the control group or experimental group.
• Experimental group recieves Treatment X.
• Posttest measures taken for both.
  • Advantages:
  • Determines if a ceiling effect has occurred.
  • Can assess the effects of randomization.
  • Can assess the effects of additional variables.
  • Allows use of analysis of covariance to statistically control for pretest differences.
  • Allows researcher to assess the change in dependent variable from pretest to posttest.
  • Weakness: Might not generalize to situations with no pretest.

Within-Participants Posttest-Only Design

• Participants Included in all conditions (also known as repeated measures designs).
• Counterbalancing necessary to eliminate linear sequencing effects.
  • Advantages:
  • Increased sensitivity because effects of individual differences are controlled
  • Fewer research participants needed
  • Disadvantages:
  • Difficult for participants
  • Potential problem of differential carryover effects

Factorial Designs

• Cell is a treatment combination of two or more IVs.
• Main effect is the effect of one IV.
• Interaction effect is when the effect of one IV that depends on the level of another IV.

Factorial Designs

• An interaction occurs when two or more IVs have an interactive effect on the DV.
• This is where the effect of one independent variable depends on another.
• When displayed graphically an interaction yields non-parallel lines.

Within-Participants Designs (Repeated Measure Design)

• Same participants, same conditions, same DVs.
• Counterbalancing is necessary to eliminate linear sequencing effects.
• Advantages:
• Increased sensitivity because effects of individual differences are controlled
• Fewer research participants needed
• Disadvantages:
• Difficult for participants (many measures)
• Potential problem of differential carryover effects

Pretest-Posttest Design

• Primary, unique characteristic is that it includes a pretest in addition to a posttest
• Controls for intragroup history, maturation, instrumentation, regression to mean, and selection

Choice of a Research Design

• Will it give an answer to the research question or hypothesis?
• Will it provide control for extraneous variables?
• Should you use a between or within design?, examination of prior research literature can guide choice of design
• Many factors to consider:
  • use of control group
  • number of comparison groups
  • pretest(s)
  • within-participants or between-participants
  • number of independent and dependent variables