Understanding Experimental Validity & Blinding

• Study notes for research methods and biostatistics, lecture 7: randomized control trial II

Characteristics of Experiments

• To be considered a true experiment, a study must have three key characteristics.
• The independent variable must be manipulated by the experimenter.
• Subjects must undergo random assignment to groups.
• A control group must be incorporated within the design.

Difference Between Random Sampling and Random Assignment.

• Random sampling occurs during the study design phase.

• Random assignment occurs after assignment.

• Manipulation of variables occurs during the assessment phase.

• A control group is included in assessment.

Learning Objectives

• Evaluate experiments to determine internal and external validity.
• Look at experimental study design, including blinding and intention-to-treat.

Internal Validity

• Determines if the experimental intervention caused the observed change in the dependent variable.
• Asks if other extraneous factors can be responsible for change.
• Confounding variables present threats to internal validity, offering explanations for the relationship between independent and dependent variables.
• These confounders interfere with cause-and-effect inferences.

External Validity

• Whether results apply to persons, settings, and times different from those in the experimental situation.
• External validity refers to which the results of a study can be generalized beyond the internal specifications of the study sample.
• External validity concerns the usefulness of information outside the experimental setting.

Main Considerations for Evaluating External Validity

• Selection of patients
• Setting of the trial
• Difference between trial protocol and routine practice
• Outcome measures and follow up
• Adverse effects of treatment

Study Designs to Enhance Internal Validity

• Essential elements include manipulation of the independent variable.
• Random assignment and control groups are key.
• Non-essential elements include blinding and intention-to-treat.

Observational Bias

• Cognitive biases influence how one perceives things.

Blinding

• Justitia or "Lady Justice" is well known in law and order.
• Since the 15th century, Justice has been depicted blindfolded, which represents objectivity, without fear or favor, wealth, power, or identity.
• Blind justice means impartiality.
• The potential for bias is of concern in experimental studies.
• Protection is best achieved using a double-blind study.
• In double-blind studies neither the subjects nor the investigators know the identity of the treatment groups until after data are collected.
• A blind design involves hiding the identity of group assignment From subjects and those who provide treatment, measure outcomes, and analyze the data.

Single-Blind Trial

• Only one party is unaware of the medication taken.
• This can be either the investigator or participant.
• It is also called a single-masked study.
• In most cases this means subjects do not know.
• Blinding occurs for subjects, not for those who provide treatment or do the analysis.

Double-Blind Trial

• Neither the participants nor the investigators know which participants are receiving which treatment.
• Double-blind trials produce objective results, affecting the outcome.
• Double-blind trials are also called double-masked studies.

Triple-blind Trial

• None of the participants, investigators and data analysts know the identity of group assignment
• Triple-blind trials blind subjects, those who provide treatment, and those who reduce the data and perform data analyses.

Open-Label Trial

• Both the researchers and the participants know the identity of group assignment
• Open-label trials reveal the treatment received to subjects and the researchers.

Blinding Technique

• Requires that treatments be coded so data collection is complete, so the code can be broken and assignments revealed.

Issues Related to Blinding

• Blinding may not always be feasible such as in surgery trials.
• Drug related effects un-blind subjects and investigators.
• The FDA recommends a questionnaire at study completion to investigate the effectiveness of blinding.
• CONSORT recommends reporting how the success of blinding was evaluated.
• When observing serious adverse events during blinded clinical trials, the FDA says sponsors should compare rates in the investigational and control arms,.
• Assessments should be done by a monitoring committee or a firewalled independent safety committee.
• Blinding occurs in the study sample stage.

Compliance

• Compliance (adherence) is the extent to which patients correctly follow medical advice.
• Compliance refers to medication compliance.
• In RCTs, patients do not comply with treatment., so the question becomes whether to exclude or include these subjects in any analysis.

Withdrawal from Trial

• Subjects may choose to end participation, resulting in missing outcomes,. leading to the question of whether to include or exclude these subjects from the analysis.

Intention-to-Treat (ITT)

• The ITT analysis includes all randomized patients in the groups to which they were randomly assigned, regardless of adherence or subsequent withdrawal.
• Means that data are analyzed assuming each patient receives the treatment to which he was assigned.
• We analyze the data according to the way we intended to treat the subjects.
• Also, noncompliant subjects are included in analysis according to the original random assignment.
• "Once randomized, always analyzed"

Non-ITT Analyses

• As-Treated: patients are included in the group corresponding to the treatment they actually receive.
• Patient compliance and "switchovers” are considered in As-Treated analysis.
• Per-protocol: those who do not meet the criteria or adhere protocol are exclude in this type of analysis.
• The restricts analysis to the comparison to ideal patients who complied with the clinical trial instructions.

Example Analysis

• The Coronary Drug Project (1966-1975) was a randomized, multi-center, double-blind, placebo-controlled trial of clofibrate for treatment of coronary heart disease, featuring 1103 men on clofibrate and 2789 men on placebo.
• ITT analysis of 5-year mortality on clofibrate was 20.0%, 20.9% on placebo (p=0.55).
• There was speculation that compliant patients would show the clofibrate benefit whereas poor patients have similar mortality to placebo subjects.
• Good compliance was defined as 80% of protocol.
• In the clofribate group, the mortality rate in 5 years was 15% in those who were compliant, versus 24.6% in those that were not compliant.
• Compliant subjects had significantly lower mortality (p=0.0001), according to -treated analysis!
• Compliant patients with placebo also had lower mortality (p<0.0001).
• Compliers had some other unknown characteristic that improved their life expectancy.
• The health habits of the subjects drove compliance rather than the other way around
• Clofibrate wasn't more beneficial than placebo, supporting the ITT result of non-significant effect.

Why ITT over others

• Non-compliance, withdrawals and other deviations from protocol related to treatment outcomes or extraneous variables.
• Non-ITT methods may introduce bias into the analysis
• ITT analysis maintains the balance from the original randomization and gives unbiased estimate of treatment effect.
• ITT analysis results are more conservative and minimizes type I error (false positive).
• Results from ITT may be better generalized to real clinical settings.
• FDA mandates that the primary design and analysis strategy for industrial clinical trials should be ITT.
• Consolidated Standards of Reporting Trials (CONSORT) guidelines on reporting of RCTs should be analyzed by "intention-to-treat" principle with number of participants.