HTH SCI 3C04 Week 3 - Appraisal of Intervention Studies - Part A W2023.pdf

HTH SCI 3C04 Research Appraisal & Utilization in Evidence Informed Decision Making Week 3 Critical Appraisal of Intervention Studies (Part A) 3.1 – Sampling Sampling Concepts Sample • the group of people in a study; a subset of the population Population • all individuals to whom the study res...

HTH SCI 3C04 Research Appraisal & Utilization in Evidence Informed Decision Making Week 3 Critical Appraisal of Intervention Studies (Part A) 3.1 – Sampling Sampling Concepts Sample • the group of people in a study; a subset of the population Population • all individuals to whom the study results should be applicable; target group for study Sampling • selecting a proportion of the population to study Generalizability • A portion of the population of interest is studied in the hope that the findings can be generalized beyond the sample to the population External validity • Degree to which the findings of a study can be generalized beyond the study sample Major Categories of Sampling PROBABILITY SAMPLING • random selection of subjects where each subject has an equal chance of being selected • most likely representative of population (not perfect but close) • can be very expensive and intensive NON-PROBABILITY SAMPLING • selection of subjects by a nonrandom method • rarely representative of the target population • may have limited generalizability due to sampling bias (interpret with caution) • convenient and economical Important Note: Non-probability sampling may be used in quantitative or qualitative studies, although non-probability sampling is more likely to be used in qualitative studies. Probability Sampling: Examples • Simple Random Sampling • Sampling frame (or a list of all population elements) is obtained • A sample (of the appropriate size) is selected at random – usually via a table of random numbers, a computer program, or an organization with expertise/services for supporting RCTs • Stratified Random Sampling • Sample is divided into strata (subsets) and subjects are randomly selected from each stratum • Enhances the sample’s representativeness Probability Sampling: Examples • Cluster Sampling • Sample is selected based on successive random sampling of units (e.g., selection of a sample of schools, from which a sample of students are then randomly selected). • Often used for national surveys • Systematic Sampling • Sample size is determined; first subject is randomly selected; then every kth individual is selected (k = sampling interval) Issues with Sampling SAMPLING ERROR The gaps between the sample's representativeness and the population’s known and unknown characteristics. Generally decreases as sample size increases. SAMPLING BIAS 1 type of health care access bias; patients referred to tertiary care centres are Referral filter bias typically more ill or have more rare disorders than patients who are “wellenough” to be followed in community hospitals. Selecting study participants from tertiary care centers may mean your sample does not “represent” a typical patient with a particular disease. SAMPLING BIAS 2 Individuals who volunteer to participate in studies may be different than those who tend not to volunteer (e.g., may have different exposures or Volunteer bias outcomes). Assessing Sampling Bias To assess whether sampling bias exists, ask: • Who was included in the sample? • What was the source of recruitment into the study? • How were subjects recruited? • Which people were approached to be in the study? (i.e., consecutive sample, volunteers, other?) • What were the inclusion and exclusion criteria? What were the demographics of the sample? Did they have medical conditions or other factors that could affect the results of the study? 3.2 – Critical Appraisal of Intervention Studies Critical Appraisal of Intervention Studies You need to ask yourself: • Are the results valid; how serious was the risk of bias? • What are the results? • How can I apply the results to patient care? Are the results valid? How serious was the risk of bias? What are some of the questions that you need to ask in relation to whether the study results are valid? Critical Appraisal of Intervention Studies Are the results valid? Randomization of patients to control and intervention group: ensures groups are the same for known & unknown factors that might influence outcome Allocation concealment: ensures randomization by preventing investigators from knowing upcoming assignments (and trying to change them) Length and completeness of follow up: ensures sufficient time has elapsed to see the outcome; and, ensures sufficient number complete the intervention Intention to treat: ensures randomization occurs by analyzing patients in groups to which assigned (“once randomized, always analyzed”), and handling missing data Critical Appraisal of Intervention Studies Are the results valid? Blinding: applies to each group involved in the execution, monitoring & reporting phases (patients, clinicians, data collectors, outcome assessors, data analysts) Baseline characteristics: similarity of groups (links to randomization) Variation between Groups: limit variation to the intervention (intervention should be only difference) Measurement bias: instruments should be reliable and valid; assessors should be blinded Random Allocation (Randomization) Were participants randomized to the treatment and control groups? • “Who/What” can be randomized?: • • • • Individuals Families Hospitals, wards or health units (cluster) Towns • Best methods for randomization: • Computer-generated methods (which does not allow for manipulation of randomization) • An agency that has no involvement in patient recruitment (e.g., pharmacy department) • An external trials office; this entails calling into (or logging into) a central registration office (or website) to ascertain group assignment • Random number table • Coin Toss (cumbersome) Other Methods of Allocation Less than ideal methods (because all of these are non-random): • • • • • Birth dates (odd/even) Chart numbers List of “next up” Day of week in clinic Convenience Randomization: The Benefits Is the sample representative and large enough? • randomization aims to ensure the sample is representative of the population from which it originated • sample size depends on factors like the desired sampling error, power and effect size • larger samples provide more accurate results and enable you to generalize to the population Are there baseline differences? • randomization reduces the likelihood of these differences Allocation Concealment Was the process of allocation concealed? • To ensure the clinician recruiting patients are unaware of which group the next patient will be allocated (and thus cannot ‘influence’ which group an individual is assigned to). • Method of allocation concealment: Use of sequentially numbered, opaque, sealed envelopes. • even with this approach, people still peek • independent randomization service best approach • Can get differences among outcomes if this is not done Completeness & Length of Follow-up What was the follow-up? (i.e., How long? How complete?) • Length: Judgment as to whether the time period was long enough for individuals to experience the outcome of interest. Use clinical experts and common sense to make this assessment. • Completeness (or “loss to follow-up”): Percent of patients that complete the study. Follow-up of > 80% is considered most ideal. Compare intervention and control group. Why is completeness of follow-up important? Intention-to-treat Analysis Was there an intention-to-treat analysis? Tell me what “intention-to-treat analysis” means and why it is considered important? Intention-to-treat Analysis Was there an intention-to-treat analysis? Need to consider: • Were patients analyzed in the groups to which they were initially randomized? • “once randomized, always analyzed”: regardless of whether patients completed intervention, got the wrong treatment, got partial treatment, died part way through the study Intention-to-treat Analysis https://www.youtube.com/watch?x-ytcl=84503534&feature=player_embedded&v=nnxg0FJwPjY&x-ytts=1421914688 Intention-to-treat & Missing Data • Intention-to-treat claims are linked to methods of handling missing data.1 • The Problem: “Depending on the assumptions made about the missing data, some methods of analysis that include all randomised individuals may be less valid than methods that do not include all randomised individuals.”2 • Recommendation: “Clinical trials should employ an ITT analysis strategy, comprising a design that attempts to follow up all randomised individuals, a main analysis that is valid under a stated plausible assumption about the missing data, and sensitivity analyses that include all randomised individuals in order to explore the impact of departures from the assumption underlying the main analysis.”2 1 Thabane et al. (2013), 2White et al. (2012), pg. 396. Study Blinding Was the study blinded? Need to consider: • Were all individuals involved in the study (e.g., patients, investigators, research staff) unaware of who is assigned to treatment or control groups? • Who can be blinded? Patients Data analysts Study personnel • Data collectors • Outcome assessors Report writers Sponsors Health care providers Study Blinding: Placebo • To aid in blinding, some trial participants may get something that is not really a treatment or “for real” • Sugar pill in vitamin C trials • Placebos less frequently utilized in many of today’s pragmatic RCTs. Group Discussion Questions 1. Why is study blinding important? 2. Are placebo-controlled trials unethical? 3. When should placebos not be used? Contamination & Co-intervention Were both groups of participants (intervention and control) treated equally except for the intervention? Co-Intervention Extra care or treatment given to one group and not the other Contamination Participants in the control group accidentally get the intervention (minimizing the potential differences in outcomes between groups), and vice versa Baseline Differences Were the two groups similar at the start of the study? • If not, was this difference taken into consideration during the analysis? Consider baseline or entry characteristics of participants in each group: • Emphasis on characteristics that are known to, or believed to, have an influence on the outcome of interest Measurement Bias Are the measures reliable and valid? Is the measurement subject to bias? References Thabane, L, Mbuagbaw, L, Zhang, S, et al. A tutorial on sensitivity analysis in clinical trials: the what, why, when and how. BMC Medical Research Methodology. 2013. 13, 92. http://www.biomedcentral.com/1471-2288/13/92 White, IR, Carpenter, J, & Horton, NJ. Including all individuals is not enough: Lessons for intention-to-treat analysis. Clinical Trials. 2012. 9, pg. 396-407,