Evaluating Randomized Controlled Trials PDF

Evaluating Randomized Controlled Trials Mattie M. Follen, PharmD Objectives Identify and describe the use of formal criteria to assess quality of randomized trials Assess validity issues in randomized trials Apply general criteria to evaluate methodological rigor in randomized trials Ev...

Evaluating Randomized Controlled Trials Mattie M. Follen, PharmD Objectives Identify and describe the use of formal criteria to assess quality of randomized trials Assess validity issues in randomized trials Apply general criteria to evaluate methodological rigor in randomized trials Evaluate common biases in randomized trials Interpret and apply key findings in clinical practice Perform calculations applicable to RCTs Importance of Critical Evaluation RCTs can produce the strongest evidence if designed and conducted correctly Poor study design and methodology may produce misleading results which could negatively impact patient care Clinicians should be able to evaluate the impact of flaws on results for translation to patient care Criteria for Evaluating and Reporting Clinical Trials Standardized reporting requirements Consolidated Standards of Reporting Trials (CONSORT) Statement Provides standards for preparing reports of RCTs Helps improve accuracy and clarity in medical literature CONSORT Checklist CONSORT Checklist CONSORT Checklist Criteria for Evaluating and Reporting Clinical Trials Quality assessment scales By 1995, there were 25 different scales to evaluate RCTs! These scales vary in reliability and validity Examples: Jadad Scale Originally developed to assess pain research Includes three items: sequence generation, blinding methodology and withdrawals and dropouts Chalmers scale Methodology, statistical analysis, and presentation of results All scales include items assessing: Blinding Sequence generation Dropouts or withdrawals Article for Evaluation  Validity Degree to which a measurement truly assesses what it is intended to measure Factors, if conducted poorly, Type Description that may lead to decreased validity Internal The study is designed and conducted Randomization, Validity in an appropriate manner so as to measurement and arrive at accurate results assessment of variables, dropouts, overall methodology External The results of the study are able to Recruitment and selection of Validity be accurately generalized to the participants, inclusion and larger population exclusion criteria, sample size, setting Validity: Empagliflozin Example Internal validity Factors that could lead to decreased internal validity History – could be a factor but a control group was used Testing May not be an issue as both groups went through Instrumentation same process of assessments Maturation Statistical regression Could occur, but randomization helps prevent these Selection Attrition – similar number of patients in treatment and control group dropped out of study; intention-to-treat was used Randomization in the clinical trial helps minimize many threats to internal validity Validity: Empagliflozin Example External validity Factors that could decrease external validity Interaction of selection and intervention Effects of testing Effects of experimental arrangements Multiple treatment effects – GLP-1 RAs with CVD and/or CKD protective effects?? Empagliflozin study was a multi-center study conducted in 241 sites in 8 countries This would help with external validity because similar results may be seen with a broader population Baseline Characteristics “The trial aimed to include large numbers of patients without diabetes, patients with an eGFR of less than 30 mL per minute per 1.73m2, and patients with low levels of proteinuria.” About 45% of the patients had diabetes About 65% of patients had an eGFR > 30 mL per minute per 1.73m2 About 51% of patients had high levels of proteinuria Critically Evaluating RCTs What is evaluated in RCTs: Study sample Randomization and blinding Intervention and control group Clinical endpoints Trial findings Limitations and implications Study Sample Investigators should review the recruitment strategies and eligibility criteria Areas in the article that review this: Recruitment strategies Inclusion criteria Exclusion criteria Sample size calculations Baseline characteristics of the study population Study Sample Recruitment Helps determine the similarities and differences of the population that is studied and the population you want to treat Difficult to determine generalizability when small sample in a single center is used Study Sample Inclusion criteria Ensure appropriate patients are included Create a homogenous group likely to respond to treatment Not specific enough → difficult to determine who the treatment may benefit Exclusion criteria Do not include patients for whom the treatment may be unsafe Extensive → difficult to extrapolate the findings to the general population Study Sample Sample size and power calculation Should be calculated for the primary endpoint a priori Helpful if you are trying to determine if study was underpowered Assessment of possibility of type II error (beta) Failing to find a difference when one actually exists (false negative) Did they meet power? Study Sample: Empagliflozin Example Power: 1,070 patients necessary to have 90% power to detect a risk of a primary outcome event that was 18% lower in the empagliflozin group than in the placebo group The study actually enrolled 6,609 patients Study Sample: Empagliflozin Example Superiority Empagliflozin would be considered better than placebo if… the risk of a primary outcome event (first occurrence of progression of kidney disease or death from cardiovascular causes) was 18% lower in the empagliflozin group than in the placebo group Randomization and Blinding Defined randomization methodology Randomization helps ensure groups are similar Stratified randomization: allows for balanced assignment to groups based on certain characteristics Blocked randomization: common form of restricted randomization used to ensure similar sample sizes of the treatment groups Successful randomization = internal validity Randomization and Blinding Blinding Study should specify blinding techniques Who was blinded? How was concealment ensured? Double-dummy format – not applicable to empagliflozin study Multiple placebos (if two active medications) Identical looking tablets/capsules Sham monitoring parameters may be required Randomization and Blinding: Empagliflozin Example Patients were randomly assigned to receive empagliflozin 10mg once daily or matching placebo – parallel study design Patients were stratified according to diabetes status, eGFR, and urinary albumin-to-creatinine ratio at baseline Study was double-blinded Placebo controlled Intervention and Control Groups An appropriate comparator is essential Placebo may be unethical Appropriate for empagliflozin study If active control used, must be correct dose and have efficacy established Treatment adherence should be assessed Determine if efficacy is due to treatment or if significant amount of patients discontinued therapy in control group Intervention and Control Group: Empagliflozin Example Placebo was an appropriate comparator Characteristics of treatment groups are similar and groups were treated similarly Protocol was adequately described at NEJM.org Number of drop-outs was similar in both groups At each follow-up visit, patients provided information regarding adherence to the assigned trial regimen and details of concomitant medication use Clinical Endpoints Primary outcomes should assess clinical outcomes Should be measured by validated instruments that have shown to evaluate the outcome studied Objective vs. subjective measures Surrogate outcomes Composite endpoints Clinical Endpoints: Empagliflozin Example Primary outcome: Composite of first occurrence of progression of kidney disease or death from cardiovascular causes Progression of kidney disease was clearly defined and appropriately measured End-stage kidney disease (the initiation of maintenance dialysis or receipt of a kidney transplant), a sustained decreased in the eGFR to less than 10 mL per minute per 1.73m2, a sustained decrease from baseline in the eGFR of at least 40%, or death from renal causes Data for progression of kidney disease and deaths from cardiovascular causes were reported individually and in composite Objective, true clinical endpoints Trial Findings Intention-to-treat analysis Only method of evaluating data that preserves randomization Patients that are enrolled in the study and provide informed consent are analyzed Sample size should always be accounted for Per-protocol Only patients that comply with treatment protocol are included in analysis Risk of creating imbalance between groups Reporting of appropriate external factors Concomitant medications Results reported for all outcomes measured Adverse effects reported Trial Findings: Empagliflozin Example Used intention-to-treat 6,609 patients were randomly assigned and included in the analysis External factors – concurrent medications Results were reported for all outcomes Adverse effects were reported Trial Findings Should report statistical test AND its appropriateness Using incorrect statistical tests = type I and type II errors Trial Findings Measures of variance (standard deviation and confidence intervals) reported in addition to p-values Authors should list confidence intervals Provides greater evidence regarding magnitude and precision of results Pre-defined treatment effect size Non-inferiority margin Superiority Trial Findings: Empagliflozin Example Appropriately used a Cox proportional hazards model Relates several risk factors simultaneously to survival time Primary outcome Progression of kidney disease or death from cardiovascular event occurred in 432 patients in empagliflozin group versus 558 patients in placebo group Hazard ratio of 0.72 (95% CI 0.64 to 0.82) Indicates risk of a primary outcome event is 28% lower in the empagliflozin group than placebo Greater than the predefined 18% → empagliflozin is considered superior to placebo Trial Findings Appropriate conclusions based on methodology Focus of conclusion should be on the primary outcome Variance noted by standard deviation and confidence interval may be useful Analysis of subgroups Should be planned a priori Many subgroups = smaller groups = type I error chance increases There should be justification for investigating that subgroup IF there is statistical significance → further studies should be conducted to confirm Trial Findings: Empagliflozin Example Author’s conclusion “In this population of patients with a wide range of GFRs, levels of albuminuria, and causes of CKD, empagliflozin led to a risk of progression of kidney disease or death from cardiovascular causes that was 28% lower than that with placebo, with no safety concerns.” Limitations and Implications Weigh potential degree of bias with the ability to generalize the results Limitations and Implications Discussion part of article Any potential sources of bias and limitations should be recognized and described Compare results with other studies Look for consistency Limitations and Implications: Empagliflozin Example Large multi-center study conducted in 8 countries Placebo controlled parallel study Utilized objective primary outcome measures Included broad eligibility criteria High level of adherence to the trial regimen and almost complete follow-up of all patients Limitations and Implications: Empagliflozin Example Authors include comparisons with other studies investigating SGLT2 inhibition on the progression of kidney disease or death from cardiovascular causes (CREDENCE and DAPA-CKD trials) Authors stated that findings were similar among trials Implications for broader population Included mostly Caucasian and Asian patients; would have preferred more diverse population May be difficult to extrapolate study results due to many subgroups (co-morbid conditions, medications, etc.) Was the benefit due to empagliflozin or other medications that may have renal or cardiovascular protective effects? How do the different SGLT2 inhibitors compare to each other? Application to Clinical Practice What is currently known about the topic? What does this study add that was previously unknown or unclear? What questions remain? How do these results translate to specific patient care? Application to Practice: Empagliflozin Example Should empagliflozin be recommended for patients with chronic kidney disease who are at risk for disease progression? Empagliflozin was better than placebo in efficacy and similar to placebo in safety Things that were not answered in this trial that are important: Who would benefit from using empagliflozin over other SGLT2 inhibitors? What is the cost-effectiveness? Application to Practice Relative risk reduction Degree of difference in risk between the groups Does not take into account the overall incidence of events and could be misleading Absolute risk reduction Takes into account the rate of events Calculations Review Risk: how likely an event will occur when an intervention is given 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑠𝑢𝑏𝑗𝑒𝑐𝑡𝑠 𝑖𝑛 𝑔𝑟𝑜𝑢𝑝 𝑤𝑖𝑡ℎ 𝑢𝑛𝑓𝑎𝑣𝑜𝑟𝑎𝑙𝑒 𝑒𝑣𝑒𝑛𝑡 𝑅𝑖𝑠𝑘 = 𝑇𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑠𝑢𝑏𝑗𝑒𝑐𝑡𝑠 𝑖𝑛 𝑔𝑟𝑜𝑢𝑝 Relative risk: difference between groups (as likely) 𝑅𝑖𝑠𝑘 𝑖𝑛 𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡 𝑔𝑟𝑜𝑢𝑝 𝑅𝑒𝑙𝑎𝑡𝑖𝑣𝑒 𝑟𝑖𝑠𝑘 = 𝑅𝑖𝑠𝑘 𝑖𝑛 𝑐𝑜𝑛𝑡𝑟𝑜𝑙 𝑔𝑟𝑜𝑢𝑝 Relative risk reduction (RRR): how much the risk is reduced in the treatment group compared to the control group (less likely) 𝑅𝑅𝑅 = 1 − 𝑟𝑒𝑙𝑎𝑡𝑖𝑣𝑒 𝑟𝑖𝑠𝑘 Absolute risk reduction (ARR): true difference in risk; includes the reduction in risk and the incidence rate of the outcome 𝐴𝑅𝑅 = | 𝑟𝑖𝑠𝑘 𝑖𝑛 𝑐𝑜𝑛𝑡𝑟𝑜𝑙 𝑔𝑟𝑜𝑢𝑝 − 𝑟𝑖𝑠𝑘 𝑖𝑛 𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡 𝑔𝑟𝑜𝑢𝑝 | Calculations Review Number needed to treat (NNT): effectiveness from an intervention; round up 1 𝑁𝑁𝑇 = 𝐴𝑅𝑅 Number needed to harm (NNH): a measure of the risk to a patient; round down 1 𝑁𝑁𝐻 = 𝐴𝑅𝑅 Evaluating Randomized Controlled Trials Mattie M. Follen, PharmD