MED 106 Internal and External Study Validity PDF

MED 106 Research Methods and Medical Statistics Internal and External Study Validity Dr Constantinos Koshiaris Assistant Professor in Medical Statistics and Epidemiology Learning Objectives Differentiate between external and internal validity Describe the main concepts of internal and external valid...

MED 106 Research Methods and Medical Statistics Internal and External Study Validity Dr Constantinos Koshiaris Assistant Professor in Medical Statistics and Epidemiology Learning Objectives Differentiate between external and internal validity Describe the main concepts of internal and external validity Judge on internal and external validity published in research studies Interprofessional Scenario Scenario You are a doctor in an outpatient practice You work in a multidisciplinary clinic team (community nurse, dietitian, pharmacist, doctor) The team is trying to develop a group approach on what diet recommendations to give patients to help reduce prostate cancer mortality by examining the evidence The team runs across this Study… Stacey A. Kenfield, Natalie DuPre, Erin L. Richman, Meir J. Stampfer, June M. Chan, Edward L. Giovannucci, Mediterranean Diet and Prostate Cancer Risk and Mortality in the Health Professionals Follow-up Study, European Urology, Volume 65, Issue 5, 2014, Pages 887-894, ISSN 0302-2838, https://doi.org/10.1016/j.eururo.2013.08.009 Study details … A random sample of 47,867 male USA Health Professionals Followed up for 24 years To investigate the association between Mediterranean Diet (Med-Diet) and prostate cancer mortality Participants filled in a Food Frequency Questionnaire about their usual dietary intake, from which a Med-Diet score was constructed Incidence of and mortality from prostate cancer were determined from national cancer registries and hospital records The results revealed a 22% lower risk of mortality (Rate Ratio: 0.78, 95% CI: 0.67-0.90) amongst PCa-diagnosed men with greater adherence to the Med-Diet compared to PCa-diagnosed men with lower adherence to the Med-Diet. Question: How do we know if this study is a “good” study and applies to our population of interest Internal and External validity Internal validity Do our results represent the truth in the population we are studying? i.e. is there really an exposureoutcome association or are our results explained by something else? External validity Can our results be generalized to the general population and other similar populations? Generalizability Truth in the study Internal validity Truth in real life External validity Internal validity Do our results represent the truth in the population we are studying? Is there really an exposure-outcome association or are our results explained by something else? Internal validity In order to judge the internal validity of a study we must determine whether our results are influenced by the following 3 factors: 1. Chance (random/sampling error) 2. Bias (systematic error) 3. Confounding Internal validity: Chance Due to the random nature of sampling, when we use a sample to derive conclusions about the population, chance (random error) will always play an important role in our results Any estimate from any study is subject to chance Any finding could potentially be a chance finding The role that chance plays in our results can be determined by the p.value and the 95% confidence intervals How to we determine whether the validity of our results is affected by chance? We always check whether our findings are statistically significant using P.values and/or 95% confidence intervals If the exposure-outcome association is statistically significant: Then we can conclude with some degree of safety that our finding is not by chance (i.e. there is a true association in the population we study) Word of caution: Always examine how wide/narrow your confidence interval is to get an idea of the precision of your estimate. If the confidence interval is very wide then it is very likely that the sample size is not big enough and any findings from the study should be interpreted with caution How to we determine whether the validity of our results is affected by chance? How to we determine whether the validity of our results is affected by chance? Example 1 Association between obesity and risk of hepatic cancer Risk Ratio: 1.70 (95% CI: 1.55 to 1.85) Statistically significant Narrow confidence interval (precise estimate) Very unlikely to be a chance finding Example 2 Association between obesity and LDL-cholesterol levels Mean difference: 2.2 mmol/L (95% CI: -1.1 to 5.5) Not statistically significant Moderate to wide width confidence interval (not very precise estimate) We cannot exclude the possibility of this finding being due to chance How can we minimize the influence of chance in our findings? Just by taking a large, representative sample A large sample size will reduce standard error (and thus width of confidence interval) and increase study power, providing more precise estimates Before we start a study we should always conduct a sample size calculation to get a better idea of how many participants we need to recruit “Chance” (Random error) in the Study? 95% Confidence intervals do not include 1 (“All-cause Mortality”) Width of CI suggests precise estimates Results are statistically significant when comparing high vs low adherence We can conclude with high certainty that the observed association between Med-Diet and prostate cancer mortality is not due to chance Internal validity of the study is not compromised by random error Internal validity: Systematic error (Bias) Two types of bias have been covered Selection bias: Errors in the process of sampling which result in selecting a non-representative sample which results in biased estimates Information bias: Errors in the process of data collection which result in inaccurate assessment of the exposure and/or outcomes variables (e.g. recall bias, interview bias, etc.) How to determine whether the validity of our results is affected by bias? Not as straightforward as with chance In some cases bias can be easy to identify, but other times it can be a very difficult task even impossible at times When we collect our sample or when we make measurements, despite our best efforts we may still end up with a non representative sample or with inaccurate measurements without even realizing it The researcher/s need to make his/her reflection and self-criticism regarding the appropriateness of the sample taken and the accuracy of the measures taken Samples collected using convenience sampling methods and measurements resulting from participant self-reports should always be expected to be suffering from bias How can we minimize the influence of bias in our findings? Choosing a sample representative of the source population that we want to investigate by ideally using random sampling methods (minimizes selection bias) Choosing assessment tools that have high accuracy (minimizes information bias) Making a thorough investigation of the accuracy of the data collected This should be done in the process of data collection But also during the analysis (data cleaning) Do we have bias in the study? Study sample is large (random sample of 47,867 men) and collected randomly (no selection bias) Mediterranean diet was measured using self-reported food intake questionnaires Measurement error (bias) in exposure assessment is very likely Prostate cancer through national registries and hospital records Measurement error (bias) in outcome assessment is less likely The internal validity of the study is compromised by the possible presence of information bias in the exposure variable Internal validity: Confounding Confounding: a third factor (confounder) explains all or part of the association between an exposure and an outcome Implications of confounding: In general, it distorts the relationship between an exposure and an outcome It can “create” associations It can “mask” associations It can overestimate/underestimate associations How do we determine whether the validity of our results is affected by confounding? Under non-experimental conditions (i.e. in observational studies) it should be expected that confounding is affecting the results at some degree Consider confounding effects: If an unexpected result is obtained (based on previous evidence) Something does not make sense biologically How can we minimize the influence of confounding in our findings? Before starting the study identify all potential confounders for a given exposure-outcome association using: The literature Clinical expertise and subject knowledge Adjust for the potential confounders during data analysis This will give confounder-adjusted estimates, which are free (as much as possible) of confounding effects. Even after adjustment, always be cautious of residual confounding! Confounding in the study? The researchers adjusted their results for the following potential confounders: Age BMI Physical activity Smoking status Ethnicity Height Diabetes Family history of PCa Vitamin supplement use Confounding in the study? Most important potential confounders were identified and adjusted for Residual confounding could still be operating due to unknown confounders and imprecision in the assessment of existing confounders (information bias relating to confounders) The internal validity of the study could be compromised by the possible presence of residual confounding but not to a large extent as many of the major confounders have been adjusted for. Overall conclusion on internal validity Chance Adequate sample size Results were statistically significant and confidence intervals suggest precise estimates The observed association between Med-diet and prostate cancer mortality is not due to chance Bias Sample was random so minimised selection bias Exposure was self-reported so high likelihood of measurement error Outcome was based on registries and hospital records so measurement error is less likely Confounding Most important potential confounders were identified and adjusted for We cannot rule out the possibility of residual confounding which can compromise our results but not to a large extent Overall what would you conclude about the internal validity of the study? External validity The multidisciplinary team wants to look into the study’s External validity In the case of External validity, we are interested in whether our findings can be generalized to: The general population To other similar populations External validity (Example) A study in Cyprus found that stress is linked to increased highfat/sugar food consumption among a sample of University students External validity deals with the following 2 questions: Does the same apply for the general population in Cyprus? Does the same apply for University students in other countries? How do we determine whether our results have low external validity? A matter of correct judgment of the researcher Based on whether the study’s sample is representative or not of the general population or similar populations in other countries Going through the literature and investigating the evidence on the association of interest is always very helpful for evaluating the external validity of our findings We should never assume that our findings are externally valid without careful thought and investigation first How do we improve the external validity of our results? We should avoid highly selected samples if we want to generalize our results to the general population In such a case, choosing a random sample is the ideal approach If we have intentionally chosen a population with specific characteristics (i.e. >65 years old, male smokers, University students etc.) then we should make sure that our sample accurately represent the specific population subgroup In such case: Generalizations to the general population should be avoided Generalizations to other similar populations could be done but always under certain assumptions External validity of the Study Study conducted among male Health Professionals in the US Not at all a representative sample of the general population Low in terms of generalizability of findings to the general population But it could be a good representative sample of Health Professionals in other similar countries Could be adequate in terms of generalizability to Health Professionals in other countries similar to the US (i.e. “western countries”) Note: If a biological mechanism is being investigated, then we could be more flexible when generalizing the results even to the general population Overall conclusion on the study What do you decide with your multidisciplinary team about this study? Would you use it as part of your recommendations? Use Don’t use Explore more Note on Internal and External validity Apart from one of the criteria of internal validity (chance): Everything else related to judging the internal and external validity of a study is: Somewhat subjective in nature and Requires experience, critical thinking, and informed judgment from the researcher Summary Before deriving conclusions and generalizing research findings: Both internal and external validity of the relevant research studies should be thoroughly examined and confirmed! Examine the evidence critically before you use it in clinical practice Further reading Hennekens CH & Buring JE, Epidemiology in Medicine, 1987, chapters 14-15. Rothman KJ, et al, Modern Epidemiology (3rd ed.), 2008, chapters 9, 33. Steckler et al, The Importance of External Validity, Am J Public Health. 2008;98(1): 9–10.