HE2801A Week 3 Measurement, Reliability, and Validity (Research Methods in Health Sciences) PDF

Session 3 Measurement, Reliability, Validity September 20, 2024 HS2801A: Research Methods in Health Sciences Fall 2024 Dr. Afshin Vafaei School of Health Studies  Today’s Class Measurement in health science  Exposure measurement  Preval...

Session 3 Measurement, Reliability, Validity September 20, 2024 HS2801A: Research Methods in Health Sciences Fall 2024 Dr. Afshin Vafaei School of Health Studies  Today’s Class Measurement in health science  Exposure measurement  Prevalence and incidence measures Measures of association (more in design sessions) Errors in measurement  Validity  Reliability Before starting the class 3  Methods of Measurement  Measurement “If a thing exists, it exits in some amount; and if exists in some amount, it can be measured.” American Psychologist E.L Thorndike (1874-1949) A Research Question Chemical A is a by- product of combustion engines used in cars which is released into the air when car engines are running. In a Thinking like a neighbourhood, health there has been researcher increasing traffic how would you during last years approach this and at the same problem? time the number of people with the What steps disease B is rapidly do you take? increasing. There is a biological theory that Exposure Measurement Available dose:  Cumulative vs. current Administrated dose:  The amount that comes in contact Absorbed dose (uptake):  The amount that enters the body Active dose (biologically effective):  That actually affects the specific target organ They can be surrogate for each other if the mechanism is known  Simple Definitions Count: The number of occurrence of an event  In a major Canadian city, average annual number of homicides: – 30 homicides per year in the 1970’s – 50 per year in the 1990’s – 90 per year in 2022-23  45 individuals older than 75 admitted to a hospital in London for hip fracture  Ratio Proportion  Relationship between 2 numbers  Relationship between 2 numbers  Numerator NOT necessarily  Numerator NECESSARILY INCLUDED in the denominator INCLUDED in the denominator  An example: (binary) sex ratio  Proportion always ranges between 0 and 1 2 =5:2 = 1 --- = 0.5 = 50% 2.5 : 4  Odd The probability of an event occurring relative to it not occurring Proportion of red (disease)=2/4=0.5 Odd of disease =2/2=1 Question: In a population of 100 older people, 25 people are diabetic, what is: a) the proportion of diabetes= 25/100=0.25; b) odd of being a diabetic=25/75=0.33 In the same population only 1 person has heart failure (HF): a) the proportion of HF= 1/100=0.01; b) odd of having HF=1/99=0.0101 Keep the conceptual and quantitative differences between ‘proportion’ and ‘odd’ in mind. Also, how they approximate for rare diseases  Rate  Speed of occurrence of an event over time Numerator  no. of EVENTS observed for a given time Denominator  population in which the events occur (population at risk)  Incorporates a set period of time Observed in 2022 2 ----- = 0.02 / year (2022) 100 Measures of Health Events  Measures of Prevalence Prevalence Rate: The proportion of the population (or population sample or sample subset) that has a given disease or other attribute at a specified time.  Obtainable from cross-sectional studies (Oct 25) Two types:  Point prevalence rate  Period prevalence rate  # with disease at specific Point PR = time Population at same time Example : Prevalence of diabetes in Canada If 3 million people in Canada have diabetes NOW and the current (2022) Canadian population is 30 million, the Point PR of diabetes is 10.0% (or 100 cases per 1000 persons) 3 ÷ 30 X 1000 = 100 cases per 1000 persons in 2022  Example #2: Prevalence of Arthritis in Postmenopausal Women 3,276 postmenopausal women participated in a research study. 597 of these participants had arthritis. The point prevalence of arthritis in this sample is: 18.2 ____% 18 per 100 persons ___ 182 ___ per 1,000 persons 1,822 per 10,000 persons  # with disease at specific time period Period PR = Total defined population at same period Example: Prevalence of meningitis in city A in 2023 (Jan 1-Dec 31) In city A (population 152,358) there were 6 case of meningitis in 2023. The Period 4 cases per 100,000 persons. PR of meningitis in this city during 2022 was ____ 6 ÷ 152,358 X 100,000  Measures of Incidence Incidence Rate: The proportion of the population at risk that develops a given disease or other attribute during a specified time period.  Obtainable from cohort (longitudinal) studies (Nov 1) What is the ‘population at risk’? What is the correct number of people going to the denominator? Whom should be excluded? Relation between Prevalence & Incidence Incidence Prevalence Resolution (recovery or death)  # new events during specified time period IR = Population “at risk” Example #1: IR cancer from 2010-2019 in county A 9,000 residents of county A were studied on Jan 1, 2010. Of these residents, 245 had a history of cancer. By Dec 31, 2019 a total of 179 new cases of cancer had been diagnosed in the “at risk” population. The 10-year cumulative incidence rate of 204 cases per 10,000 people. cancer (first diagnosis) in this cohort was ___ # new events = 179 at risk population = 9,000-245=8,755  Example #2: IR of Depression during 1st year of University 2,012 first year university students were sampled on Sept. 1st 2023. Of these, 112 had depression. On April 30th 2024, 345 of the sample had depression, including 79 of those who had depression in September. What was the point prevalence rates of depression on Sept 1st 2023 and April 30th, 2024? Sep 1st, 2023: (112/2012)=5.6%; April 30th, 2024 : 345/2012=17% How many “resolution” cases were there? 112-79 (remained depressed until April)=33 The IR of new depression during the freshman year (Sept 1st 2023 to May 30th, 2024) in this sample was ____%. 14.0 Note: Let’s assume those who had depression (either resolved or remained depressed) at the inception of the study (n=112) are not at risk of new depression. We have to removed them from the denominator (population at risk). (345 – 79) ÷ (2012 – 112) = 14.0% Not new Not at cases risk Incidence vs. Prevalence Incidence Measures frequency of disease onset What is new Prevalence Measures population disease status What exists All may be expressed in any power of 10 Per 100; 1,000; 10,000; 100,000 More when we learn ‘designs’ Measures of Association  Four Hallmarks of Health Studies 1) A research question/plausible theory 2) A well thought design to address the research question 3) Measurement of exposure and outcome 4) Analysis to compare groups i.e. rates of disease among Outcome exposed vs. in Yes No unexposed Exposure Yes A B No C D  Relative Risk Tells us how many times as likely it is that someone who is ‘exposed’ to something will experience a particular health outcome compared to someone who is not exposed Tells us about the strength of an association Can be calculated using any measure of disease occurrence: Prevalence Incidence rate Does not tell us anything about how much more disease is occurring  Calculation of Relative Risk Developed Disease Yes No Exposed to risk factor a b Not exposed to risk c d factor [a / (a + b)] Incidence in Relative Risk = exposed [c / (c + d)] Incidence in unexposed  Calculation of Relative Risk [328/(328+332)=0.50] Relative Risk = =1.14 [288/(288+370)=0.44)] Compared to those who did not receive a call:  Those who received a call were 1.14 (50/44) times (or 14%) more likely to attend for immunization  Example #1: Risk of heart disease in smokers 20,000 individuals without heart disease were identified in Sept 2015. Of these, 3,614 were smokers and 16,386 were non-smokers. Over a 5 year follow-up period, 552 of the smokers and 1,431 of the non-smokers developed heart disease. The Relative Risk of heart disease in smokers relative to non-smokers was ____. Developed Disease Yes No Exposed to risk factor a b Not exposed to risk factor c d Fill out the table and do the calculation  Example #1: Risk of heart disease in 20,000 individuals without heart disease were identified in Sept 2015. smokers Of these, 3,614 were smokers and 16,386 were non-smokers. Over a 5 year follow-up period, 552 of the smokers and 1,431 of the non-smokers developed heart disease. The Relative Risk of heart disease in smokers relative to non-smokers was ____. Developed Disease Yes No Exposed to risk factor 552 3,062 Not exposed to risk factor 1,431 14,955 Example #1: Risk of heart disease in smokers Developed Disease Yes No Exposed to risk factor a 552 b 3,062 Not exposed to risk factor c 1,431 d14,955 a / (a + b) Relative Risk = c / (c + d) 552 / (552 + 3062) = 1431 / (1431 + 14955) 0.153 = = 1.76 (probability of the disease in exposed 0.087 compared to non-exposed increased by 76%)  Example 2: Protein Deficiency and Kwashiorkor In case-control studies we in are only able to Kenyan estimate odds of occurrence (happening Children versus not happening) Odds Ratio=(odds of disease in exposed/odds of disease in unexposed) Disease Present Yes No Exposed to risk factor a 212 72 b More Nov. 1 Not exposed to risk factor c 88 228 d Odds of diseases in exposed 212 / 72 2.94 Odds = = 4 = 7.63 (a / b)of diseases in Odds Ratio= 88 / 228 0.38 unexposed (c/d) 6 Interpretation similar to other relative measures More about design- specific measures of association later Truth in Measurement  Random Systematic Error Error Error due to “chance” Error due to recognizable source This is systematic error in measurement, don’t confuse with systematic error in design(=bias ) next week  Random and Systematic Error It’s possible to have both random and systematic error  Precision Vs. Accuracy in Measurement A measurement tool/scale with high precision is reliable with high accuracy is valid  Insufficient Precision Can happen because 1. The measurement tool is not precise enough (a ruler in cm is not precise when 6.2 meaningful differences are in millimeter) ? 6.3 ? 2. Two (independent) interviewers rate the same person differently using the same scale (inadequate training?) 3. The same interviewer rates the same person differently Statistical procedures are developed to quantify these reliability issues  A Valid Measure We need to clearly conceptualize the variable we want to measure (recall questionnaire design) Then, make sure that the scale 1) measures what it is supposed to measure, relevant information, underlying construct 2) does not include unnecessary questions 3) is able to do what it is designed to do: predict, discriminate, etc. More relevant when a set of questions are used to measure a disease such as disability or depression, a behavioural traits such as gender role, etc.  Example Does the job it was designed to do: predicts, Measures what it is supposed to measure separates (discriminates) the sample No unnecessary questions More next week Vafaei et al. (2014) Evaluation of the Late-Life Disability Instrument (LLDI) in low-income older populations. J Aging Health; 26(3):495-515. doi:  Sources of Measurement Error Interviewer or observer Record abstracting (random) error Biased overestimation or underestimation Participants Recall Random or systematic  Identify the most Likely Sources of Measurement Error 1) Asking mothers with preterm birth if they were exposed to pesticide during pregnancy 2) Using 1st year psychology students to administer a questionnaire about depression without any training 3) Using only ‘students with a history of depression’ for 2 4) Asking a group of older adults about the number of their friends in high school Note: in real research always think of the other exposure or outcome group  Reducing Measurement Error Little (or nothing) can be done to fix measurement error once it has occurred Measurement errors must be avoided through careful study design and conduct Clear measurement protocols, pilot tests, validation Appropriate choice of instrument Is it accurate? Precise? Measurement errors cannot be “controlled” in the analysis What happens after a measurement error (for any reason)? Next week!  Also, Next week Random sampling error, bias Heavy (relatively) reading, study the three book chapters from different books on OWL Study “Bias – Foundations of Epidemiology” and “Random Sampling error” in full; Study pages 298-294 of “Chapter 15 Bias (289-294)  Pages after 294 (Confounding_EM) are for Week 6 (Oct. 13) “Webb_2017_Chapter 7_Supplementary” is NOT mandatory but a good reading These will be revisited several times in ‘design’ sessions Go to BrightSpace>>Assessment>>Quiz 1; you have time until 12:30 - Those with approved accommodations already have longer access - 6 True/false, MCQ; 4 fill in the blank (all with one word) - Save each question after answering, the quiz becomes unavailable at 12:30 - Marks will be released this afternoon, around 2-3pm - You may use your paper and digital notes - No googling or communicating others