Session 4 - Measurement Instruments PDF

Session 4: 4th Oct - pt1 Created @September 24, 2024 12:27 PM Tags Measurement Instruments: Lead Teachers: Andrew Sommerlad To be covered: Choosing Measures for mental health research, Domains of outcome me...

Session 4: 4th Oct - pt1 Created @September 24, 2024 12:27 PM Tags Measurement Instruments: Lead Teachers: Andrew Sommerlad To be covered: Choosing Measures for mental health research, Domains of outcome measurement Reading list: PSBS0002: Core Principles of Mental Health Research | University College London (talis.com) Class Prep: Read: New Trends in Assessing the Outcomes of Mental Health Interventions Read: How Confident can clinicians be About Using Outcome Measures Across Cultures? Website: CORC Child Outcomes Research Consortium Read: Overcoming Barriers to Recruiting Ethnic Minorities to Mental Health Research: a Typology of Recruitment Strategies - BMC Psychiatry Read: Outcome Measures Recommended for Use in Adult Psychiatry Webpage: Catalogue of Mental Health Measures Read: Quality Criteria Were Proposed for Measurement Properties of Health Status Questionnaires - Journal of Clinical Epidemiology Webpage: Making Mental Health Research More Inclusive Prep Notes: Session 4: 4th Oct - pt1 1  What do we Measure in Mental Health? Objectives: To introduce to some of the main domains of measurement in mental health research To familiarise with pros and cons of the measures Selecting a measure for mental health Which Contexts are scales used: Research Studies: Describe participants incl. diagnosis Measure outcomes of intervention (or of fl/u) Measure potential aetiological variables, or other variables that may influence outcomes Clinical practice: Chart individual progress and assess individual needs Audit or evaluate services Assess needs within a service What is the purpose of making a research diagnosis? Epidemiology: cannot compare rates without all meeting similar criteria Treatment studies: need to understand who is being treated to understand outcomes Alternative: dimensional measurements Session 4: 4th Oct - pt1 2  Types of measures used in diagnosis Semi Structured Interviews: Structured Interviews (self-report or lay interviewer) Allows exploration of symptoms in depth Reasonable correlation with full diagnostic assessment Allows participants to tell their story rather than ticking boxes Don’t require clinician/extensive training Gold standard assessment Harder to adapt to participant e.g. Relatively expensive and elaborate with cognitive difficulties E.g. SCAN (Scales for E.g. Clinical Interview Neuropsychotic Assessment) Schedule Symptom Measures 1. Overall screening measure of psychological ‘caseness’ - e.g. The GHQ (General Health Questionnaire) 2. Measure covering a range of symptoms of psychiatric disorder - e.g. the BPRS (Brief Psychiatric Rating Scale) 3. Measures developed specifically for a particular disorder e.g. BDI (Beck Depression Inventory) Psychological Measures Many measures of psychological characteristics find application in mental health research e.g.: Personality traits relevant to mental health problems Global Personality Assessment, assessments of specific traits Coping Strategies & Cognitive Style The COPE Inventory, The Paranoid Thoughts Scale Cognitive functioning Global: Mini-mental state examination, Addenbrooke’s Cognitive Examination Session 4: 4th Oct - pt1 3  Specific: E.g. Wisconsin Card Sort for executive functioning Relationships and attachment: Adult Attachment Interview, Camberwell Family Interview for Expressed Emotion Social Functioning and Disability Social Functioning - carrying out daily living activities and social roles - from personal care to work Disability - limitations on activities that result from a physical impairment Scales often informant-rated (clinicians, relatives) e.g. Social Functioning Questionnaire, Life Skills Profile Interaction between illness/impairment and society: how much an illness restricts social functioning depends both on social role expectations and on how much society helps you overcome the impairment Scales include: Global Assessment of Functioning (GAF) WHO Disability Assessment Schedule (WHODAS) Social Functioning Questionnaire (SFQ) Quality of Life Two types of measure: 1. Objective measures - used in health economic analysis e.g. in calculating how much a year’s good quality life costs a. E.g. EQ-5D — includes social functioning, experiences of pain and anxiety, self care, mobility 2. Subjective measures - where people asked to rate their satisfaction with various aspects of life a. E.g. MANSA (Manchester Short Assessment of Quality of Life) b. Correlation between objective and subjective quality of life is often low Outcomes that service users value Session 4: 4th Oct - pt1 4  When asked, service users are often quite uninterested in symptoms. PROMS - Patient-Rated Outcome Measures: Measures of self-rated recovery and hope reflect recovery movement in mental health (focus on personal journey) E.g. the QPR (Questionnaire on the Process of Recovery) Measures of well-being are values for capturing positive aspect of how people feel E.g. Warwick-Edinburgh Wellbeing Scale Usefulness in measuring recovery from mental health problems not fully clear Service users also often place high value on social relations. Focus on measure including social network size, objective and subjective social support, loneliness, social capital Defining Data - Applied Statistics Objective: You will be able to describe the difference between populations and samples and comment on statistical inference You will be able to comment on the different types of data and variables that we can create using the data from our samples Terminology: Defining Exposure and Outcome The outcome variable is the variable that is often the focus of our attention, whose variation or occurrence we are seeking to investigate and understand Session 4: 4th Oct - pt1 5  E.g. depression; eating disorders; psychosis; bipolar We are often interested in identifying risk factors or exposures that may influence the occurrence or severity of the outcome The purpose of a statistical analysis is often to quantify the magnitude of the association between one or more exposure variables and the outcome variable Power and Samples In research studies, we collect data on a sample from a much larger group (population). The sample is of interest not in its own right but for what it tells us about the population. Statistics allows us to use the sample to make inferences about the population. Because of chance, different samples from the population will give different results, and this must be considered when using a sample to make inferences about the population. The concept of sampling variation is at the heart of frequentist statistics and will be explained in the interpreting statistics lecture of the core module. Example: The population is often referred to in epidemiology and medical statistics as the “target population” For example, a researcher wishes to test her hypothesis that being lonely increases the risk of depression among young adults Session 4: 4th Oct - pt1 6  The target population may be defined as all adults aged 18-25 living in the UK Specifying the target population In any research study, it is important to carefully and precisely specify the target population Care should also be taken to ensure that the sample represents the target population The researcher may take a random sample of university students to test her hypothesis What are the potential problems with her approach? - If students differ from other young people in any way that affects their experiences of loneliness or depression (exposure and outcome), the sample and the finding may not represent the population. The finding will not be generalisable and will apply only to the population of UK university students Type of Data The raw data from a research study consist of observations made on individuals The number of individuals is called the sample size Any aspect of an individual that is measured - for example - their depressive symptoms, exposure to loneliness, age, gender or highest educational qualification is called a variable A first step in choosing how best to display and analyse data is to classify the variables into their different types This is important because the choice of statistical test to use depends on the nature of the outcome (i.e. how the outcome variable is classified) The main division is between numerical (quantitative) variables, categorical (qualitative) variables and rates Outcome: Numerical Variable Session 4: 4th Oct - pt1 7  A numerical variable is either continuous or discrete A continuous measurement can take on any number within the possible / plausible range E.G. BMI (26.42, 28.35) A discrete variable can only take on certain scores (whole numbers) such as the number of depressive episodes in 10 years (0, 2, 3, 4, 12) Note: Often, variables that are technically discrete are described as continuous and continuous is often used to mean numerical Categorical Variable A categorical variable assigns people to one of two or more qualitatively distinct categories (E.G. 1, 2, 3) A binary variable is categorical variable with only two categories E.G. clinical diagnoses (diagnosed with schizophrenia or not, 0 or 1) An ordered categorical variable assigns people to ordered categories E.G. socioeconomic status: low / middle / high Nominal categorical variables assign people to categories with no underlying order E.G. eye colour. Rates Rates of disease are measured in longitudinal studies and are the fundamental measure of the frequency of occurrence of events (such as illness or death) over time For example, 30-year mortality rates among adults with depression The rate of occurrence of psychosis in the Swedish population Summary The appropriate statistical methods to use depend on the nature of the outcome variable of interest. There are three main types of outcome: 1. Numerical (quantitative/continuous) variables such as the severity of depressive symptoms (e.g. a score ranging 0-27) 2. Binary outcomes (proportions, risks or odds) E.G. the proportion of people who do/do not meet diagnostic criteria for eating disorders Session 4: 4th Oct - pt1 8  3. Rates of morbidity, mortality or survival measured over time. E.G. number of people per 100,000 diagnosed with psychosis per year Notes: Measurement Instruments Objectives: To review the core principles of measurement selection, with a focus on practical and psychometric considerations. To practically apply principles of questionnaire design and psychometric properties and outline a plan for how these would be tested To understand some of the challenges in conducting research, including outcome measurement in a culturally diverse population Prep Group Task: You are designing a randomised controlled trial to assess whether a new form of group CBT benefits symptoms of Schizophrenia. You are considering the following outcome measures: The Positive and Negative Syndrome Scale (PANSS) The Psychotic Symptoms Rating Scale (PSYRATS) The Psychosis Screening Questionnaire What are the pros and cons of each of the instruments as primary outcomes for your trial and which would you choose? - for an 8 minute presentation (PowerPoint not needed unless wanted), ending with conclusion about which would be selected for the trial. Take into account the available evidence on the psychometric properties of each of these tools as well as your own views on their suitability. You will then be asked to present your ideas and findings to another study group in the session on Fri 4th Oct. Class task: A Questionnaire has been selected to be used in a survey of the prevalence and severity of depression in the general population. It appears to have good validity, and a pilot Session 4: 4th Oct - pt1 9  study was conducted in the course of its development. Could anything still go wrong? - ?has anything been changed or amended from the pilot Will the questionnaire pick up changes over time and differences between groups? Responsiveness: does the questionnaire pick up changes over time sufficiently well? May be defined in relation to stakeholder views: is a difference agreed by patients and/or clinicians as important detected by the instrument? Or there are ways of calculating from statistical distribution e.g. statistical definitions e.g. standardised response mean calculated from average pre/post difference and standard deviation (SD) Responsiveness is often included alongside reliability and validity as one of core psychometric properties in recent checklists of essential psychometric properties (e.g. COMET checklist) Does the questionnaire produce a useful range of scores? Precision: Do scores vary sufficiently among respondents for measurements to be useful? Floor and ceiling effects: Where many people score at the upper or lower end of a scale so results are not useful Could be a problem with the scale or use in an inappropriate population E.g. a scale to measure symptoms among people with severe mental health problems used in general population - may be more inclined to answer at lower end of scale These are sometimes subsumed within definition of responsiveness – some variations between lists of core psychometric properties. Is the questionnaire appropriate for the study in which it is being used? Appropriateness is generally study-specific rather than a property of questionnaires. Is this an appropriate tool for the study question? E.g. Does what’s measured map exactly onto the study research question? Session 4: 4th Oct - pt1 10  Is it designed for/suitable for this population? E.g. a measure of resilience developed and piloted with undergraduates may not be suitable for people with severe mental health problems – new pilot study often needed Is it going to be practical to use this tool? Feasibility: can you use the tool in the proper way in practice. Impediments (generally study/situation specific) might include: Lack of time or a lengthy tool Lack of access to training or of money to pay for an expensive tool Not having researchers who have the required training to use the tool Is the questionnaire acceptable to the people we plan to use it with? Acceptability is very important. Reasons for an instrument being unacceptable (which can change over time) include: People find it rude or stigmatising Language is hard to comprehend, or seems irrelevant Tone is pessimistic or distressing It looks off-putting, is glitchy, long or hard to use Response rate or partial completion is an indicator. Another reason why piloting/stakeholder consultation is essential, even for previously used instruments. Cross-cultural validity (sometimes defined as a component of construct validity) Cross-cultural validity: defined as measurement invariance – a scale performs the same way across cultures May aim for this or for instruments developed for or adapted to specific groups Either way – should be confident instrument is valid among people you use it with Can be challenging in highly diverse contexts e.g. London Cross-cultural validation and instrument translation: Session 4: 4th Oct - pt1 11  Processes required to produce a valid version of an instrument in another culture/language include: Translation and back-translation to ensure the meaning remains the same Exploring whether language/items have same meaning, often through qualitative work Exploring whether important aspects of the concept being measure are being covered Investigating psychometric properties (reliability, validity, responsiveness), piloting Conducting valid research in a diverse environment Concepts of cross-cultural validity can be challenging in a diverse contemporary environment E.g. where many people’s identities are complex and mixed, not straightforwardly part of a specific culture An apparently cross-culturally valid instrument could miss important aspects of minorities’ experiences E.g. impacts of racism – measuring exactly the same concepts in everyone may not always be right When developing and testing measures: study samples should in any case be diverse and representative of local populations Some important challenges to conducting research that is relevant for people from Black and Minority Ethnic backgrounds (an introduction) Small minority of mental health academics/researchers are from BAME backgrounds Study samples tend to under-recruit people from minorities whose mental health appears impacted by injustice and inequality Measures and study methods may not reflect important aspects of minority experience E.g. Impacts of racism and discrimination Questions that are important from a BAME perspective are not addressed: E.g. Very limited research on why people from Black backgrounds in UK are much more likely to be admitted to hospital compulsorily, even though this has been demonstrated again and again. Session 4: 4th Oct - pt1 12  Class task: How would you pilot questionnaire once designed and what would you be trying to establish? 1. Individuals complete the questionnaire - healthcare professionals and individuals who have used primary care 2. Assess how easy the questionnaire is to use or understand, does it assess what we’re looking for (construct validity), how long does it take, the things involved Session 4: 4th Oct - pt1 13