Lecture 4 2024: Best Practices in Equality, Diversity & Inclusion in Survey Research PDF

(Some) best practices in equality, diversity and inclusion in survey research The case of hard-to-reach populations Outline Discuss the importance of EDI in survey research Outline best practices for incorporating EDI principles Highlight common challenges and solutions...

(Some) best practices in equality, diversity and inclusion in survey research The case of hard-to-reach populations Outline Discuss the importance of EDI in survey research Outline best practices for incorporating EDI principles Highlight common challenges and solutions Hard-to-reach populations Questionnaire design What is “EDI?” Equality: we must ensure that individuals, or groups of individuals, are not treated less favourably because of their characteristics Equality also means equality of opportunity. Diversity: recognising, respecting and celebrating each other's differences Inclusion: creating an environment where everyone feels welcome and valued. An inclusive environment can only be created once we are more aware of our unconscious biases, and have learned how to manage them. Total Survey Error framework Hard-to-reach populations Groups of people who are difficult to engage with or include in surveys due to different factors These populations can be defined social, geographic, cultural, or structural barriers. These may impact onto the access. Important issue (1) Hard-to-reach populations may be impossible to fully enumerate with even a hypothetical sampling frame The sampling frame may be even missing They frequently constitute a small proportion of the general population and are floating or socially “invisible”. Important issue (2) They may experience social marginalization from engaging in stigmatized activities (Johnston, 2014; Tourangeau, 2014): people who inject drugs men who have sex with men, and survivors of sex trafficking. These groups are difficult to identify and recruit due to their marginalized status, desire for anonymity, stigma associated with their identities or behaviors, and/or fear of legal repercussions Other issues: mistrust of researchers, who are rarely members of the community under study How to sample these people? Respondent-Driven Sampling Non-probability-based Convenience Snowball sampling Probability-based Time-location sampling Respondent-driven sampling Convenience sampling Implementation: Enroll participants that are accessible to the researcher and available. Examples include interviewing people outside of stores, advertising on social media, visiting locations, or events where members of the target population may be found Strengths: Exploratory or qualitative research; relatively cheap, convenient, fast Weaknesses: Difficult or impossible to measure systematic error; results are not generalizable Snowball sampling Implementation: Recruit one eligible person who refers their peers, who then recruit their own peers Examples: Covid surveys starting from a person who was positive then this person suggested contacts. Strengths: Exploratory or qualitative research; helpful for conducting formative research (identifying/defining a public health problem); burden of recruitment is placed on members of the population; useful when trust is required to recruit subjects. Weaknesses: Difficult or impossible to measure systematic error; inability to know how the sample resembles the target population, resulting in limited generalizability; seeds have strong impact on sample composition; dependent on knowledge/skill of team; time-demanding; sometimes results are interpreted as if obtained after probability-based sampling when they should not be [14 Time-location sampling (1) Implementation: 1. Develop complete list of places visited by members of the target population and times at which each place might be visited (sampling frame); 2. randomly select place/time pairs and sample at random persons at the selected places/times; 3. use information about the sampling design to estimate probabilities of inclusion (optionally including self-reported information on frequency of venue attendance); 4. weight the analysis, using the inverse of the probabilities of inclusion Time-location sampling (2) Strengths: Unbiased estimators exist if assumptions are met; generalizable to the target population; reproducible, which can be important for multi-city or multi-year projects Weaknesses: Time-consuming and costly to establish and update list; people who do not attend any venues are not included; non-response may correlate with venue; calculating weights requires inclusion of questions about attendance that rely on recall; some venues may be inaccessible if proprietors refuse participation. Respondent-driven sampling Implementation: 1. Recruit a few members of the target population (seeds) non-randomly; 2. the seeds recruit the first wave of participants; 3. the first wave, in turn, recruits the second wave, and so on, until the target sample size is achieved; recruitment is tracked; formative research is needed to characterize the target population; weights are established based on participants’ network sizes. Strengths: it can be useful to sample populations who may not trust sponsoring agencies or visit public venues and therefore may recruit participants unknown to the investigators; the burden of identifying participants is placed on population members Weaknesses: Success depends on network structure; quantifying network size of participants is necessary but may be challenging; participants must know whether others in their network belong to the population of interest; it is difficult to estimate the refusal rate. Quota samples: within the non-random samples Example based on age (the people are not sampled at random!) The Evidence for Equality National Survey https://www.evensurvey.co.uk/ EVENS is the largest and most comprehensive survey of ethnic and religious minorities in the UK. EVENS has 14,000 participants including 10,000 who identify as ethnic minorities. EVENS uses innovative non-probability survey methods to improve representation across ethnic groups. Data deficiencies: EDI! Social surveys in the UK tend to represent a limited number of (broad) ethnic groups Survey sampling favours (by design) areas of residential clustering of ethnic minorities General surveys do not have questions bespoke to the concerns and experiences of minority groups Census/administrative data have good population coverage but limited topic coverage No prior application of rapidly developing non-probability survey methods to (numerically) small population in the UK. Diagrammatic summary of routes into the EVENS survey Overview 30 minute questionnaire, developed in collaboration with partners Completed online via open web link or via Computer Assisted Telephone Interview (CATI), available in 14 languages All who consider themselves to be ethnic or religious minorities are invited to take part (non probability survey design) No exclusion/inclusion based on ethnic/religious minority identification neighbourhood Residency of England, Wales or Scotland required for eligibility Incentive of £10 (voucher) issued upon completion of survey (do you remember this incentive?) Data collection from February to October 2021 Original questions plus those borrowed/developed from existing surveys Administered by Ipsos Full ethical approval (including amendments), University of Manchester. Responsive recruitment Using a non probability approach means there is no sampling frame and no (standard) response rates Target quotas were set by ethnicity*age*sex*region to maximise the representativeness of the sample An initial registration/screening questionnaire ensured eligibility in terms of GB residence and ethnic minority (self) identification Quota targets were monitored daily to enable responsive and adaptive recruitment i.e. increase/decrease recruitment efforts according to whether quota targets were being met Study representativeness constantly. Issues with online surveys and in the EVENS too In March 2021, the daily monitoring of the survey responses by the EVENS team and Ipsos revealed a spike in survey completion (and completion via snowballing) We identified features of completions that caused concern including: clustering in certain language and ethnic group options non-standard questionnaire timing and completion times use of fake postcode information suspicious open-ended responses suspicious IP addresses use of suspicious email addresses (for receipt of voucher) Patterns suggested completes had come from ‘survey farms’ and digital ‘bots’ The survey was paused in order to instigate data quality initiatives Follow up email communications confirmed suspicious cases; voucher payments were not made for completions considered to be spurious. Additional quality assurance measures Additional digital fingerprinting Introduction of reCAPTCHA question at the beginning of the survey Extra validation before supply of snowballing links Switch from digital to postal delivery of vouchers Revised daily data validation checks, Establishing trust… Ethnic minorities in the UK Data adjustments: weights Weights have been created for the EVENS dataset. Applying the weights enables you to use the data as if it were representative of the GB population. EVENS weights account for coverage errors and selection bias Adjustments for coverage error align the EVENS sample with the GB population in terms of key demographic characteristics: ethnic group, age (group), sex, region Adjustments for selection bias correct for the greater likelihood of some people (with particular characteristics) being more likely to take part in a/the survey Propensity score approach (quasi randomisation) is used that links EVENS participation to that from a reference probability sample Selection bias was adjusted on the basis of: voting eligibility, interest in politics, subjective general health, participation in religious events, religiosity, citizenship, trust in parliament, trust in the police Supporting data used: Censuses 2011 and 2021, Annual Population Survey, Ethpop estimates (2019), European Social Survey. Another example of hard- to-reach populations LGBTIQ populations The EU Agency for Fundamental Rights (FRA) EU LGBTIQ survey: https://fra.europa.eu/en/publication/2024/lgbt iq-crossroads-progress-and-challenges (now third round) Some references Johnston LG. Sampling Migrants: How Respondent Driven Sampling Works. In: Tyldum G, Johnston LG, editors. Applying Respondent Driven Sampling to Migrant Populations: Lessons from the Field. London: Palgrave Macmillan UK; 2014. Tourangeau R. Defining Hard to Survey Populations. In: Edwards B, Johnson TP, Wolter KM, Bates N, Tourangeau R, editors. Hard-to- Survey Populations. Cambridge: Cambridge University Press; 2014.