Lecture 3 2024 - Conducting A Survey PDF

Conducting A Survey Block 2, 2024 Week 3: Sampling frame and non-response  This week’s goals Discussing analysis plan. Start preparing for your field work: Total Quality Design (Dillman 2007) Advance letters? Letters of introduction...

Conducting A Survey Block 2, 2024 Week 3: Sampling frame and non-response  This week’s goals Discussing analysis plan. Start preparing for your field work: Total Quality Design (Dillman 2007) Advance letters? Letters of introduction Reminders? Incentives/rewards? Permission needed Choosing Mode: Face-to-Face, Telephone, Self-Administered Surveys Sampling respondents Total survey error perspective (Biemer 2001) Power analysis Sampling design Random samples (vs. non-random) Your sampling frame Sampling guideliness for non-probability samples 2  Letters of introduction Introduction letters (send with Q) ▪ Personalize your letters ▪ Explain who is doing the survey ▪ Why survey results are important to society ▪ How address of respondent is acquired and why respondent is eligible ▪ Time schedule ▪ Never ever forget the privacy statement ▪ Express gratitude ▪ Give information possibility (telephone number of helpdesk) ▪ Start instructions on how to answer questions 4 5  Reminders After the questionnaire is sent Example of schedule (timetable): ▪ Week 1 advance letter ▪ Week 2 survey posted or put on Internet or start field work ▪ Week 3-6 survey in field ▪ Beginning week 5 reminder by mail ▪ End week 6 telephone round Non-response conversion ▪ Short letter in which you state that not all surveys are returned and that you would appreciate if respondent could take the effort to return filled out survey ▪ Add a new form ▪ Keep track of non-response: the more ‘personal’ the stronger the appeal later very important for non-response analyses and measuring bias 6  Permission needed If survey is your own initiative you will need permission from schools, organizations, companies Surveys need informed consent of the respondent (APA). Sending it back is considered informed consent You need permission of parents to interview children younger than 15 years. You may need permission to use parts of existing questionnaires! (copyright / reference) Make sure to mention that research is by UU students 7  Sampling: why? Census versus sample from target population Representative (fair, no preference treatment) Random (inclusion probability known, lottery) Sample size (power and expected nonresponse) – Statistics for the population (including p-values etc) can only be computed for probability-based samples! – Therefore random (=probability-based) sample 8  Sampling frame The source or sources that include all population members and from which the sample is selected 1. First step is to define your target population 2. Then find or construct a sampling frame ▪ Sources for addresses ▪ Sources for background information Telephone books (landlines) Lists for special populations (medical, police, schools) 3. Match target population with sampling frame Coverage errors ▪ List contains units that are no longer a member ▪ List does not contain all members (missing information) ▪ Inaccurate information ▪ Double listing 9  Sampling How many respondents? Statistical power How to reach them? Population Sample Sampling design Sampling method Data Result Sampling frame 10 Sampling error versus sampling bias − Sampling error refers to sampling variance − Level of sampling error is controlled by sample size » As samples get larger, the distribution of possible sample outcomes gets tighter around the true population figure − Sample bias is not controlled by sampling size » Increasing sample size does nothing to remove systematic biases » Random sample does (external validity) − Not how many, but who 11 Ideal image Population Sample 12 Reality Population Target population (list available) Not on list Sample 13 Coverage errors may result in a distorted view of reality (biased results) Samples and coverage errors Small coverage Target population errors (list available) Not on list Not covered Part of the population covered during Large coverage sampling process errors 14 Generalizing ▪ Random sample: People are selected at random Sample is representative for the entire population Generalizing from sample to population possible High external validity 17  Random sampling ▪ In a simple random sample: All participants have the same chance of being selected All combinations of participants have the same chance of being selected ▪ Providing a list is unfortunately not always possible 18  Simple random sampling ▪ Simple random sample (SRS) ▪ Requires a list of all members of population ▪ Use computer to randomly select participants 19  Problem ▪ What happens when distinct groups in the population differ systematically from each other? ▪ Examples: Children in refugee centers in the towns of Utrecht, Zeist and Amersfoort show different signs of depression 20 Towards another sampling strategy Population Sample 21 If we use simple random sampling… Population Sample 22 Stratified sample design – why? Population Sample A stratified sample consists of the combination of multiple samples drawn from subgroups in the population 23  Cluster sampling Population Sample Why? You might not have the list of people but a list of groups…example? School/classes/pupils Confidentiality issues: they might not give you a list of pupils. 24 Multistage sampling Population Sample 25 Systematic sampling 26 Non-random sampling Accidental sampling Convenience Volunteer opt-in Quota sampling Non–random stratified Purposive sampling Educated guess Eligible subjects Why is this important? – Self-selection bias – Weighting (week 6) – Statistics (known inclusion probability) 27 Calculating sample size Traditional way: see formula Other ways: based on Power calculations For non-probability samples: guidelines 28 Calculating sample size: formula in case of simple random sampling 𝑛 𝑡 2 𝑝 × (1 − 𝑝 ) 𝑛 = 1− × 𝑁 𝑑2 n = the sample size or the number of completed interviews with eligible elements N = the size of the eligible population 𝑡 2 = the squared value of the standard deviation score that refers to the area under a Normal distribution of values 𝑝 = the proportion of a category for which we are computing the sample size 𝑑2 = the squared of one-half the precision interval around the sample estimate Calculating sample size Expressed in words, the formula states that sample size is function of a finite population correction factor (1-n/N) times the probability level for this sample occurrence times the variance or the variability of our variable in the population divided by the size of the confidence interval that we want for our estimate.” (Czaja & Blair). 30 1. Probability level − The question is how confident you want to be that your sample confidence interval (d) includes the population value. − In other words, out of 100 samples (of the same size), do you want 68, 95 or 99 samples to include the population value? − One standard deviation includes approximately 68% of the sample values (score=1.0), two standard deviations include approximately 95% of the sample values (score=1.96), and three standard deviations include approximately 99% of the sample values (score=2.58). 31 2. Variance − Express variable of interest as a percentage in two categories. − For example, 70% of the Dutch population visit a cinema (p) and 30% (q=1-p) do not. − The variance of a percentage variable is the product of the two percentages. − It can be difficult to estimate your variable of interest. You can use a pilot study (for example of 30 subjects), use the result from a previous or different study, or at the worst case make an educated guess. − The variance component does not affect the solution of the formula as much. If your estimate of the proportion is too high, meaning less variance in the population, your final confidence interval would be smaller than anticipated. If your estimate is too low, your final confidence interval would be larger. 24 3. Confidence interval margin of error that you tolerate. − the width of the range in which you want your estimate to fall. − The component d is expressed as plus and minus and represents one-half of the range. Thus, if we assume that 70% of the Dutch population visit a cinema, d is how large or small you need the estimated range to be around 70%. Is it ok to be 5% off (e.g. 75% or 65%) or do you need a more accurate number (