Survey Design and Methodology

Conducting A Survey

• The presentation covers survey design and methodology, focusing on sampling frames and non-response.
• The analysis plan is a topic of discussion.
• Total Quality Design (Dillman 2007) is mentioned as a technique for field work preparation, including advance letters, letters of introduction, reminders, incentives/rewards, permission needs, and mode choices such as face-to-face, telephone, or self-administered surveys.
• The total survey error perspective (Biemer 2001) is considered in relation to sampling error, power analysis, and sampling design.
• The importance of random samples versus non-random ones is discussed, in addition to the importance of a proper sampling frame for non-probability samples.
• Introduction letters are crucial, to be personalized to the recipient, explaining the survey's purpose, importance to society, how respondent details were obtained, and their eligibility.
• Time schedules, privacy statements, expressing gratitude, and information access (help desk) are key elements for successful introduction letters.
• Reminders are needed after the questionnaire is sent using a timetable, reminding participants (using mail and/or phone).
• Important aspects of nonresponse include understanding non-response conversion techniques, keeping track of nonresponses, and its analysis.
• Permission to conduct surveys needs to be obtained from those governing institutions, organisations, or companies.
• Permission is needed for studies with children under 15, so parent permission should be sought.
• Copyright and referencing guidelines must be followed.
• A proper sampling plan is necessary to ensure that the sample is representative of the population.
• The concept of census versus sampling is discussed, along with the goal of representative sampling with a fair selection process.
• Random sampling (known probability of inclusion) is vital for statistical calculations. Sample size calculations based on probability sampling are crucial for accuracy.
• The idea of a simple random sample was discussed, and the lack of availability of comprehensive list in all cases.
• Coverage errors (incompleteness of the sampling frame) can occur so a sampling frame is needed for each population. 
• Calculating sample size is required.
• Conventional formulas to compute sample sizes for probability samples.
• Calculating sample size formula involves factors such as the sample size (n), the population size (N), the squared value of the standard deviation (t²), and the proportion of categories concerned with calculating sample size (p.)

Sampling Frame

• Defining the target population and constructing a sampling frame are first steps to survey research in which the source of data is relevant to the target population.
• Important sources include addresses, background information, telephone books, and lists of special populations like medical, police, and school records.

Sampling

• Understanding sampling design, method, and frame is essential for collecting data that generalizes well to a larger population. 
• Calculation of sample size and statistical power need consideration. How many respondents are needed? The statistical power is essential to consider.

Sampling Bias Versus Error

• Sampling error is controlled by sample size and increasing the sample size makes the distribution closer to the population figure.
• Sampling bias has no connection to sample size, it will not be reduced by increasing sample size.
• Random sampling minimizes bias, so understanding bias or variance is important for correct data collection and analysis.

Ideal Sampling vs. Reality

• Ideal sampling (in theory) would capture all population members and sample, but such a plan often has limitations. 

Samples and Coverage Errors

• Differences between the sampling frame and the target population can lead to coverage errors, and may affect the results of the study. 

Generalizing

• A random sample allows generalizing from the sample to the population with high external validity, in other words results can be generalized to the larger population.

Random Sampling

• Each participant has an equal chance of being selected (including all possible combinations).
• A list of all participants is often necessary for simple random sampling, which is not always practically possible.

Simple Random Sampling

• A simple random sample (SRS) requires a list of the entire population.
• Computation and computer methods are necessary for ensuring accurate selection of participants.

Problem

• Systematic differences between groups in a population can lead to biased results. 

Towards Another Sampling Strategy

• If the population has meaningfully distinct subgroups (e.g., clusters), a stratified approach might be desirable.

Stratified Sample Design

• A stratified sample combines multiple samples from subgroups within the population for varied subgroups.

Cluster Sampling

• If an exhaustive list of individuals is not available, but groups (e.g., schools, classes) are, a cluster sampling strategy may be useful.  

Multistage Sampling

• A multistage sampling approach involves selecting samples from successively smaller groups (e.g., stages/levels) within the population.

Systematic Sampling

• A systematic sampling technique involves selecting every nth individual from a list.

Non-random Sampling

• Convenience, volunteer opt-in, quota, purposive, and educated guess methods can be used in non-random sampling situations.

Calculating Sample Size

• Various traditional and power calculation methods for determining sample size based on the population studied

Confidence Interval

• The goal is to determine the confidence interval that accurately describes the value of the parameter for the population from the sample, and to account for variation

Probability Level

• Express the confidence with a specific confidence level (68%, 95%, 99%).

Variance

• The variability in the population for the variable of interest is important for determining sample size. 

Guidelines for Non-Random Samples

• Non-random samples should have a larger size in comparison to the variable being studied.

After Selecting A Non-Random Sample

• The quality of non-random samples needs to be investigated in terms of who is in the survey including descriptive statistics and frequency distributions.

Nonresponse Rates

• Different types of non-response and their rates (e.g., refusal, non-contact, ineligible).
• Measures such as the response rate, cooperation rate, refusal rate, and contact rate are used to interpret the findings of nonrandom samples.

Unit Nonresponse

• Nonresponse in a study can be problematic if it is not random.
• If nonresponse does not have a pattern, then it is less problematic than if it has a pattern/systematic relationship.

Nonresponse Error

• Non-response can bias study findings if it relates to variables of interest.

Examples of NR Error

• Provide real-world examples of how non-response can affect surveys, in terms of variables of interest.

What You Should Mention About Response Rates

• Include specific numbers, such as the number and types of invitations, complete surveys, partial (dropout) surveys, and nonresponses in survey reports.

Sampling Plan

• Important questions for planning out a survey include who the participants are and if there is an available sampling frame, developing the ideal sampling plan, and assessing realistic limitations associated with the sampling plan.

This quiz explores key concepts in survey design and methodology, emphasizing techniques such as Total Quality Design and total survey error. It discusses the importance of sampling frames, random vs. non-random samples, and effective communication strategies for engaging respondents. Gain insights into best practices for conducting successful surveys.