Survey Methodology Week 6: Weighting and Scaling

Week 6: Weighting and Scaling

• Survey methodology, Block 2, 2024
• Focus on weighting and scaling techniques for survey data
• Covers scales, reliability, missing data mechanisms, and unit/item non-response

Scales and Reliability of Scales

• Many constructs are complex and cannot be measured with a single question
• Several items combined to measure a construct create a scale score
• Summing or averaging items into a scale score results in a new variable
• A scale's level of measurement is an advantage
• Scales are reliable when their items measure the same concept
• Cronbach's alpha and item analysis assess scale reliability (next slide)

Reliability of Scales: Output

• Cronbach's alpha (closer to 1 indicates higher internal consistency)
• Item-total statistics (examines each item in relation to the scale)
• Scale mean if item deleted
• Variance if item deleted
• Corrected item-total correlation
• Cronbach's alpha if item deleted

Introduction to Missingness Mechanisms

• Every data point has a probability of being missing, governed by a mechanism (MCAR, MAR, MNAR)
• Missing Completely at Random (MCAR): Missingness probability is unrelated to other data
• Missing at Random (MAR): Missingness probability relates to observed data
• Missing Not at Random (MNAR): Missingness probability relates to unobserved data

What It Would Look Like

• Example table demonstrating missing data (respondent, response, weight, gender)
• "R," "Y," and "X" variables in a table format (Respondent, Response, Weight, Gender) show a few example cases with missing values.

Missing Completely at Random (MCAR)

• The probability of a data point being missing is the same for all data points
• Missingness is unrelated to the data itself

Missing at Random (MAR)

• The probability of a data point being missing depends on the observed data
• Missingness relates to observable data.

Missing Not at Random (MNAR)

• The probability of a data point being missing depends on unobserved data
• Missingness relates to unobservable data

Random Sampling and Generalization

• Target population → sample (simple random sampling)
• Randomness ensures generalizability to the population, but chance is involved (especially in small samples)
• Check for sample bias by comparing important variables and checking for differences between respondents and non-respondents
• Generalizability can be problematic if not all intended respondents participate

Does Sample Resemble Population? Example

• Target population: all those using a fitness center
• Administration may provide percentages for males, females, students, and staff
• Sample percentages could be compared against population percentages

Testing a Proportion in SPSS

• Data analysis using SPSS to test sample proportions, such as calculating the proportion of females in the sample in statistical software.

Does the Sample Resemble the Population?

• Percentages for different groups (students, staff) in the given sample can be contrasted with the population percentages provided by the administration to assess sample characteristics.
• Chi-square test example of sample data comparisons regarding the different groups

Unit vs Item Non-response

• Generalizability of a survey depends on all intended respondents participating
• Categories of non-response: Not reached, Refused to participate, Did not answer all questions
• Item non-response: Respondents complete most important parts but not all
• Unit non-response: Respondents not contacted or refused to participate (data not collected)

Weighting

• Under- or over-sampling can be corrected through weighting
• Weights are calculated to adjust for population proportions in the sample which were not equally distributed
• A weighting variable is added to adjust for sampling differences

Output Before/After Weighting

• Tables showing frequency distributions of a variable before and after applying weights
• Weights are applied to the sample to make it representative for the target population.

Stratified Sampling and Weighting

• Population and age distribution (examples based on elderly people in a home)
• Random sampling from each age group creates a more representational sample.

Unit Non-response

• Generalizability issues arise if all respondents do not participate
• Non-respondents differ from respondents (e.g. age, gender)

Unit Non-response in SPSS

• Create data files for respondents, analyze their data compared to non-respondents.
• Use SPSS to analyze differences between respondents, non-respondents (Categorical & Numerical)

Comparing Distributions

• Charts comparing different groups frequency distributions (e.g., gender and responses) in SPSS
• Chi-square and other statistical tests are used to analyze whether there is a significant relationship between the categorical variables

Comparing Averages

• Independent samples t-test used to compare average ages of respondents versus non-respondents in SPSS
• Determine whether differences exist between these groups (using t-tests or cross-tabs).

More on Weighting

• Weighting is essential in complex survey designs like clustering, stratification, etc
• Understanding weighting necessitates advanced courses

Weighting Summarized

• Several issues can cause a sample to not accurately represent the population (stratified sampling, failed randomization, non-random non-response)
• A weighted sample accounts for these factors by applying appropriate weights

Next Topic: Imputation; Short Break

• Upcoming topic: imputing missing data; a break is suggested

Item Missings

• Example table shows "Respondent," "R," "Y(Grade)," and "X(Hours)"
• Data points are highlighted if they have missing values (e.g., "MISSING").

MCAR, MAR, MNAR and Imputation

• If data missing completely at random (MCAR) or missing at random (MAR), missing data mechanisms can be ignored using multiple imputation and maximum likelihood procedures
• If missing not at random (MNAR), missing mechanisms should not be ignored

Regression Imputation

• Regression imputation estimates missing values using the best possible value from a statistical model
• The imputed value minimizes potential errors; True grade is uncertain; predictions do not fully account for the uncertainty of the imputed value

Some Theory

• Sample mean for incomplete data may differ, but the complete data is still representative
• Multiple imputation handles sample uncertainty

Regression Multiple Imputation

• Using regression to impute data values

Stochastics Regression Imputation

• Regression imputation that explicitly accounts for the uncertainty of the missing data.

SPSS - Analyze Pattern

• Overview of a statistical software package’s functions.
• Example summaries for missing values and patterns

SPSS - Analyze Pattern

• Visual display of missing data in SPSS
• Charts showing the percentage of complete and incomplete data sets
• Summary of missing values and patterns

SPSS

• SPSS functionality to impute missing data values
• Various methods available, including selecting an imputation method, specifying the number of imputations (e.g., 5), and determining the output location.

Multiple Imputation (Predictive Mean Matching) - Descriptives

• Example outputs showing results of data imputation.
• Mean, standard deviation, minimum, maximum, calculated for the imputed and original data sets, and after imputation.

Multiple Imputation (PMM) - Pooled Regression

• Output showing estimates of coefficients in a regression model
• Various methods of imputation and parameters for each such as Imputation Method (automatic or custom selected), number of imputations and Maximum iterations.

Pooled – Why?

• Multiple imputations produce different datasets that need to be combined
• Combines between and within imputation variance

Missing data mechanisms

• Diagrams (MCAR, MAR, MNAR) visually represent the relationships between variables and missing data

This quiz focuses on Week 6 of Survey Methodology, specifically exploring weighting and scaling techniques for analyzing survey data. Key topics include scale construction, reliability assessments like Cronbach's alpha, and approaches to handling missing data. Test your understanding of these vital components in survey research.