Survey Design and Data Privacy Issues

Introduction to Survey Research

• Survey research has evolved through three eras: Invention, Expansion, and the current era, which incorporates "organic data."
• The Era of Invention (1930-1960) focused on area probability sampling and face-to-face/mail surveys.
• The Era of Expansion (1990-present) saw the rise of non-probability sampling methods like computer-assisted online surveys and the integration of big data sources.
• "Designed data" is supplemented by "organic data"

Non-Probability Sample Surveys

• Online non-probability panels are databases of potential respondents who've stated they'll cooperate for future data collection.
• Quota sampling involves representing specific population characteristics proportionally in the sample.
• River sampling is used to invite website visitors to immediate surveys.
• Recruitment can occur via targeted advertisements on social media platforms like Facebook and Twitter.
• A potential problem with non-probability samples is that some parts of the population may be systematically excluded because participation is voluntary.

Digital Trace Data

• Digital trace data is a kind of data that is collected through various online activities.
• It is collected through sources like social media, web browsing history, applications and geographic location data.
• It can be collected directly from the web through APIs or web scraping.
• Data donation from survey participants, such as web browsing history, specific apps used, and activity data (e.g., Strava), is also a source of digital trace data.
• Designed big data and smart surveys also contribute to digital trace data; collected within or after a survey, through apps, and browsers.
• This data can be obtained by using data download packages (DDPs).

Introducing "Design" to Digital Trace Data

• Example workflow: Data collected with sensors. An initial population register feeds into a sample selection process.
• Respondents participating provide consent, and their data is processed, with sensor data captured.

Privacy Concerns

• Potential risks related to sensor data include interception by unauthorized parties.
• Connecting multiple data streams can potentially re-identify previously anonymous users.
• Information from sensor data can be used to negatively impact an individual's credit, employment, or insurability.
• Higher privacy concerns tend to correlate with decreased willingness to participate in studies.

Hypothetical Willingness from the LISS Panel

• The data from the LISS panel, which covers roughly 2,678 Dutch smartphone users, illustrates that willingness to share sensor data varies across the sharing of locations, videos, photos of a house and photos of yourself.

CBS Consent Survey

• Data on the willingness to share sensor data, from the CBS Consent Survey, for approximately 1,883 Dutch smartphone and tablet users, illustrates that willingness differs among types of data. GPS data has greater willingness than sharing photo of homes and videos.

Examples of New Forms of Data

• Smartphone sensors such as NFC, Bluetooth, thermometers, Wi-Fi, GPS, cellular networks, fingerprint sensors, barometers, accelerometers, pedometers, gyroscopes, and camera are examples.

Examples of Use of Sensor Data

• SurveyMotion: JavaScript-based and total acceleration.
• Completion Behaviour: Fitness Tasks.
• Wearables. Wrist-worn GENEActiv, Axivity ax3 at the upper thigh, total physical activity.

More on Sensors

• Several organizations are using cameras to scan receipts to learn about consumer spending patterns.
• Some work focuses on linking sensor data to administrative data, such as tax records, for research purposes.

Examples of App Data

• The Tabi app with Statistics Netherlands, focuses on travel mode and history.

Some Examples of Studies Using Sensors

• Presented are several examples, in a table format, of specific studies using sensors to investigate areas, such as, social networks, spatial segregation, urbanicity, mobility, and more.

Some Examples of Studies Collecting Biomeasures

• A table summarises studies collecting specific biomeasures, such as blood and saliva, from particular geographical areas, and associated methodologies.

Some Examples of Studies Using Linkage to Administrative Data

• A table summarises examples of research studies using linkages with administrative data.

Questions Concerning Study Conclusions

• Researchers analyzing tweets to gauge climate change worry levels among Germans. Researchers advertise a survey regarding social isolation on Instagram, with the aim of finding out the social isolation levels from this sample.

Social Media Use as a Replacement for Official Statistics

• Initially, social media was a promising alternative to official statistics due to its alignment with existing metrics.
• However, issues emerged, as social media usage patterns changed leading to a degradation of the correlation with traditional metrics. Subsequently, job loss trends in social media data were not aligned to actual official unemployment figures.

Concerns of using Social Media as a basis for study

• Micro-decisions during the analysis may strongly affect the results of the study
• Social media data may not be appropriate for replacing survey data, except under a limited set of highly controlled conditions.

Comparing Two Types of Online Survey Samples

• Opt-in samples, used in online surveys, are significantly less accurate than probability-based panel samples, which use random sampling methods.

Probability vs Non-Probability Sample Surveys

• Probability samples are preferred for generalizable inferences but are often expensive and time-consuming.
• Non-probability samples are usually more affordable, timely, and convenient but might not allow for generalization.
• Diverse methods like surveys, digital traces, and others are included in non-probability samples.

Probability Surveys vs Digital Trace Data

• Designed data is used for research purposes.
• Researchers have control over content and a large number of covariates in probability surveys, while organic data, from digital traces, is collected from different purposes with often little control.

What can we do if none of the sources are perfect?

• Combining data sources might be helpful. Combining the strengths of multiple data sources can improve inference while mitigating biases from any single source.

Two Principles of Data Integration

• Data integration (DI) is context dependent, i.e. it depends on the specific purpose of the data being integrated and the data in question.
• DI, considered as a puzzle, requires aligning data quality, such as timeliness, coverage, quality, and size.

Additional Reading

• This section provides further sources of scholarly material that support the themes and examples discussed in the presentation.