PPDAC Framework and Study Errors

PPDAC

• PPDAC is a framework for conducting empirical studies, its steps are Problem, Plan, Data, Analysis, Conclusion.
• Problem: A clear statement of the study's objectives.
• Plan: The procedures used to carry out the study including how the data will be collected.
• Data: The physical collection of the data, as described in the Plan.
• Analysis: The analysis of the data collected, accounting for considerations in the Problem and the Plan.
• Conclusion: The conclusions that are drawn about the Problem, and any limitations of the study.

Three Groups of Units

• Target population: The population we want to study.
• Study population: The population we could study.
• Sample: The population we actually study.

Three Types of Error

• Study error: Any difference between the target population and the study population.
• Sample error: Any difference between the study population and the sample.
• Measurement error: Any difference between the measured value and the true value of a variate.

Data Step

• The Data step follows the Plan.
• Variates must be clearly defined and satisfactory methods of measuring them must be used.
• Mistakes can occur in recording or entering data into a database.
• It is useful to put checks in place to avoid these mistakes, and detect those that are made.

Sample Error

• Sample error occurs if the attributes in the sample differ from the attributes in the study population.
• Sample error is not guaranteed just because the study population and sample are different.
• There must be a difference in attributes between the groups.

Measurement Error

• If the measured value and the true value of a variate are not identical, the difference is called measurement error.
• Measurement error is extremely common and can severely undermine results.
• The impact of measurement error can be hard to anticipate.
• The relationship between human papillomavirus (HPV) and cervical cancer was underestimated due to measurement error in the identification of HPV infection.

Example: AI Study

• Target population: Students enrolled in the UCAS system in November 2023, who were given the option to participate in the survey.
• Study population: Students enrolled in the UCAS system in November 2023.
• Sample: The 1,250 students who completed a survey.

Example: AI Study - Measurement Error

• In the AI study, participants may not tell the truth about their use of AI tools in university work, or give different opinions to what they truly believe about whether the use of such tools is acceptable.

AI Study - Concerns

• The sample consists of people who (presumably) completed the survey voluntarily.
• This raises concerns because students who use AI tools might be more likely to participate in a survey about AI than those who have not used AI tools.
• The sample might overestimate the proportion of students in the study population who have used AI tools in their studies.
• There are other possible sources of sample error.

Key Points

• Concerns about sample error and measurement error should be discussed.
• Statistical models are used to quantify the size of sample error.
• Accuracy of measurements is crucial.
• Variates corresponding to attributes of interest are measured.
• Other variates that can aid the analysis may be measured.