9 Questions
What is the main idea behind re-analyzing the Bem data with Bayesian methods?
To capture the idea that 'extraordinary claims require extraordinary evidence'
What is the result of having too good a fit to the data?
Overfitting
What is the consequence of adding complexity to a model?
Poorer generalization to new data
What is the purpose of cross-validation?
To evaluate the performance of a model on new data
Why is overfitting a problem?
Because it leads to poorer generalization to new data
What is the consequence of using a model that is overly complex?
Poorer prediction on new data
What is the relationship between model complexity and generalization?
Increased complexity leads to poorer generalization
What is the main issue with using a model that fits the data perfectly?
It is overfitting the data
Why is it important to evaluate a model's performance on new data?
To ensure the model generalizes well to new data
Study Notes
Cross Validation
- A safeguard against overfitting, a technique to evaluate model performance
- Leaves one out cross validation (LOOCV) is the most common method
- In LOOCV, each subject/data point is left out (one at a time) and the process is repeated to evaluate performance of the model on the predicted data
Downsides of Cross Validation
- Time-intensive, requiring fitting a large number of models to the data
- Not easy to perform in SPSS, but can be done using specialized packages in R, MATLAB, or Python
Overfitting
- When a model is too complex and fits the data perfectly, it may not generalize well to new data
- Added complexity can result in poorer generalization to new data, making it unable to generalize to new samples, paradigms, or to the population at large
Model Comparison
- Comparing models by fitting a subset of data (training data) and evaluating performance on the remaining subset (validation data)
- The model that performs better on the validation data should be preferred
- Simple models can be preferred over complex models if they perform similarly or better on the validation data
Bayesian Methods
- Easier to implement, especially with software like JASP
- Can conduct Bayesian equivalents of ANOVAs, t-tests, regressions
- Recommended paper: Etz, A., & Vandekerckhove, J. (2018). Introduction to Bayesian inference for psychology. Psychonomic Bulletin & Review, 25, 5-34.
Learn about cross validation, a technique to evaluate model performance and prevent overfitting, including its downsides and challenges in implementation.
Make Your Own Quizzes and Flashcards
Convert your notes into interactive study material.
Get started for free
