Cross Validation in Machine Learning

Cross Validation

• A safeguard against overfitting, a technique to evaluate model performance
• Leaves one out cross validation (LOOCV) is the most common method
• In LOOCV, each subject/data point is left out (one at a time) and the process is repeated to evaluate performance of the model on the predicted data

Downsides of Cross Validation

• Time-intensive, requiring fitting a large number of models to the data
• Not easy to perform in SPSS, but can be done using specialized packages in R, MATLAB, or Python

Overfitting

• When a model is too complex and fits the data perfectly, it may not generalize well to new data
• Added complexity can result in poorer generalization to new data, making it unable to generalize to new samples, paradigms, or to the population at large

Model Comparison

• Comparing models by fitting a subset of data (training data) and evaluating performance on the remaining subset (validation data)
• The model that performs better on the validation data should be preferred
• Simple models can be preferred over complex models if they perform similarly or better on the validation data

Bayesian Methods

• Easier to implement, especially with software like JASP
• Can conduct Bayesian equivalents of ANOVAs, t-tests, regressions
• Recommended paper: Etz, A., & Vandekerckhove, J. (2018). Introduction to Bayesian inference for psychology. Psychonomic Bulletin & Review, 25, 5-34.

Learn about cross validation, a technique to evaluate model performance and prevent overfitting, including its downsides and challenges in implementation.