Overfitting in Loglinear Models

Overfitting

• Overfitting occurs when a model fits the data perfectly but will not generalize well to new data.
• Adding complexity to a model should be justified by an improvement in goodness of fit.

Cross Validation

• Cross validation is a technique to evaluate a model's performance on unseen data.
• It involves fitting a model to a subset of the data (training data) and evaluating its performance on the remaining subset (validation data).
• The model that performs better on the validation data is preferred, as it exhibits better generalization to new data.
• Cross validation is an effective way to compare models and prevent overfitting.

Comparing Models

• Cross validation is useful when comparing different models, such as models with different numbers of predictors or interaction terms.
• It helps to determine which model is preferred based on its performance on validation data.

Leave One Out Cross Validation (LOOCV)

• LOOCV is a common method of cross validation, where each data point is left out in turn and the model is evaluated on the remaining data.
• The process is repeated for each data point, and the performance of the model is averaged across all iterations.

Downsides of Cross Validation

• Cross validation can be time-intensive, as it requires fitting multiple models to the data.
• It is not easy to perform in SPSS, but specialized packages are available in R, MATLAB, and Python.

Regularization

• Regularization is a technique to reduce the complexity of a regression model.
• The most common technique is lasso regression, which adds a penalty term to the error term to discourage large coefficients.
• Regularization pushes estimates of small or weak predictors to zero, resulting in a simpler model.

Lasso Regression

• Lasso regression requires specifying the regularization term, which can be difficult to specify in some cases.
• If the regularization is too strong, all coefficients are pushed to zero.

Advantages of Regularization

• Regularization naturally produces a simpler model with fewer significant predictors.
• It can be used to prevent predictors from getting non-zero estimates even if they are not contributing to the model.

Cross Validation

• A safeguard against overfitting, a technique to evaluate model performance
• Leaves one out cross validation (LOOCV) is the most common method
• In LOOCV, each subject/data point is left out (one at a time) and the process is repeated to evaluate performance of the model on the predicted data

Downsides of Cross Validation

• Time-intensive, requiring fitting a large number of models to the data
• Not easy to perform in SPSS, but can be done using specialized packages in R, MATLAB, or Python

Overfitting

• When a model is too complex and fits the data perfectly, it may not generalize well to new data
• Added complexity can result in poorer generalization to new data, making it unable to generalize to new samples, paradigms, or to the population at large

Model Comparison

• Comparing models by fitting a subset of data (training data) and evaluating performance on the remaining subset (validation data)
• The model that performs better on the validation data should be preferred
• Simple models can be preferred over complex models if they perform similarly or better on the validation data

Bayesian Methods

• Easier to implement, especially with software like JASP
• Can conduct Bayesian equivalents of ANOVAs, t-tests, regressions
• Recommended paper: Etz, A., & Vandekerckhove, J. (2018). Introduction to Bayesian inference for psychology. Psychonomic Bulletin & Review, 25, 5-34.