Model Evaluation in Data Science

Study Notes

Evaluating model performance in data science is important to avoid overfitting.
There are two methods of evaluating models in data science: hold-out and cross-validation.
In k-fold cross-validation, we divide the data into k subsets of equal size.
We build models k times, each time leaving out one of the subsets from training and use it as the test set.
Training errors: Errors committed on the training set
Test errors: Errors committed on the test set
Generalization errors: Expected error of a model over random selection of records from same distribution
Underfitting: when model is too simple, both training and test errors are large
Overfitting: when model is too complex, training error is small but test error is large
Reasons for model overfitting: not enough training data, over training (train NN with large number of iterations).
Model selection is performed during model building.
The purpose of model selection is to ensure that the model is not overly complex and that generalization error is estimated.
Two types of evaluation are used in data science: classification evaluation and regression evaluation.
Classification evaluation uses a confusion matrix to measure the number of correct and incorrect predictions made by the classification model.
Regression evaluation uses performance metrics such as accuracy and precision.
The key factors to consider when evaluating a regression model are its sensitivity, specificity, and RMSE.
The RMSE is a popular measure of error for regression models, but it can only be compared between models whose errors are measured in the same units.
The RSE can be compared between models whose errors are measured in the different units, while the MAE is usually similar in magnitude to the RMSE but smaller.