Regression Model Performance Metrics

Study Notes

Regression model performance is measured using error metrics and goodness-of-fit metrics.
Error metrics include:
Mean Absolute Error (MAE): average of the absolute differences between predicted and actual values.
Mean Squared Error (MSE): average of the squared differences between predicted and actual values.
Mean Absolute Percentage Error (MAPE): average of the absolute percentage differences between predicted and actual values.
Goodness-of-fit metrics include:
R-squared (R²): proportion of the variance in the dependent variable that is predictable from the independent variables.

Classification accuracy is the number of correct predictions made divided by the total number of predictions made, multiplied by 100.
Classification accuracy alone is not enough to determine whether a model is good enough to solve a problem.
Accuracy paradox: a model with high accuracy may not provide valuable or meaningful predictions, especially in imbalanced datasets.
Other metrics used to evaluate classification models include:
Confusion Matrix: a table showing the performance of the classification model, including true positives, true negatives, false positives, and false negatives.
Precision: proportion of true positive predictions among all positive predictions made.
Recall: proportion of true positive predictions among all actual positive instances.
F1-score: conveys the balance between precision and recall.

Averaging precision and recall using the average of the two values is not sufficient.
F1-score is a better way to convey the balance between precision and recall.

ROC-AUC is a performance measurement for classification problems at various threshold settings.
ROC curve is a graphical representation of a classifier's performance.
AUC is a single scalar value that summarizes the performance of the classifier across all threshold values.

Overfitting: when a model learns not only the underlying pattern in the training data but also the noise and outliers.
Underfitting: when a model fails to capture the underlying pattern in the data.
Characteristics of overfitting:
High accuracy on training data.
Low accuracy on validation/test data.
Model is too complex.
Low bias and high variance.
Characteristics of underfitting:
Low accuracy on both training and validation/test data.
Model is too simple.
High bias and low variance.