Regression Model Performance Metrics

Questions and Answers

What does it mean when a model has low bias and high variance?

The model is flexible and captures training data complexity well. (correct)

The model is too simple and does not generalize well.

The model is simple and stable.

The model is too complex and lacks adaptation to unseen data.

What is underfitting in machine learning?

Poor fitting of the hypothesis function to the trend of the data.

Underfitting is often caused by a hypothesis function that is too _____ or uses too few features.

simple

Match the following solutions with their respective fitting issue in machine learning:

Simplify the model = Overfitting Increase the complexity of the model = Underfitting Signup and view all the answers

Which of the following are error metrics used for evaluating regression models?

Mean Absolute Error (MAE) Signup and view all the answers

Regularization techniques are useful to address underfitting in machine learning.

True Signup and view all the answers

What is the formula for Mean Squared Error (MSE) in regression?

MSE = The average of the squared differences between predicted and actual values Signup and view all the answers

R-squared (R²) ranges from 0 to 1.

True Signup and view all the answers

Precision is calculated as (Of all patients where we predicted ______, what fraction actually has cancer?)

cancer Signup and view all the answers

Match the metrics used for classification performance evaluation:

Confusion Matrix = Shows true positives, true negatives, false positives, and false negatives Precision = Positive Predictive Value (PPV) Recall = Measures true positive predictions among all actual positives F1-score = Balance between precision and recall ROC-AUC = Performance measurement for classification at different threshold settings Signup and view all the answers

Study Notes

Performance Measurement in Regression

Regression model performance is measured using error metrics and goodness-of-fit metrics.
Error metrics include:
Mean Absolute Error (MAE): average of the absolute differences between predicted and actual values.
Mean Squared Error (MSE): average of the squared differences between predicted and actual values.
Mean Absolute Percentage Error (MAPE): average of the absolute percentage differences between predicted and actual values.
Goodness-of-fit metrics include:
R-squared (R²): proportion of the variance in the dependent variable that is predictable from the independent variables.

Classification

Classification accuracy is the number of correct predictions made divided by the total number of predictions made, multiplied by 100.
Classification accuracy alone is not enough to determine whether a model is good enough to solve a problem.
Accuracy paradox: a model with high accuracy may not provide valuable or meaningful predictions, especially in imbalanced datasets.
Other metrics used to evaluate classification models include:
Confusion Matrix: a table showing the performance of the classification model, including true positives, true negatives, false positives, and false negatives.
Precision: proportion of true positive predictions among all positive predictions made.
Recall: proportion of true positive predictions among all actual positive instances.
F1-score: conveys the balance between precision and recall.

Trading off Precision and Recall

Precision and recall can be traded off by adjusting the threshold value.
Increasing the threshold value increases precision but decreases recall.
Decreasing the threshold value increases recall but decreases precision.

Averaging Precision and Recall

Averaging precision and recall using the average of the two values is not sufficient.
F1-score is a better way to convey the balance between precision and recall.

ROC-AUC

ROC-AUC is a performance measurement for classification problems at various threshold settings.
ROC curve is a graphical representation of a classifier's performance.
AUC is a single scalar value that summarizes the performance of the classifier across all threshold values.

Overfitting and Underfitting

Overfitting: when a model learns not only the underlying pattern in the training data but also the noise and outliers.
Underfitting: when a model fails to capture the underlying pattern in the data.
Characteristics of overfitting:
High accuracy on training data.
Low accuracy on validation/test data.
Model is too complex.
Low bias and high variance.
Characteristics of underfitting:
Low accuracy on both training and validation/test data.
Model is too simple.
High bias and low variance.