Model Evaluation Metrics Quiz

Model Evaluation Metrics

• Precision: Measures the proportion of accurately predicted positive cases out of all predicted positive cases. Represents the model's ability to correctly identify positive instances.
• Recall (Sensitivity, True Positive Rate): Measures the proportion of correctly predicted positive cases out of all actual positive cases. Represents the model's ability to identify all positive instances.
• Specificity (True Negative Rate): Measures the proportion of correctly predicted negative cases out of all actual negative cases. Represents the model's ability to correctly identify all negative instances.
• F1 Score: A harmonic mean of precision and recall, providing a balanced measure of the model's performance. It emphasizes both correctly identified positive cases and avoiding false positives.

Confusion Matrix & Error Types

• True Positive (TP): Correctly classified as the class of interest
• False Negative (FN): Incorrectly classified as not the class of interest (Type II error)
• False Positive (FP): Incorrectly classified as the class of interest (Type I error)
• True Negative (TN): Correctly classified as not the class of interest
• Type I Error: False Positive - predicting a positive instance when it is actually negative.
• Type II Error: False Negative - predicting a negative instance when it is actually positive.

Accuracy and its Limitations

• Accuracy: Measures the overall proportion of correct predictions made by the model.
• Accuracy Limitations: Accuracy can be misleading when dealing with imbalanced datasets. In cases where one class significantly outweighs the other, a model could achieve high accuracy by simply predicting the majority class, even if it fails to identify a significant portion of the minority class.

ROC Curve

• ROC Curve: (Receiver Operating Characteristic) Plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various threshold settings.
• TPR (Sensitivity): TP / (TP+FN)
• FPR (1-Specificity): FP / (FP+TN)
• AUROC (Area Under the ROC Curve): Represents the model's overall performance across different threshold levels. A larger AUROC value indicates better model performance.

Overfitting and Cross-Validation

• Overfitting: Occurs when a model learns the training data too well, potentially capturing noise and random fluctuations. This leads to poor generalization performance on unseen data.
• Internal Validation: Validates the model on the current dataset using techniques like cross-validation.
• External Validation: Evaluates the model on a completely new dataset to assess its generalization ability.
• Cross-Validation: A technique used to assess the performance of a model on unseen data by repeatedly training and testing the model on different subsets of the data. This helps in:
  • Choosing the best parameter setting for the model
  • Ensuring that the model does not overfit the training data

Types of Cross-Validation

• Holdout Validation: Splits the data into training and testing sets, typically with an 80/20 split. This method is simple and computationally efficient, but can be prone to variance depending on the data split.
• Leave-One-Out Cross Validation (LOOCV): Uses (N-1) samples for training and a single sample for testing, repeating this process for each sample in the dataset. It maximizes data usage, reduces variance, but is computationally expensive, especially for large datasets.
• K-fold Cross Validation: Divides the data into k subsets, trains the model on (k-1) subsets and tests on the remaining subset, repeating this process k times. Offers a balance between computational efficiency and variance reduction, and typically uses k values of 5 or 10.