Machine Learning Metrics

What is the primary purpose of evaluation metrics in machine learning?

What type of problem does the choice of evaluation metric depend on?

What is the main benefit of a confusion matrix?

What type of metrics are used for classification problems?

What is the main difference between classification metrics and regression metrics?

What is the purpose of evaluating the performance of a machine learning model?

What is the primary purpose of a Confusion Matrix in binary classification?

What is the formula to calculate the Accuracy of a model?

What is the problem with using Accuracy as a metric in a 2-class problem where one class has a much larger number of instances?

What is the most widely-used metric for evaluating the performance of a machine learning model?

What is the main issue with a model that predicts everything to be Class NO in a 2-class problem?

What is the value of Accuracy for a model that predicts everything to be Class NO in a 2-class problem with 990 Class NO examples and 10 Class YES examples?

Why is detecting the rare class often more interesting in a 2-class problem?

What can be inferred from a Confusion Matrix for a 2-class problem?

Metrics for Performance Evaluation

Evaluation metrics are used to assess how well a machine learning model performs, helping to compare different models for a specific task.
The choice of metric depends on the type of problem and the nature of the data.

Types of Metrics

Classification Metrics: used for binary classification and multi-class classification problems.
Regression Metrics: used for regression problems.
Unsupervised Learning Metrics: used for unsupervised learning problems.

Confusion Matrix

A table layout used to visually summarize the performance of a classification model.
Provides information on correct and incorrect predictions for each class.
Example of a Confusion Matrix for binary classification:
- ACTUAL CLASS
- Yes: TP (True Positives), FN (False Negatives)
- No: FP (False Positives), TN (True Negatives)

Accuracy

The percentage of test set tuples that are correctly classified.
Formula: (TP + TN) / (TP + TN + FP + FN)
Problem with Accuracy: it can be misleading in imbalanced datasets, where a trivial model can achieve high accuracy without detecting the rare class.

Example of a Problem with Accuracy

Consider a 2-class problem with 990 Class NO examples and 10 Class YES examples.
If a model predicts everything to be Class NO, accuracy is 99% (990/1000), but this model does not detect any Class YES examples.
Detecting the rare class is usually more interesting (e.g., frauds, intrusions, defects, etc).