Learning with Imbalanced Data: Cost-Sensitive Learning

Cost-Sensitive Learning

• Cost-sensitive learning is a sub-field of Machine Learning that accounts for different costs (e.g., cost of predictive error) into training the model.
• Goal of cost-sensitive learning is to minimize the cost of a model on a training dataset.
• Cost-sensitive learning and learning from imbalanced dataset are not the same but have considerable overlap.

Costs in Machine Learning

• According to Peter Turney, there are nine types of costs in ML, but we are dealing with only one in imbalanced learning.
• Specificity is 1- FPR, making it an unbiased evaluation metric for imbalanced classification.

Youden's J Statistic

• Youden's J statistic is used to optimize the threshold for classification.
• J = Sensitivity + Specificity - 1
• This is the threshold corresponding to argmax(tpr-fpr).

Moving Probability Threshold using PR Curve

• PR curve is a plot of precision and recall at different thresholds.
• Precision is Positive Predictive Power (True Positives / (True Positives + False Positives)).
• Recall or sensitivity is True Positives / (True Positives + False Negatives).
• A model with perfect skill is depicted as a point at (1,1).

Weighted XGBoost for Imbalanced Data

• The scale_pos_weight value is used to scale the gradient for the positive class.
• This scales the model's errors made during training on the positive class and encourages the model to overcorrect them.
• For an imbalance of 1:100, this can be set to 100.

Weighted Random Forest for Imbalanced Classification

• Random Forest is similar to bagging but has a slight difference (bootstrap samples).
• Decision Tree typically uses a modified purity calculation algorithm to reflect class weighting.
• This favors a mixture that favors the minority class and tolerates false positives for the majority class.

Weighted Random Forest with SkLearn and ImbLearn

• SkLearn RandomForestClassifier with class-Weight class_weight argument takes a dictionary for 0 and 1 labels.
• If balanced, it picks up the inverse ratio from training data.
• With bootstrap class_weight, class weight is set at the bootstrap sample level.
• Imblearn BalancedRandomForestClassifier provides random undersampling.

Ensemble with Adaboost

• Imbalanced_learn library provides EasyEnsembleClassifier.
• Select all examples from the minority class and the subset from the majority class.
• AdaBoost is a sequence of boosted decision trees.
• It works by first fitting a decision tree on the dataset, then determining the errors made by the tree and weighing the dataset's examples by those errors.

One Class Classifier and Overall Steps

• One class classifier is an ML approach to detect anomalies.
• These algorithms are trained on majority inliers or normal data.
• The trained models are used to classify new samples as outlier (positive) or normal (negative).

Downside of using OCC for Imbalanced Classification

• The positive samples (however small in number) are NOT used in training at all.
• The advantage of this technique comes at a price.

This quiz covers the concept of cost-sensitive learning in machine learning, a sub-field that accounts for different costs in predicting incorrect outcomes. It explores the techniques used in cost-sensitive decision trees and their applications in imbalanced datasets. Test your knowledge of this important concept in AI and ML.