Gradient Boosting Overview Quiz

Gradient Boosting

• Gradient boosting is a powerful ensemble technique in machine learning.
• Unlike traditional models that learn independently, boosting combines predictions from multiple weak learners to create a single more accurate and strong learner.
• A weak learner is a machine learning model that performs better than random guessing.
• A decision tree is a popular weak learner.
• Gradient boosting has become widely used in machine learning applications, including customer churn prediction, asteroid detection, and recommendation systems (like Netflix).
• Gradient boosting is successful in Kaggle competitions.

Gradient Boosting Algorithm

• Input: Tabular data with features (X) and a target (y).
• Aim: Learn from the training data to generalize well to unseen data.
• Example: Using customer age, purchase category, purchase weight to predict purchase amount.

Loss Function

• Loss function in machine learning quantifies the difference between predicted and actual values, measuring model performance.

• It calculates errors by comparing predicted output with ground truth values.

• Evaluation metric comparison of loss on different datasets (training, validation, and testing) for model generalization assessment.

• Mean Squared Error (MSE): A common regression loss function measuring the sum of squared differences between actual and expected values.

• Gradient boosting often uses a variation of MSE for more accurate evaluation.

• Cross Entropy: A common loss function for classification models using the difference between probability distributions, where targets have discrete categories.

Step 1: Initial Prediction

• The initial prediction/guess is the average of the target variable.
• For e.g., average of the target variable (purchase amount) is used as initial prediction.

Step 2: Pseudo-residuals

• Calculate the difference between observed values and the initial prediction.
• E.g., 156 (initial prediction) - Observed Value = Pseudo-residuals.

Step 3: Build a Weak Learner

• Construct a decision tree using features (e.g., age, category, purchase weight) to predict the residuals.

Step 4: Iterate

• Iterate on Step 3 to build more weak learners.

Hyperparameter Tuning

• Controls the algorithm's direction and loss function.
• Mean Squared Error (MSE) for regression; Cross-Entropy for classification.

Learning Rate

• Controls the contribution of each weak learner (shrinkage factor).
• Smaller values decrease the contribution of each weak learner.
• But leads to more computing time.

Number of Trees

• Controls the number of weak learners to be built.
• Higher trees tend towards being more complex, allowing for capturing more patterns in the data.

Max Depth

• Controls the number of levels in each weak learner (decision tree).
• A deeper decision tree (more levels) leads to more complex and computationally expensive models.
• Choose a value close to 3 to avoid overfitting.

Minimum Number of Samples Per Leaf

• Controls how branches split in decision trees.
• Setting a low value for the number of samples makes the algorithm noise-sensitive, and avoiding a large value helps prevents overfitting.

Subsampling Rate

• Controls the proportion of the data used to train each weak learner (decision tree).

Feature Sampling Rate

• Samples rows and features.

• For datasets with hundreds of features, it's recommended to select a feature sampling rate between 0.5 and 1 to reduce the risk of overfitting.

Cluster Analysis

• Segment customers based on demographic variables.

Unsupervised Classification

• Groups data based on similarities in input values.

K-Means Algorithm

• Data points input and number of clusters.
• K-means groups these into specified number of clusters.

Hierarchical Clustering

• Goal: To build a hierarchy over the data points.

• Agglomerative: Starts with each data point as a separate cluster, then groups closest clusters.

• Divisive: Starts with a single cluster of all data points, then splits into smaller clusters.