Machine Learning: Gradient Boosting Techniques

Gradient Boosting

• Gradient boosting is a powerful ensemble technique in machine learning
• It combines predictions from multiple weak learners to create a stronger, more accurate model.
• Unlike traditional models that learn independently, boosting models work together.

Weak Learner

• A weak learner is any machine learning model that performs better than random guessing.
• A simple example would be a decision tree.

Real World Applications

• Gradient boosting is used in various industries:
  • Predicting customer churn
  • Detecting asteroids
  • Building recommendation systems (e.g., Netflix)
• It is used in various areas including retail, finance, healthcare, and advertising.

The Gradient Boosting Algorithm (Step-by-Step)

• Input: Tabular data with features (X) and a target variable (y).

• The algorithm learns from the training data to generalize to unseen data.

• An example sales dataset is used to understand gradient boosting:

  Age	Category	Purchase Weight (kg)	Amount ($USD)
  25	Electronics	2.5	123.45
  34	Clothing	1.3	56.78
  42	Electronics	5.0	345.67
  19	Homeware	3.2	98.01

• The goal is to predict the purchase amount.

The Loss Function in Gradient Boosting

• A loss function measures the difference between predicted and actual values.

• It quantifies how well a machine learning model is performing.

• It calculates errors by comparing predicted output to the ground truth (observed values).

• Using the evaluation metric, model performance is assessed by comparing loss on training, validation, and test datasets. This helps avoid overfitting.

• Common Loss Functions:

  • Mean Squared Error (MSE): Measures the sum of squared differences between predicted and actual values.
  • Gradient boosting often uses a variation of MSE.
  • Cross-Entropy: Measures the difference between two probability distributions. Commonly used in classification where targets have discrete categories.

Step 1: Make an Initial Prediction

• Start with an initial prediction, often the average of the target variable's values in the training set.
• For the example data, the initial prediction is $156.

Step 2: Calculate the Pseudo-Residuals

• Calculate the differences between each observed value and the initial prediction.
• These differences are called pseudo-residuals.

Step 3: Build a Weak Learner

• Build a decision tree (weak learner) to predict the residuals using features like age, category, purchase weight.
• Use a simplified decision tree with a few terminal nodes for this example.

Step 4: Iterate

• Repeat steps 2 and 3 multiple times to build more weak learners.
• Each iteration refines the model's accuracy.

Hyperparameter Tuning

• Parameters affecting the direction and loss function of the algorithm.
• For regression, Mean Squared Error (MSE) is often used.
• For classification, Cross-Entropy might be used.

Learning Rate

• This hyperparameter controls the contribution of each weak learner (decision tree).
• Smaller values (closer to 0) reduce the influence of individual weak learners.
• This often requires more training data and iterations.

Number of Trees

• This hyperparameter defines the number of weak learners in the ensemble.
• More trees generally lead to a stronger model but also potentially higher complexity and overfitting.

Max Depth

• Controls the tree's depth.
• Choosing values close to 3 helps prevent overfitting. Higher max depths are more complex.

Minimum Number of Samples per Leaf

• Determines the minimum number of samples required for a terminal node in a decision tree.
• A lower value can make the algorithm sensitive to noise.

Subsampling Rate

• Controls the proportion of data used to train each weak learner.
• This can influence training speed and overfitting tendencies (lower rates might be faster but could lead to overfitting).

Feature Sampling Rate

• Controls the proportion of features used to train each tree.
• Recommended for datasets with many features.
• Values from 0.5 to 1 can limit overfitting.

Cluster Analysis

• Grouping similar data points in large datasets.

Why Segmentation?

• Methods such as clustering are used to create segments of customers based on data such as demographics.

Unsupervised Classification

• Categorization based on similarities in input values without pre-defined categories.

What is Clustering?

• Grouping data into clusters based on similarities.

K-Means Algorithm

• Initializes 'K' random cluster centers.
• Assigns each data point to the closest cluster center.
• Updates cluster centers via the mean/average of assigned points.
• Repeats these steps until convergence, ensuring clusters don't change significantly.

Hierarchical Clustering

• Builds a hierarchy of clusters based on a proximity measure.
• Can be agglomerative or divisive in approach.
• Agglomerative starts with individual data points as clusters.
• Divisive starts with all data points in the single cluster.

Agglomerative Clustering

• Starts with each data point as a cluster.
• Repeatedly merges the closest clusters.
• This continues until the desired number of clusters is reached.