Decision Trees in Machine Learning

Study Notes

A type of supervised learning algorithm that uses a tree-like model to classify data or predict continuous values.
Decision trees are composed of nodes, which represent features or attributes, and edges, which represent the decision-making process.
The tree is constructed by recursively partitioning the data into smaller subsets based on the values of the input features.
Decision trees can be used for both classification and regression tasks.

A type of supervised learning problem where the goal is to predict a categorical label or class.
In a decision tree, classification is performed by traversing the tree from the root node to a leaf node, where the predicted class is determined.
Classification metrics:
- Accuracy: proportion of correctly classified instances
- Precision: proportion of true positives among all positive predictions
- Recall: proportion of true positives among all actual positive instances
- F1-score: harmonic mean of precision and recall

A type of supervised learning problem where the goal is to predict a continuous value or range.
In a decision tree, regression is performed by traversing the tree from the root node to a leaf node, where the predicted value is determined.
Regression metrics:
- Mean Squared Error (MSE): average squared difference between predicted and actual values
- Mean Absolute Error (MAE): average absolute difference between predicted and actual values
- R-squared: proportion of variance in the dependent variable that is predictable from the independent variable(s)

Techniques that combine the predictions of multiple base models to produce a more accurate and robust prediction model.
Ensemble methods can be used to improve the performance of decision trees by:
- Reducing overfitting
- Increasing accuracy
- Improving robustness to outliers and noise
Common ensemble methods:
- Bagging (Bootstrap Aggregating)
- Boosting (Gradient Boosting)
- Random Forest

A measure of the contribution of each feature to the prediction model.
In decision trees, feature importance can be calculated using various methods, such as:
- Permutation importance: measures the decrease in model performance when a feature is randomly permuted
- Gini importance: measures the decrease in node impurity when a feature is used to split the data
- Recursive feature elimination: recursively eliminates the least important features until a specified number of features is reached
Feature importance can be used for:
- Feature selection: selecting the most important features for the model
- Model interpretation: understanding the relationships between features and the target variable

A type of supervised learning algorithm that uses a tree-like model to classify data or predict continuous values
Composed of nodes representing features/attributes and edges representing the decision-making process
The tree is constructed by recursively partitioning the data into smaller subsets based on input feature values
Can be used for both classification and regression tasks

A type of supervised learning problem where the goal is to predict a categorical label or class
In decision trees, classification is performed by traversing the tree from the root node to a leaf node, determining the predicted class
Classification metrics include:
- Accuracy: proportion of correctly classified instances
- Precision: proportion of true positives among all positive predictions
- Recall: proportion of true positives among all actual positive instances
- F1-score: harmonic mean of precision and recall

A type of supervised learning problem where the goal is to predict a continuous value or range
In decision trees, regression is performed by traversing the tree from the root node to a leaf node, determining the predicted value
Regression metrics include:
- Mean Squared Error (MSE): average squared difference between predicted and actual values
- Mean Absolute Error (MAE): average absolute difference between predicted and actual values
- R-squared: proportion of variance in the dependent variable predictable from independent variable(s)

Techniques combining multiple base model predictions to produce a more accurate and robust prediction model
Can improve decision tree performance by reducing overfitting, increasing accuracy, and improving robustness to outliers/noise
Common ensemble methods include:
- Bagging (Bootstrap Aggregating)
- Boosting (Gradient Boosting)
- Random Forest

A measure of the contribution of each feature to the prediction model
Can be calculated using methods such as:
- Permutation importance: measures decrease in model performance when a feature is randomly permuted
- Gini importance: measures decrease in node impurity when a feature is used to split the data
- Recursive feature elimination: recursively eliminates least important features until a specified number is reached
Feature importance is used for:
- Feature selection: selecting the most important features for the model
- Model interpretation: understanding relationships between features and the target variable