Decision Tree Hyperparameter Tuning Quiz

Decision Tree Basics

• A decision tree is composed of nodes, which are chosen to split features optimally.
• The tree stops growing when the maximum depth is reached, which is set by the max_depth hyperparameter.

Hyperparameter Tuning

• max_depth is a hyperparameter that controls the depth of the decision tree.
• Increasing max_depth allows the tree to add more decision boundaries and improve its accuracy.

Gini Impurity vs. Entropy

• Gini impurity measures the frequency of mislabels when randomly labeling a dataset.
• Entropy measures the disorder of features with the target.
• Gini impurity is faster and less computationally expensive than entropy.

Gini Impurity

• A node's Gini attribute measures its impurity, with a "pure" node having a Gini score of 0.
• The Gini score can be calculated using the formula: 1 - (p1^2) - (p2^2) - ... - (pk^2), where p1, p2, ..., pk are the proportions of each class in the node.

White Box vs. Black Box Models

• Decision trees are white box models, meaning their decisions are easy to interpret.
• Models like SVM, RF, and neural networks are black box models, meaning their decisions are hard to interpret.

Estimating Class Probabilities

• A decision tree can estimate the probability that an instance belongs to a particular class.
• The estimated probabilities can be identical for some instances.

CART Training Algorithm

• The CART algorithm is used to train decision trees in Scikit-Learn.
• The algorithm recursively splits the training set into subsets using the feature and threshold that produces the purest subsets.
• The cost function that the algorithm tries to minimize is the impurity of the subsets.
• The algorithm stops recursing when it reaches the maximum depth or cannot find a split that reduces impurity.