Hyper-parameter Tuning in Machine Learning

Lecture 25: The ART of Choosing Hyper-parameters

• The lecture is about choosing hyper-parameters for machine learning models.
• Key topics include what hyper-parameters are, why choosing the right ones matters.
• Different methods: search, grid search, random search and Bayesian optimization.
• Advantages and disadvantages of each method.

What are Hyper-parameters?

• Hyper-parameters are set before learning begins (not learned from the data).
• Examples: learning rate in gradient descent, number of hidden layers in a neural network, and regularization parameters in support vector machines.

Why Choose the Right Hyper-parameters?

• Hyper-parameter choice significantly impacts model performance.
• Poor hyper-parameters lead to improper learning or overfitting.

Methods for Choosing Hyper-parameters

• Common methods include manual search, grid search, random search, and Bayesian optimization.

Manual Search

• Simplest method: manually try different hyper-parameter values.
• Evaluate the model's performance on a validation set.
• Effective for small search spaces but can be time-consuming.

Grid Search

• More systematic method: create a grid of possible hyper-parameter values.
• Evaluate model performance for each value combination.
• More efficient than manual search but time-consuming for large search spaces.

Random Search

• More efficient than grid search: randomly select hyper-parameter values.
• Evaluate model performance for each combination.
• Efficient as it avoids evaluating unlikely optimal combinations.

Bayesian Optimization

• Sophisticated method: uses a probabilistic model to estimate performance for different value combinations.
• Uses this model to select the next combination to evaluate.
• More efficient than grid and random search but is more complex to implement.

Advantages and Disadvantages of Each Method

Conclusion

• Choosing the right hyper-parameters is vital in machine learning.
• Methods vary in advantages/disadvantages, and the best method depends on the problem and resources.
• **

Improving Hyperparameter Selection (ART)

• Hyperparameter optimization aims to maximize performance on the dev set.
• Hyperparameters are like "knobs" that tune the learning algorithm.

Hyperparameter Tuning

• Use random values (Coarse -> Fine) if you have plentiful computational resources.
• Intuition comes from the fact random search will find the most important hyperparameter.
• Coarse to Fine involves zooming in on the best region after the first pass.