Hyperparameter Tuning Overview

Hyperparameter Tuning

• Hyperparameters define model structure and training process
• Set before training begins
• Examples include learning rate, batch size, epochs, and regularization

Hyperparameter Details

• Learning Rate: Speed of model data incorporation.
  • Higher rate = faster convergence but risk of overshooting optimal solution
  • Lower rate = more precise convergence but slower
• Batch Size: Number of data points considered at a time.
  • Smaller batch size = more stable learning but slower computation
  • Larger batch size = faster computation but potentially less stable updates.
• Number of Epochs: Number of iterations over the entire training dataset.
  • Too few epochs = underfitting
  • Too many epochs = overfitting (model learns noise in data)
• Regularization: Balances model simplicity and complexity.
  • Increasing regularization reduces overfitting.

Hyperparameter Tuning Methods

• Grid Search, Random Search
• Automatic Model Tuning (AMT) services (e.g., SageMaker)

Overfitting

• Model performs well on training data but poorly on new data.
• Causes include:
  • Small, non-representative training dataset
  • Excessive training (too many epochs)
  • High model complexity (learning noise)

Preventing Overfitting

• Increase training data size (more representative)
• Implement early stopping (stop training before overfitting)
• Data augmentation (increase data diversity)
• Adjust existing hyperparameters (learning rate, batch size, epochs) but this is generally not the primary solution to overfitting