Aleatoric and Epistemic Uncertainty

• Uncertainty in machine learning involves quantifying the reliability of model predictions
• Two primary types of uncertainty are: aleatoric and epistemic

Aleatoric Uncertainty

• Aleatoric uncertainty stems from inherent randomness
• It is irreducible, also known as statistical uncertainty
• Characterized as data-dependent
• Gathering more data will not reduce it
• Types of aleatoric uncertainty:
  • Homoscedastic: Constant uncertainty across all inputs
  • Heteroscedastic: Uncertainty varies depending on the input data points

Epistemic Uncertainty

• Epistemic uncertainty arises from a lack of knowledge about the correct model
• It is also known as systematic uncertainty and is model-dependent
• More data can reduce epistemic uncertainty

Modeling Aleatoric Uncertainty

• Regression tasks model the distribution parameters
• This includes both mean and variance
• Predictions involve estimating the most probable parameters
• Classification tasks use softmax output as a predictive probability

Modeling Epistemic Uncertainty

• Bayesian Neural Networks (BNN)
  • BNNs assign a prior probability distribution to the model's weights
  • The posterior distribution of the weights is computed given the observed data
  • Predictions are made by averaging over this posterior distribution
  • A challenge is the intractability of computing the posterior distribution
• Monte Carlo (MC) Dropout
  • MC Dropout applies dropout during the test phase
  • Multiple predictions are generated using different dropout masks
  • These predictions are then averaged to produce a final result
  • MC Dropout approximates Bayesian inference in a Deep Gaussian Process
• Deep Ensembles
  • This trains multiple neural networks using varied random initializations
  • The aggregated average of the different networks are taken for the final prediction
  • Deep Ensembles can also estimate aleatoric uncertainty

Problems Due to Overconfidence

• Calibration ensures that confidence levels reflect the actual likelihood of correctness
• Overconfidence can cause issues in:
  • Reinforcement learning exploration
  • Self-driving cars in unexpected situations
  • Medical diagnosis leading to incorrect diagnoses
• Models tend to be overconfident when dealing with data outside the training distribution

Improving Uncertainty

• Calibration
  • Post-processing methods adjust probabilities without retraining the model
  • Examples include Platt scaling, Isotonic regression, and Temperature scaling
  • The primary focus is to improve the model's calibration
• Training Data
  • Create better training datasets through increased quantity and diversity
  • Data augmentation can improve uncertainty estimates
• Model Architecture
  • Use a Bayesian neural network
  • Model architecture can be modified to provide better uncertainty estimates
• Loss Function
  • Modify the loss function
  • Use focal loss to improve uncertainty estimates
• Regularization
  • Use regularization methods
  • Use dropout to improve uncertainty estimates
• Ensembles
  • Use deep ensembles
  • Ensembles can improve uncertainty estimates