Bayesian Linear Regression

• Lecture focuses on Bayesian linear regression

Bayesian vs. Maximum Likelihood

• Maximum Likelihood (ML) finds a single "best fit" parameter $\theta_{ML}$, where $\theta_{ML} = argmax_{\theta} p(D|\theta)$
• The Bayesian approach finds a distribution over parameters $p(\theta|D)$, where $p(\theta|D) = \frac{p(D|\theta)p(\theta)}{p(D)}$; $p(\theta)$ is the prior, $p(D|\theta)$ is the likelihood, $p(\theta|D)$ is the posterior, and $p(D)$ is the marginal likelihood.

Prediction

• Maximum Likelihood plugs in the "best fit" $\theta_{ML}$, so $p(x^|D) \approx p(x^|\theta_{ML})$
• The Bayesian approach integrates over the posterior distribution, $p(x^|D) = \int p(x^|\theta)p(\theta|D)d\theta$ ; $p(x^*|\theta)$ is the predictive distribution

Advantages of Bayesian Approach

• It naturally prevents overfitting
• It has the ability to incorporate prior knowledge
• Uncertainty can be quantified

Bayesian Linear Regression

• Model definition: $y = w^T x + \epsilon$, where $\epsilon \sim N(0, \sigma^2)$ and $p(y|x, w, \sigma^2) = N(y|w^T x, \sigma^2)$ and $p(w) = N(w|0, \alpha^2I)$
• The posterior is given by $p(w|D) = \frac{p(D|w)p(w)}{p(D)}$, where $p(D|w) = \prod_{n=1}^N p(y_n|x_n, w, \sigma^2)$ and $p(w|D) = N(w|\mu_N, \Sigma_N)$
  • $\mu_N = \frac{1}{\sigma^2} \Sigma_N X^T y$
  • $\Sigma_N^{-1} = \frac{1}{\alpha^2}I + \frac{1}{\sigma^2}X^T X$
• Prediction can be shown as $p(x^|D) = \int p(x^|w)p(w|D)dw$ and $p(x^|w) = N(x^|w^T x^, \sigma^2)$ and $p(x^|D) = N(x^|\mu_N^T x^, \sigma^2 + x^{T} \Sigma_N x^)$

Steps

• First, define a prior distribution over the parameters
• Second, compute the posterior distribution over the parameters given the data
• Third, predictions are made by averaging over the posterior distribution