Statistical Machine Learning Assignment

• Statistical Machine Learning Assignment focuses on maximum likelihood estimation, loss functions, probabilistic models, and linear regression with Gaussian priors.

Question 1

• $\mathcal{X}$ represents a set of objects, and $Y = {1, \dots, K}$ represents a set of $K$ class labels.
• The dataset consists of i.i.d observations $(\mathbf{x}_1, y_1), \dots, (\mathbf{x}_N, y_N)$ where $\mathbf{x}_i \in \mathcal{X}$ and $y_i \in Y$.
• $p(y = k \mid \mathbf{x}; \theta)$ is the probability of class $k$ given object $\mathbf{x}$, parameterized by $\theta$.
• The task involves deriving the maximum-likelihood estimator of $\theta$ and the gradient of the log-likelihood with respect to $\theta$.

Question 2

• The loss function is given by $\ell(y, \hat{y}) = \max(0, 1 - y\hat{y})$, where $y \in {-1, 1}$ is the ground-truth label and $\hat{y} \in \mathbb{R}$ is the predicted output.
• This loss function is called the hinge loss.
• The questions ask whether the loss function is convex in $\hat{y}$, differentiable in $\hat{y}$ and to derive the gradient of this loss function with respect to $\hat{y}$.

Question 3

• A probabilistic model is defined as $p(x, y \mid \theta) = p(y \mid \theta) p(x \mid y, \theta)$, where $x \in \mathbb{R}$, $y \in {0, 1}$, and $\theta = (\theta_1, \theta_2, \theta_3) \in \mathbb{R}^3$.
• Probabilities are given as $p(y=1 \mid \theta) = \frac{1}{1 + e^{-\theta_1}}$, $p(x \mid y=0, \theta) = \frac{1}{\sqrt{2\pi}} e^{-\frac{1}{2}(x - \theta_2)^2}$, and $p(x \mid y=1, \theta) = \frac{1}{\sqrt{2\pi}} e^{-\frac{1}{2}(x - \theta_3)^2}$.
• The task requires to write down the log-likelihood function and to derive its gradient with respect to $\theta$.

Question 4

• The linear regression model has a Gaussian prior on the weights: $p(\mathbf{y} \mid X, \mathbf{w}, \sigma^2) = \mathcal{N}(\mathbf{y} \mid X\mathbf{w}, \sigma^2 I)$ and $p(\mathbf{w} \mid \alpha) = \mathcal{N}(\mathbf{w} \mid \mathbf{0}, \alpha^{-1} I)$.
• $\mathbf{y} \in \mathbb{R}^N$, $X \in \mathbb{R}^{N \times D}$, $\mathbf{w} \in \mathbb{R}^D$, $\sigma^2 > 0$, and $\alpha > 0$.
• The tasks are to derive the maximum a posteriori (MAP) estimator of $\mathbf{w}$ and justify if the MAP estimator of $\mathbf{w}$ is a convex function of $\mathbf{w}$.