Maximum Likelihood Estimator Derivation

Derivation of the Maximum Likelihood Estimator

Bernoulli Distribution

• The probability mass function for the Bernoulli distribution is defined as ( P(x|\theta) = \theta^x(1-\theta)^{(1-x)} ), where ( x \in {0,1} ).
• The log-likelihood function ( \ell(\theta) ) is derived as ( x\log\theta + (1-x)\log(1-\theta) ).
• The derivative of the log-likelihood with respect to ( \theta ) is ( \frac{x}{\theta} - \frac{1-x}{1-\theta} ).
• Setting the derivative to zero and solving for ( \theta ) leads to the MLE ( \hat{\theta} = x ).

Gaussian Distribution

• The probability density function for the Gaussian distribution is ( P(x|\mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}exp\left{-\frac{1}{2\sigma^2}(x-\mu)^2\right} ).
• The log-likelihood function ( \ell(\mu,\sigma^2) ) is ( -\frac{1}{2}log(2\pi\sigma^2) - \frac{1}{2\sigma^2}(x-\mu)^2 ).
• The derivative of the log-likelihood with respect to ( \mu ) is ( \frac{(x-\mu)}{\sigma^2} ).
• Setting the derivative to zero and solving for ( \mu ) gives the MLE ( \hat{\mu} = x ).
• The derivative of the log-likelihood with respect to ( \sigma^2 ) is ( -\frac{1}{2\sigma^2} + \frac{(x-\mu)^2}{2(\sigma^2)^2} ).
• Setting the derivative to zero and solving for ( \sigma^2 ) results in the MLE ( \hat{\sigma}^2 = (x-\mu)^2 ).