Statistical Inference: Point Estimation

• Statistical inference involves drawing conclusions about a population based on a sample.

Point Estimation

• A point estimator, denoted as $\hat{\Theta}$, is a statistic used to estimate a parameter $\theta$ of a probability distribution.
• $\hat{\Theta}$ is a function of the random sample $X_1, X_2,..., X_n$, expressed as $\hat{\Theta} = g(X_1, X_2,..., X_n)$.

Methods for Finding Estimators

• Two common methods are the Method of Moments and Maximum Likelihood Estimation (MLE).

Method of Moments

• This method is used when estimating parameters $\theta_1, \theta_2,..., \theta_k$ of a probability distribution.
• First, the i-th population moment $\mu_i = E(X_i)$ is expressed in terms of $\theta_1, \theta_2,..., \theta_k$.
• Second, the i-th sample moment is calculated as $\hat{\mu_i} = \frac{1}{n}\sum_{i=1}^{n} X_i$.
• Finally, a system of equations ($\mu_i = \hat{\mu_i}$) is solved for $\theta_1, \theta_2,..., \theta_k$.

Maximum Likelihood Estimation (MLE)

• This method is used when parameters $\theta$ of a probability distribution.
• First, the likelihood function $L(\theta; x_1, x_2,..., x_n) = f(x_1, x_2,..., x_n;\theta) = \prod_{i=1}^{n} f(x_i;\theta)$ is found.
• Then, the log-likelihood function is computed as $l(\theta; x_1, x_2,..., x_n) = ln(L(\theta; x_1, x_2,..., x_n))$.
• Next, $l(\theta; x_1, x_2,..., x_n)$ is differentiated with respect to $\theta$ and set equal to 0.
• Finally, the equation is solved for $\theta$ to obtain the MLE.

Properties of Estimators

• Important properties include bias, variance, and mean squared error (MSE).

Bias

• The bias of an estimator $\hat{\Theta}$ is defined as $B(\hat{\Theta}) = E(\hat{\Theta}) - \theta$.
• An estimator is unbiased if its bias is zero: $B(\hat{\Theta}) = 0$.
• An estimator is biased if its bias is not zero: $B(\hat{\Theta}) \neq 0$.

Variance

• The variance of an estimator $\hat{\Theta}$ is defined as $Var(\hat{\Theta}) = E[(\hat{\Theta} - E(\hat{\Theta}))^2]$.

Mean Squared Error (MSE)

• The MSE of an estimator $\hat{\Theta}$ is defined as $MSE(\hat{\Theta}) = E[(\hat{\Theta} - \theta)^2]$, which can be decomposed into $Var(\hat{\Theta}) + [B(\hat{\Theta})]^2$.

Example: Unbiased Estimator of Variance

• Given a random sample $X_1, X_2,..., X_n$ from a population with mean $\mu$ and variance $\sigma^2$, the sample variance $S^2 = \frac{1}{n-1}\sum_{i=1}^{n} (X_i - \bar{X})^2$ is an unbiased estimator of $\sigma^2$ because $E(S^2) = \sigma^2$.

Theorem: Minimum Variance Unbiased Estimator of the Mean

• For a random sample $X_1, X_2,..., X_n$ from a population with mean $\mu$ and variance $\sigma^2$, the sample mean $\bar{X}$ is the minimum variance unbiased estimator of $\mu$.