Sample Statistics PDF

SAMPLE STATISTICS A random sample of size n from a distribution f (x) is a set of n random variables x1 , x2 , . . . , xn which are independently and identically distributed with xi ∼ f (x) for all i. Thus, the joint p.d.f of the random sample is f (x1 , x2 , . . . , xn ) = f (x1 )f (x2 ) · · · f (x...

SAMPLE STATISTICS A random sample of size n from a distribution f (x) is a set of n random variables x1 , x2 , . . . , xn which are independently and identically distributed with xi ∼ f (x) for all i. Thus, the joint p.d.f of the random sample is f (x1 , x2 , . . . , xn ) = f (x1 )f (x2 ) · · · f (x2 ) = n
f (xi ). i=1 A statistic is a function of the random variables of the sample, also know as the sample points. Examples are the sample mean x̄ = xi /n and the sample variance s2 = (xi − x̄)2 /n A random sample may be regarded as a microcosm of the population from which it is drawn. Therefore, we might attempt to estimate the moments of the population’s p.d.f f (x) by the corresponding moments of the sample statistics. To determine the worth of such estimates, we may determine their expected values and their variances. Beyond finding these simple measures, we might endeavour to find distributions of the statistics, which are described as their sampling distributions. We can show, for example, that the mean x̄ of a random sample is an unbiased estimate of the population moment µ = E(x), since E(x̄) = E  x  i n = 1 n E(xi ) = µ = µ. n n Its variance is V (x̄) = V  x  i n = 1  n 2 σ2 . V (x ) = σ = i n2 n2 n Here, we have used the fact that the variance of a sum of independent random variables is the sum of their variances, since the covariances are all zero. Observe that V (x̄) → 0 as n → ∞. Since E(x̄) = µ, this implies that, as the sample size increases, the estimates become increasingly concentrated around the true population parameters. Such an estimate is said to be consistent The sample variance, however, does not provide an unbiased estimate of σ 2 = V (x), since     1 1 2 (xi − x̄) = E (xi − µ) + (µ − x̄) E(s ) = E n n   1 (xi − µ)2 + 2(xi − µ)(µ − x̄) + (µ − x̄)2 =E n 2 = V (x) − 2E{(x̄ − µ)2 } + E{(x̄ − µ)2 } = V (x) − V (x̄). Here, we have used the result that  E  1 (xi − µ)(µ − x̄) = −E{(µ − x̄)2 } = −V (x̄). n 1 2 It follows that E(s2 ) = V (x) − V (x̄) = σ 2 − σ2 (n − 1) = σ2 . n n Therefore, s2 is a biased estimator of the population variance and, for an unbiased estimate, we should use n σ̂ = s = n−1 2 2  (xi − x̄)2 . n−1 However, s2 is still a consistent estimator, since E(s2 ) → σ 2 as n → ∞ and also V (s2 ) → 0. The value of V (s2 ) depends on the form of the underlying population distribution. It would help us to know exactly how the estimates are distributed. For this, we need some assumption about the functional form of the probability distribution of the population. The assumption that the population has a normal distribution is a conventional one, in which case, the following theorem is of assistance: , . . . , xn be a random sample from the normal Theorem. Let x1 , x2  population 2 = ai xi is normally distributed with E(y) = ai E(xi ) = N (µ, σ ). Then, y   µ ai and V (y) = a2i V (xi ) = σ 2 a2i . In general, any linear function of a set of normally distributed variables is itself normally distributed. Thus, for example, if x1 , x2 , . . . , xn is a random sample from the normal population N (µ, σ 2 ), then x̄ ∼ N (µ, σ 2 /n). The general result is best expressed in terms of matrices. Let
µ = [µ1 , µ2 , . . . , µn ] = E(x) denote the vector of the expected values of the elements of x = [x1 , x2 , . . . , xn ]
and let Σ = [σij ; i, j = 1, 2, . . . , n] denote the matrix of their variances and covariances. If a = [a1 , a2 , . . . , an ]
is a constant vector of order n, then a
x ∼ N (a
µ, a
Σa) is a normally distributed random variable with a mean of  ai xi E(a
x) = a
µ = and a variance of V (a
x) = a
Σa =  i ai aj σij =  j a2i σii + i  i ai aj σij . j=i An important case is when the vector a = [a1 , a2 , . . . , an ]
becomes a vector of n units, denoted by ι = [1, 1, . . . , 1]
and described as the summation vector. Then, if x = [x1 , x2 , . . . , xn ]
is the vector of a random sample with xi ∼ N (µ, σ 2 ) for all i, there is x ∼ N (µι, σ 2 In ), where µι = [µ, µ, . . . , µ]
is a vector with µ repeated n times and In is an identity matrix of order n. Writing this explicitly, we have     2 µ x1 σ  µ   0  x2      x=  ...  ∼ N  ..  ,  ... . 0 xn µ  2  0 0  . ..  .  0 σ2 .. . ··· ··· .. . 0 · · · σ2 Then, there is x̄ = (ι
ι)−1 ι
x = 1
ι x ∼ N (µ, σ 2 /n) n and 1 2 ι
{σ 2 I}ι σ 2 ι
ι σ2 = = σ = , n n2 n2 n where we have used repeatedly the result that ι
ι = n . Even if we do not know the form of the distribution from which the sample has been taken, we can still say that, under very general conditions, the distribution of x̄ tends to normality as n → ∞. Thus we have Theorem. The Central Limit Theorem states that, if x1 , x2 , . . . , xn is a random sample from a distribution with mean µ and variance σ 2 , then the distribution 2 of x̄ tends √ to the normal distribution N (µ, σ /n) as n → ∞. Equivalently, (x̄ − µ)/(σ/ n) tends in distribution to the standard normal N (0, 1) distribution. To describe the distribution of the sample variance, we need to define the chisquare distribution. If x ∼ N (0, 1) is distributed as a standard normal variable, then x2 ∼ χ2 (1) is distributed as a chi-square variate with one degree of freedom. Moreover Theorem. The sum of two independent chi-square variates is a chi-square variate with degrees of freedom equal to the sum of the degrees of freedom of its additive components. In particular, if x ∼ χ2 (n) and y ∼ χ2 (m), then (x + y) ∼ χ2 (n + M ). It follows that, if x1 , x 2 , . . . , xn is a random sample from a standard normal if x1 , x2 , . . . , xn is a random N (0, 1) distribution, then x2i ∼ χ2 (n). Moreover,  2 sample from an N (µ, σ ) distribution, then (xi − µ)2 /σ 2 ∼ χ2 (n). Consider the identity   2 (xi − µ)2 = ({xi − x̄} + {x̄ − µ})  = {xi − x̄}2 + n{x̄ − µ}2 ,  which follows from the fact that the cross product term is {x̄ − µ} {xi − x̄} = 0. This decomposition of a sum of squares features in the following result: The Decomposition of a Chi-square statistic. If x1 , x2 , . . . , xn is a random sample from a standard normal N (µ, σ 2 ) distribution, then n  (xi − µ)2 i=1 σ2 with (1) = n  (xi − µ)2 σ2 n  (xi − x̄)2 i=1 (3) σ2 i=1 i=1 (2) n  (xi − x̄)2 n σ2 (x̄ − µ)2 +n , σ2 ∼ χ2 (n), ∼ χ2 (n − 1), (x̄ − µ)2 ∼ χ2 (1), σ2 3 where the statistics under (2) and (3) are independently distributed. Definitions. (1) If u ∼ χ2 (m) and v ∼ χ2 (n) are independent chi-square variates with m and n degrees of freedom respectively, then  F = u m   v ∼ F (m, n), n which is the ratio of of the chi-squares divided by their respective degrees of freedom, has an F distribution of m and n degrees of freedom, denoted by F (m, n). (2) If x ∼ N (0, 1) is a standard normal variate and if v ∼ χ2 (n) is a chi-square variate of n degrees of freedom, and if the two variates are distributed independently, then the ratio  v t=x ∼ t(n) n has a t distributed of n degrees of freedom, denoted t(n). Notice that x2 t = ∼ v/n  2 χ2 (1) 1  χ2 (n) n  = F (1, n). CONFIDENCE INTERVALS Consider a standard normal variate z ∼ N (0, 1). From the tables in the back of the book, we can find numbers a, b such that, for any Q ∈ (0, 1), there is P (a ≤ z ≤ b) = Q. The interval [a, b] is called a Q × 100% confidence interval for z. We can minimise the length of the interval by disposing it symmetrically about the expected value E(z) = 0, since z ∼ N (0, 1) is symmetrically distributed about its mean of zero. We can easily construct confidence intervals for the parameters underlying our sample statistics. Since they are concerned with fixed parameters, such confidence statements differ in a subtle way from those regarding random variables. A confidence interval for the mean of the N (µ, σ 2 ) distribution. Let x1 , x2 , . . . , xn be a random sample from a normal N (µ, σ 2 ) distribution. Then  x̄ ∼ N σ2 µ, n  and x̄ − µ √ ∼ N (0, 1). σ/ n Therefore, we can find numbers ±β such that  P x̄ − µ √ ≤β −β ≤ σ/ n 4  = Q. But, the following events are equivalent:  x̄ − µ √ ≤β −β ≤ σ/ n Hence  P    σ σ ⇐⇒ −β √ ≤ x̄ − µ ≤ β √ n n   σ σ ⇐⇒ −β √ ≤ µ − x̄ ≤ β √ n n   σ σ . ⇐⇒ x̄ − β √ ≤ µ ≤ x̄ + β √ n n σ σ x̄ − β √ ≤ µ ≤ x̄ + β √ n n  = Q. √ √ This says that the probability that the random interval [x̄ − βσ/ n, x̄ + βσ/ n] falls over the true value µ is Q. Equivalently, given a particular sample that has a mean value of x̄, we are Q × 100% confident that µ lies in the resulting interval. Example. Let (1.2, 3.4, 0.6, 5.6) be a random sample from a normal N (µ, σ 2 = 9) distribution. Then x̄ = 2.7 and 2.7 − µ x̄ − µ √ = ∼ N (0, 1). 3/2 σ/ n Hence  P  2.7 − µ ≤ 1.96 −1.96 ≤ 3/2 = Q, and it follows that (0.24 ≤ µ ≤ 5.64) is our 95% confidence interval. Usually, we have to esA confidence interval for µ when σ 2 is unknown.  2 2 2 timate σ . The unbiased estimate of σ is σ̂ = (xi − x̄)2 /(n − 1). With this estimate replacing√σ 2 , we have to replace the standard normal distribution, which is appropriate to n(x̄ − µ)/σ, by the t(n − 1) distribution, which is appropriate √ to n(x̄ − µ)/σ̂. To demonstrate this result, consider writing  x̄ − µ √ = σ̂/ n x̄ − µ √ σ/ n    (xi − x̄)2 , σ 2 (n − 1) and observe that we can cancel the unknown value of σ from the numerator and the denominator.  Now, (xi − x̄)2 /σ 2 ∼ χ2 (n − 1), so the denominator contains the root of a chi-square variate divided by its n − 1 degrees of freedom. The numerator contains a standard normal variate. That is to say, the statistic has the form of   N (0, 1) χ2 (n − 1) n−1 5  ∼ t(n − 1). To construct a confidence interval, we proceed as before, except that we replace the numbers ±β, obtained from the table of the N (0, 1) distribution, by the corresponding numbers ±b, obtained from the t(n − 1) table. Our statement becomes  P σ̂ σ̂ x̄ − b √ ≤ µ ≤ x̄ + b √ n n  = Q. A confidence interval for the difference between two means. Imagine a treatment that affects the mean of a normal population without affecting its variance. An instance of this might be the application of a fertiliser that increases the yield of a crop without adversely affecting its hardiness. We might wish to estimate the effect of the fertiliser; and, in that case, we would probably want to construct a confidence interval for the estimate. To establish a confidence interval for the change in the mean, we would take samples from the population before and after treatment. Before treatment, there is   σ2 2 and x̄ ∼ N µx , , xi ∼ N (µx , σ ); i = 1, . . . , n n and, after treatment, there is  yj ∼ N (µy , σ ); 2 j = 1, . . . , m and ȳ ∼ N σ2 µy , m  . Then, on the assumption that the two samples are mutually independent, the difference between the sample means is  (x̄ − ȳ) ∼ N Hence σ2 σ2 + µx − µy , n m  . (x̄ − ȳ) − (µx − µy )  ∼ N (0, 1). σ2 σ2 + n m If σ 2 were known, then, for any given value of Q ∈ (0, 1), we could find a number β from the N (0, 1) table such that  P  (x̄ − ȳ) − β σ2 σ2 + ≤ µx − µy ≤ (x̄ − ȳ) + β n m  σ2 σ2 ++ n m  = Q. This would give a confidence interval for µx − µy . Usually, we have to estimate σ 2 from the sample information. We have  (xi − x̄)2 σ2 ∼ χ2 (n − 1) and 6  (yj − ȳ)2 ∼ χ2 (m − 1), σ2 which are independent variates with expectations equal to the numbers of their degrees of freedom. The sum of independent chi-squares is itself a chi-square with degrees of freedom equal to the sum of those of its constituent parts. Therefore,   (xi − x̄)2 + (yj − ȳ)2 ∼ χ2 (n + m − 2) 2 σ has an expected value of n + m − 2, whence   (xi − x̄)2 + (yj − ȳ)2 σ̂ = n+m−2 2 is an unbiased estimate of the variance. If we use the estimate in place of the unknown value of σ 2 , we get       2 2 (x̄ − ȳ) − (µx − µy ) (x̄ − ȳ) − (µx − µy ) (xi − x̄) + (yj − ȳ)   $ =   σ 2 (n + m − 2) σ2 σ2 σ̂ 2 σ̂ 2 + n m + n m N (0, 1) = t(n + m − 2). ∼$ 2 χ (n+m−2) n+m−2 This is the basis for determining a confidence interval that uses an estimated variance in place of the unknown value. A confidence interval variance. If xi ∼ N (µ, σ 2 ); i = 1, . . . , n is a  for the 2 random  sample, then (xi − x̄) /(n − 1) is an unbiased estimate of the variance and (xi − x̄)2 /σ 2 ∼ χ2 (n − 1). Therefore, by looking in the back of the book at the appropriate chi-square table, we can find numbers α and β such that  P   (xi − x̄)2 α≤ ≤β =Q σ2 for some chosen Q ∈ (0, 1). From this, it follows that  P 1 1 σ2 ≥ ≥ 2 (xi − x̄) α β     (xi − x̄)2 (xi − x̄)2 2 = Q ⇐⇒ P ≤σ ≤ =Q β α and the latter provides a confidence interval for σ 2 . We ought to choose α and β so as to minimise the length of the interval [α−1 , β −1 ]. The chi-square is an asymmetric distribution, so it is tedious to do so. The distribution becomes increasingly symmetric as the sample size n increases, and so, for large values of n, we may choose α and β to demarcate equal areas within the two tails of the distribution. The confidence interval for the ratio of two variances. Imagine a treatment that affects the variance of a normal population. We might also wish to allow for the possibility that the mean is also affected. Let xi ∼ N (µx , σx2 ); i = 1, . . . , n 7 be a random sample taken from the population before treatment and let yj ∼ N (µy , σy2 ); j = 1, . . . , m be a random sample taken after treatment. Then  (xi − x̄)2 σ2 ∼ χ (n − 1) 2 and  (yj − ȳ)2 ∼ χ2 (m − 1), 2 σ are independent chi-squared variates, and hence    (xi − x̄)2 (yj − ȳ)2 F = ∼ F (n − 1, m − 1). σx2 (n − 1) σy2 (m − 1) It is possible to find numbers α and β such that P (α ≤ F ≤ β) = Q, where Q ∈ (0, 1) is some chose probability value. Given such values, we may make the following probability statement: (  )  σy2 (yj − ȳ)2 (n − 1) (yj − ȳ)2 (n − 1) P α ≤ 2 ≤ β = Q. σx (xi − x̄)2 (m − 1) (xi − x̄)2 (m − 1) 8