Probability Theory & Statistics Lecture Notes PDF

Probability theory and statistics Lecture Notes Rostyslav Hryniv Ukrainian Catholic University Computer Science Programme 4th term Spring 2018 The z-test The t-test (unknown σ 2 ) Testing the variance Other tests...

Probability theory and statistics Lecture Notes Rostyslav Hryniv Ukrainian Catholic University Computer Science Programme 4th term Spring 2018 The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Lecture 14. Testing hypotheses for normal distribution The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Outline 1 The z-test One-sided hypotheses for the mean Uniformly the most powerful tests p-value Double-sided tests 2 The t-test (unknown σ 2 ) Testing the mean for unknown σ 2 Testing means of two samples 3 Testing the variance Known mean Unknown mean Testing variances for two samples: the f -test 4 Other tests Goodness-of-fit test Chi-squared Pearson test of categorical data The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Setting of the problem Trapezna prices The widespread belief among UCU students/personnel is that the new pricing policy at Trapezna café has led to price increase Trapezna managers claim the mean student lunch receipt has remained at the old level µ0 Record the price data x1 ,... , xn ; these are realizations of X1 ,... , Xn — sample from a distribution family with E(Xk ) = µ and Var(Xk ) = σ 2 (the latter assumed known) Although Xk need not be normally distributed, X will be approximately normal by the CLT WLOG, assume normality: Xk ∼ N (µ, σ 2 ) Task: Test H0 : µ = µ0 vs H1 : µ > µ0 The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Testing H0 : µ = µ0 vs H1 : µ > µ0 , σ 2 known When H1 is simple: µ = µ1 > µ0 , the best test of size ≤ α is the likelihood ratio test: n Lx (H1 ) o Cα := x ∈ Rn | ≥ cα Lx (H0 ) it takes the form of the z-test Cα = {x ∈ Rn | z(x) ≥ z1−α } (*) √ for z(x) := n(x − µ0 )/σ Therefore, Cα is independent of µ1 Thus Cα is the likelihood ratio region for all µ1 > µ0 Corollary: Cα of (*) is uniformly the most powerfull test of size ≤ α Example: µ0 = 30, σ = 5, x = 31.5, n = 100, α = 0.05 z0.95 ≈ 1.65, z(x) = (31.5 − 30) × 10/5 = 3 > z0.95 The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Uniformly the most powerful tests Recall that the power of a test with rejection region C is β(θ) := P(X ∈ C | θ) In the above case, √ n βLRT (µ) = P( σ (X − µ0 ) ≥ z1−α0 | µ) √ √ = P( σn (X − µ) ≥ z1−α0 + n σ (µ0 − µ) | µ) √ n = 1 − Φ(z1−α0 + σ (µ0 − µ)) Among all tests for H0 : µ = µ0 vs H1 : µ > µ0 of size ≤ α0 , βLRT assumes the largest possible values for all µ > µ0 That’s why we call it unformly the most powerful test of size ≤ α0 In fact, it is also uniformly the most powerful for H0 : µ ≤ µ0 vs H1 : µ > µ0 look at R The z-test The t-test (unknown σ 2 ) Testing the variance Other tests p-value of the test Assume H0 got rejected at the level α = 0.1, i.e., x ∈ Cα=0.1 To sound more confident, test it at the level α = 0.05: construct Cα=0.05 check whether x ∈ Cα=0.05 Assume H0 got rejected again; can try smaller α for which H0 can be rejected For each such α must recalculate the rejection region! Alternative: calculate once the p-value Definition (p-value) The p-value of the test is the smallest size α at which H0 can be rejected for given observation x Instead of trying different α, calculate the p-value p(x): if p(x) is small enough (p(x) ≤ α), reject H0 if p(x) is relatively large (p(x) > α), do not reject H0 The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Calculating the p-value Among all rejection regions Cα , look for the smallest one still containing x: p(x) = min{α | x ∈ Cα } = min{α | z(x) ≥ z1−α } Clearly, such a minimal α satisfies z1−α = z(x), or Φ(z1−α ) = Φ(z(x)) ⇐⇒ α = 1 − Φ(z(x)) Therefore, p(x) = 1 − Φ(z(x)) We can reject H0 at significance level p(x), but not at smaller one! If p(x) is small enough, reject H0 ; otherwise, do not reject! Example: µ0 = 30, σ = 5, x = 31.5, n = 100 p(x) = 1 − Φ(3) ≈ 0.001 The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Testing H0 : µ = µ0 vs H1 : µ < µ0 Start with simple alternative H1 : µ = µ1 < µ0 , use the LRT and Neyman–Pearson lemma to get the best test (left-sided z-test): Cα = {x ∈ Rn | z(x) ≤ zα } Cα is independent of µ1 =⇒ can use it for the composite alternative H1 : µ < µ0 This is uniformly the most powerful test for H1 : µ < µ0 It is uniformly the most powerful test also for testing H0 : µ ≥ µ0 vs H1 : µ < µ0 The p-value is equal to p(x) = Φ(z(x)) The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Alternative of no importance When the alternative to H0 : µ = µ0 is not important, can use the general alternative H1 : µ 6= µ0 slightly abusing: H1 : µ is arbitrary i.e. µ ∈ R More generally, if in Hj : θ ∈ Θj one has Θ0 ⊂ Θ1 , can use the Generalized Likelihood Ratio Test: Reject H0 at (:::::::::::: approximate) size α when (m) 2 log Lx (H0 , H1 ) ≥ x1−α with (m) xγ quantile of the χ2m of level γ m = |Θ1 | − |Θ0 |, with |Θj | being the degree of freedom The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Testing H0 : µ = µ0 vs general alternative H1 : µ ∈ R Lx (H1 ) = maxµ fX (x | µ, σ 2 ) = fX (x | µ = x, σ 2 ) Lx (H0 , H1 ) = exp{n(x − µ0 )2 /(2σ 2 )} 2 log Lx (H0 , H1 ) = n(x − µ0 )/σ 2 = z 2 (x) Therefore, the GLRT suggest to reject H0 when (1) z 2 (x) ≥ x1−α ⇐⇒ |z(x)| ≥ z1−α/2 This is the two-sided test No chances to be uniformly the most powerful for µ 6= µ0 ! The p-value: p(x) = 2Φ(−|z(x)|) The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Testing means of two samples Trapezna prices: setting of the problem The widespread belief among UCU students/personnel is that the new pricing policy at Trapezna cafe has led to price increase Trapezna managers claim the mean student lunch receipt has remained at the old level Record the price data x1 ,... , xn after the changes and y1 ,... , ym before them WLOG,assume independence and normality: Xk ∼ N (µ0 , σ 2 ), Yk ∼ N (µ1 , σ 2 ) with real µ0 , µ1 and the same variance σ 2 Task: Test H0 : µ0 = µ1 vs H1 : µ0 > µ1 The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Testing H0 : µ0 = µ1 vs H1 : µ0 6= µ1 Can use the generalized LRT with test statistics 2 log Lx,y (H0 , H1 ) Calculate it to get mn (x − y)2 (1) 2 log Lx,y (H0 , H1 ) = 2 ≥ x1−α (m + n) σ Set r mn x − y Z = Z (X, Y) := ; (m + n) σ then under H0 , Z ∼ N (0, 1) therefore, the above is equivalent to the test of exact size α: |z(x, y)| ≥ z1−α/2 p-value: p(x, y) = 2 Φ(−|z(x, y)|) Example: µ0 = 30, σ = 5, x = 31.5, y = 31 n = m = 100, α = 0.05 √ √ z0.975 ≈ 1.96, z(x, y) = (31.5 − 31) × 100/5 20 = 5 > z0.975 The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Other tests for µ0 and µ1 One-sided alternative H1 : µ0 < µ1 Can use the same statistics z(x, y): makes sense to reject H0 in favour of H1 for small values of z(x, y) rejection region: Cα = {(x, y) ∈ Rn+m | z(x, y) ≤ zα } p-value: p(x, y) = Φ(z(x, y)) Analogously, can test H0 vs H1 : µ0 > µ1 None of these tests is uniformly the most powerful The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Test statistics when σ 2 is unknown Assume X1 ,... , Xn are i.i.d. r.v.’s with distribution N (µ, σ 2 ) with unknown µ and σ 2 Earlier, used test statistics √ n Z (X) := (X − µ0 ) σ which under H0 has distribution N (0, 1) p For unknown σ, can use its estimate S(X) = SXX /(n − 1) The corresponding test statistics √ n Z (X) T (X) := (X − µ0 ) = S(X) S(X)/σ has the Student Tn−1 -distribution with n − 1 degrees of freedom Therefore, can replace the Z (X) statistics with the T (X) statistics and use the quantiles for the distribution Tn−1 This way get the t-tests for the mean The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Testing H0 : µ = µ0 For one-sided alternative H1 : µ > µ0 use (n−1) Cα = {x ∈ Rn | t(x) ≥ t1−α } this is the test of p-value p(x) := 1 − FTn−1 (t(x)) For one-sided alternative H1 : µ < µ0 use Cα = {x ∈ Rn | t(x) ≤ tα(n−1) } this is the test of p-value p(x) := FTn−1 (t(x)) For two-sided alternative H1 : µ 6= µ0 use (n−1) Cα = {x ∈ Rn | |t(x)| ≥ t1−α/2 } this is the test of p-value p(x) := 2FTn−1 (−|t(x)|) none of the above tests is uniformly the most powerful The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Testing H0 : µ0 = µ1 for two samples Assume Xk ∼ N (µ0 , σ 2 ), Yk ∼ N (µ1 , σ 2 ) with real µ0 , µ1 and the same unknown variance σ 2 In the above tests, replace σ in r mn X − Y Z (X, Y) := m+n σ with its estimator (SXX + SYY )/(n + m − 2) to get the t-statistics r s mn n+m−2 T (X, Y) := (X − Y) m + n SXX + SYY whose distribution under H0 is the Student Tn+m−2 distribution The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Testing H0 : σ 2 = σ02 vs H1 : σ 2 > σ02 , µ known When µ is known, use test H0 vs a simple alternative σ 2 = σ12 > σ02 Use the LRT to get the rejection region (n) X Cα = {x ∈ Rn | (Xk − µ)2 /σ02 ≥ x1−α }, (n) where x1−α is the quantile of the distribution χ2n As Cα is independent of σ12 , can use it for the composite alternative H1 : σ 2 > σ02 Also, can take the composite null hypothesis H0 : σ 2 ≤ σ02 This is uniformly the most powerful test The p-value is X  p(x) = 1 − Fχ2n (xk − µ)2 /σ02 The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Testing H0 : σ 2 = σ02 vs H1 : σ 2 6= σ02 , µ known Use the two-sided test with test statistics X V = (Xk − µ)2 /σ02 that under H0 has distribution χ2n Thus (n) (n) Cα = {x ∈ Rn | v (x) ≥ x1−α/2 or v (x) ≤ xα/2 } The p-value is p(x) = 2 min{Fχ2n (v (x)), 1 − Fχ2n (v (x))} The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Testing H0 : σ 2 = σ02 vs H1 : σ 2 > σ02 , µ unknown In the above tests, replace µ with its estimate X to get the test statistics V (X) := SXX /σ02 which under H0 has distribution χ2n−1 the rest is the same: the rejection region (n−1) Cα = {x ∈ Rn | v (x)/σ02 ≥ x1−α }, (n−1) where x1−α is the quantile of the distribution χ2n−1 As Cα is independent of σ12 , can use it for the composite alternative H1 : σ 2 > σ02 Also, can take the composite null hypothesis H0 : σ 2 ≤ σ02 This is not uniformly the most powerful test The p-value is   p(x) = 1 − Fχ2 v (x)/σ02 n−1 The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Testing H0 : σ02 = σ12 vs H1 : σ02 > σ12 , µ unknown The idea is that Sxx /(n − 1) is the estimate for σ02 Syy /(m − 1) is the estimate for σ12 Under the null hypothesis, the test statistics SXX /(n − 1) F (X, Y) := SYY /(m − 1) has the Fischer distribution Fn−1,m−1 under H1 , F would assume smaller values Thus the rejection region can be Cα := {x ∈ Rn , y ∈ Rm | f (x, y) ≤ fα } where fα is the quantile of level α for the Fischer distribution Fn−1,m−1 p-value of the test is FF (f (x, y)) The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Kolmogorov’s Goodness-of-fit test We have data x1 , x2 ,... , xn sampled from a distribution F F is not known; for a given hypothetical F0 , want to test H0 : F = F0 vs H1 : F 6= F0 By LLN, the sample c.d.f. F bx of x is close to F thus under H0 the statistics d := sup |F bx (t) − F0 (t)| t∈R should take small values under H0 , the distribution of d is independent of F0 and is called the Kolmogorov distribution Dn under H1 , d assumes larger values this is the Kolmogorov’s goodness-of-fit test with rejection region (n) Cα := {x ∈ Rn | d ≥ d1−α } For two samples, use modification: Kolmogorov–Smirnov test Look at R The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Is a die fair? Chi-squared Pearson test Roll a die n times and record the numbers of times n1 , n2 ,... , n6 that k = 1, 2,... , 6 showed up do the data confirm that the die is symmetric? Let pi be the probability of outcome i Want to test the hypothesis H0 : pi = pi (θ) vs H1 : pi are unrestricted use the Pearson chi-squared statistics X (oi − ei )2 t := ei i under H0 has approx. a χ2r distribution with df = r ; under H1 takes larger values oi := ni observed outcomes; ei predicted under H0 r = k − 1 − |Θ| The z-test The t-test (unknown σ 2 ) Testing the variance Other tests Other χ2 tests χ2 -test for homogeneity χ2 -test for independence Example A researcher pretended to drop pencils in a lift and observed whether the other occupant helped to pick them up. Helped Did not help Total Men 370 950 1,320 Women 300 1,003 1,303 Total 670 1,953 2,623 Are the two characteristics independent?