Lecture 5 -- Hypothesis Tests (1) PDF

Understanding Hypothesis Testing Psyc610  Understanding probability Null Hypothesis Significance Testing (NHST) Agenda The Z distribution An alternative viewpoint to NHST  Using statistics to test our...

Understanding Hypothesis Testing Psyc610  Understanding probability Null Hypothesis Significance Testing (NHST) Agenda The Z distribution An alternative viewpoint to NHST  Using statistics to test our hypotheses is all about Why do we determining the probability that care about we would obtain the data we do, given that the null is true probabilitie s? In psychology, we have set this standard at a 5% chance Possible Outcomes True No True Difference Difference Find a True Positive False Positive Difference (α or Type I Error) Do Not find a False True Negative Difference Negative (β or Type II Error)  In psychology, we set =.05 (most of the time) We also often try to Alpha, Beta, and set β=.20 Power Related to this is something called Power (1- β =Power) Power is our ability to detect a difference when one exists in the population… more on this later!  Using the.05 standard is arbitrary In some cases, we may set it lower because we are worried about a false positive, or higher Comprehensio because we are worried about a false negative n Check 1. What might be a situation where a false positive is dangerous? 2. What might be a situation where a false negative is dangerous? How do we know an effect is large enough to “matter? Two main approaches Frequentist (Null Hypothesis Significance Testing, or NHST) We are willing to tolerate a certain level of chance (usually 5%) of being wrong about an inference from our data “There is a 50% chance we would find this result if the treatment didn’t work. Therefore, the treatment does not work.” NO GRAY AREAS… it is either significant, or it is not! Based on frequency of an event assuming the null is true. This is what most journals and professors use (and therefore, what I teach) Bayesian We adjust our confidence about an outcome after seeing some data or learning about conditions. “Prior data suggested the treatment had a 50% chance of working, but we also know it works 4x better for women. Since we are treating women, there is an 80% chance it will work, so odds are good that this treatment will work.  Null Hypothesis Significance Testing NHST Hypothesis Testing Any sample we collect from a population will be a little different from UG Grad another sample Stats Stats Are they different for a real reason, or are they different due to chance? N=37 N=28 M=4. M=3. 5 75 Differences between Means given the Null is true Frequency -1.5.5 1 0.5 1 1.5 Differences between sample means based on 100 score Area Under the Curve Frequency -1.5.5 1 0.5 1 1.5 Differences between sample means based on 100 score How hypothesis testing works 1. Establish question Am I a better instructor for undergrads or grads? 2. Obtain random samples under each condition 3. Set up null hypothesis (H0) There is no difference between UG and Grad ratings 4. Set up alternative hypothesis (H1) Two-tailed: UG ratings will be different than Grad ratings One-tailed: UG ratings will be lower/higher than Grad ratings  Calculate probability Determine mean of finding such a difference found in mean difference sample given H0 is true How Compare this hypothesis obtained probability (p-value) to the If probability is testing standard we set for a false positive.05, fail to reject H0 Differences between Means Frequency -1.5.5 1 0.5 1 1.5 Differences between sample means based on 100 score This is important! “What is the probability that I would have obtained these data given that the null hypothesis is true” NOT “What is the probability that the null hypothesis is true given the data we obtained”  Never use the words “prove” or “accept” Tips for We can reject the null hypothesis (significant writing finding) about hypothesis We can fail to reject the null hypothesis (non-significant testing findings) If we sample 3,000 people, and they all have two arms, does this mean that we have proven that all people have two arms? Common Mistakes “The probability of such a swing of 697 votes from the expected results was about 6 percent. Putting it another way, if past elections are a reliable guide to current voting behavior, there is a 94 percent chance that irregularities in the absentee ballots, not chance alone, swung the election to the Democrat. New York Times, 1994 Common Mistakes A p-value of