PS219 Lecture 6 PDF - Statistical Significance - 2025

PS219: Research Methods in Psychology Lecture 6 - Statistical Significance January 28th, 2025 Rory Coyne PS219 Module Coordinator [email protected]  Recap on yesterday... The correlation of a sample is represented by Pearson's r. The range of possible va...

PS219: Research Methods in Psychology Lecture 6 - Statistical Significance January 28th, 2025 Rory Coyne PS219 Module Coordinator [email protected]  Recap on yesterday... The correlation of a sample is represented by Pearson's r. The range of possible values for a correlation is between -1 to +1. A positive correlation (+) indicates that as scores on one variable increase/decrease, so too do scores on another variable. A negative correlation ( - ) indicates that as scores on one variable increase, scores on another variable decrease - and vice versa. A correlation coefficient can either be weak, moderate or strong. Weak correlation: r between 0 and ± 0.29 Moderate correlation: r between ± 0.3 and ± 0.59 Strong correlation: r between ± 0.6 and ± 1.00 For instance: r = -0.4 is stronger than r = +0.3 Today we will cover... Samples and populations: 01 generalising and inferring 02 Statistical significance One-tailed and two-tailed tests, 03 and Type I and II errors  Inferential Statistics In statistics, we use a sample of scores to make general statements or draw conclusions that apply well beyond that sample. This is a branch of statistics called inferential statistics. It is so-called because it involves drawing inferences about all scores in the population from just a sample of those scores.  Samples & Populations A sample is just a small number of scores selected from the entirety of scores. A population is the entire set of scores. Basically, a sample is a small set, or a subset, taken from the full set or population of scores. Both population and sample refer to scores on a variable. In some cases, the entire population of scores could be measured........but more often, the the population of scores is infinite and cannot feasibly be measured.  More Examples Population Sample The top 50 search results for advertisements Advertisements for IT jobs in the Netherlands for IT jobs in the Netherlands on May 1, 2020 Winning songs from the Eurovision Song Songs from the Eurovision Song Contest Contest that were performed in English 300 undergraduate students from three Undergraduate students in the Netherlands Dutch universities who volunteer for your psychology research study Countries with published data available on All countries of the world birth rates and GDP since 2000 So most of the time, the sample is known to the researcher, whereas the population generally is not. But what can we really say about the population based on our knowledge of just the sample? Quite a lot. For example, the mean of the sample is typically used as an estimation of the mean of the population.  Random Samples In statistical inference, it is generally assumed that samples are drawn at random from the population. Obtaining a random sample of scores entails selecting scores in such a way that each score in the population has an equal chance of being selected. Many ways to obtain a random sample - manual random number tables, electronic random number generators, etc, pulling names out of a hat, etc. Most random samples have a mean that is very close to the population mean. Therefore, a sample mean obtained from random sampling that is very different from the population mean is relatively rare.  Statistical Significance As psychologists, we are interested in which sample means are very unlikely to occur through random sampling. In statistics, the extreme 5% of these samples are of interest, and so these samples are called significant. Significance in statistics means that the means of the sample are very different from those of the population from which it was drawn. Significance at the 5% level means that the sample score lies within the 5% of samples which are most different from the population. This 5% is obtained from looking at the extreme lower 2.5%, and the extreme upper 2.5%.  Statistical Significance In statistics, we try to understand which are the likely scores in samples from a population, and which are the unlikely ones. Scores in the middle 95% of scores are likely. Scores in the extreme 5% (extreme upper and extreme lower 5%) are unlikely. It is not unreasonable to suggest that if a score from a sample has only a 1 in 20 chance of occurring, then it is unlikely to represent the population.  The null hypothesis (H0) always makes a statement of no (null) difference/relationship between the values of a population (e.g. means) or between two variables: H0 = There is no relationship between the two variables being measured.  Statistical significance and the null hypothesis The null hypothesis is used to define a population in which there is no relationship between two variables (i.e., the middle 95%). We try to decide whether or not it is possible that the sample comes from this population, defined by the null hypothesis. If it is unlikely that the sample comes from the middle 95%, the possibility that the null hypothesis is true is rejected.  p-values We use a p-value to find out the probability of our result occurring assuming that the null hypothesis is true. The null hypothesis states that there is no relationship between the two variables being studied (one variable does not affect the other). It states the results are due to chance and are not statistically significant. The alternative hypothesis states that the independent variable did affect the dependent variable, and the results are significant in terms of supporting the theory being investigated (i.e. not due to chance). How do you know if a p-value is statistically significant?  A p-value, or probability value, is a number describing how likely it is that your data would have occurred by random chance (i.e. that the null hypothesis is true). The level of statistical significance is often expressed as a p-value between 0 and 1. The smaller the p-value, the stronger the evidence that you should reject the null hypothesis.  A p-value less than 0.05 (p <.05) is statistically significant. It indicates strong evidence against the null hypothesis, as there is less than a 5% probability the null is correct (and the results are random). Therefore, we reject the null hypothesis. A p-value higher than 0.05 (p > 0.05) is not statistically significant and indicates strong evidence for the null hypothesis. This means we fail to reject the null hypothesis. You should note that you cannot accept the null hypothesis, we can only reject the null or fail to reject it.  A statistically significant result cannot prove that a research hypothesis is correct (as this implies 100% certainty). Instead, we may state our results “provide support for” or “give evidence for” our research hypothesis (as there is still a slight probability that the results occurred by chance and the null hypothesis was correct – e.g. less than 5%).  Statistical significance and correlation A statistically significant relationship is one that is unlikely to have occurred in the sample if there's no relationship in the population. The issue of whether a result is unlikely to happen by chance is an important one in establishing cause-and-effect relationships* Importantly: a correlation can be weak but still statistically significant. Meaning in plain English that the association is small, but not zero. *but it is not enough by itself. How to report a p-value in APA style  How to report a p-value in APA style Report exact p values (e.g., p =.031) to three decimal places. However, report p values less than.001 as p <.001. Use italics (p is always italicized). The opposite of statistically significant is "not statistically significant", not "insignficant".  Type I and Type II Errors Type I Error: Deciding that the null hypothesis is false when it is actually true (i.e., a false positive). Type II Error: Deciding that the null hypothesis is true when it is actually false (i.e., a false negative). In psychology, we use a very narrow threshold for statistical significance (p <.05) to reduce the likelihood that Type I or Type II errors are made.  One-Tailed and Two-Tailed Significance Testing A one-tailed test specifies the direction of the hypothesis e.g. ‘Those who attend 80% of PS219 lectures throughout term will perform better than those who do not.’ Two-tailed tests allow the hypothesis to go in either direction e.g. ‘There will be a difference in end of term grades between those who attend 80% of PS219 lectures throughout term than those who do not.’ Two-tailed tests are more widely used. Problems with p-values  Problem #1: P Hacking The inappropriate manipulation of data to produce a statistically significant result. Making up data points Removing data points Altering existing data points Running lots of statistical tests until you find one that produces a statistically significant result. Problem #2: p-values don't show absolute certainty A statistically significant result cannot prove that a research hypothesis is correct (as this implies 100% certainty). American Statistical Association: "P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone."  Problem #3: p <.05 does not imply a strong effect A p-value is a number describing how likely it is that your data would have occurred by random chance (i.e. that the null hypothesis is true). It does not measure how big the association or the difference is. To investigate the strength of a difference or association, we can look at effect sizes (more in week 10). Solla, F., Tran, A., Bertoncelli, D., Musoff, C., & Bertoncelli, C. M. (2018). Why a P-value is not enough. Clinical spine surgery, 31(9), 385-388. Quiz time!  This is a: a. Strong positive correlation b. Weak positive correlation c. Strong negative correlation  This is a: a. Strong positive correlation b. Weak positive correlation c. Strong negative correlation  This is a: a. Moderate positive correlation, which is statistically significant. b. Strong positive correlation, which is statistically significant. c. Strong positive correlation, which is not statistically significant.  This is a: a. Moderate positive correlation, which is statistically significant. b. Strong positive correlation, which is statistically significant. c. Strong positive correlation, which is not statistically significant.  What does p <.05 mean? a. Your finding is statistically significant. This means there is strong evidence against the null hypothesis, as there is less than a 5% probability the null is correct. b. Your finding is statistically significant. This means there is strong evidence against the null hypothesis and that the alternate hypothesis is correct. c. Your finding is not statistically significant. This means there is not sufficient evidence against the null hypothesis, as there is a more than 5% probability the null is correct.  What does p <.05 mean? a. Your finding is statistically significant. This means there is strong evidence against the null hypothesis, as there is less than a 5% probability the null is correct. b. Your finding is statistically significant. This means there is strong evidence against the null hypothesis and that the alternate hypothesis is correct. c. Your finding is not statistically significant. This means there is not sufficient evidence against the null hypothesis, as there is a more than 5% probability the null is correct.  Recommended Reading Chapter 10: 'Samples & Populations' Chapter 11: 'Statistical Significance'  Thank you! For questions, email me at [email protected]