Book Summary: Types of Research PDF

Book summary Chapter 1: Types of research Hypothesis generating vs. Hypothesis testing A hypothesis needs to be narrowed down as far as possible to show what will be tested and what outcomes count as support or counter evidence. An important principle in findi...

Book summary Chapter 1: Types of research Hypothesis generating vs. Hypothesis testing A hypothesis needs to be narrowed down as far as possible to show what will be tested and what outcomes count as support or counter evidence. An important principle in finding evidence is the principle of falsification. We will always try to prove that our hypothesis is wrong. Generalizability refers to the extent to which findings in a study can be generalized to a population that is larger than the samples tested. Statistics helps assess how representative a sample is and how reliable it is to make generalizations of the findings of the sample to a larger population. Case studies vs. group studies In group studies, the similarity between the group us crucial. The group is selected with specific characteristics. There are two problems that can arise: 1. When there are more differences than similarities between the individuals within the group. 2. When the groups represent rather random selections of a continuous phenomenon Case studies are often used of qualitative analyses, in which a detailed description can be derived from several data of a single person or from interaction between two people. Case studies can provide the opportunity to observe human behaviour in real life context. Observations in case studies cannot be generalized to a larger population of similar individuals, but case studies can be used for generalization to the theory. Theoretical principles can be falsified when not observed in case studies. Case studies use a holistic approach. Description vs. Explanation Explanation of variance is expecting a difference between two groups but not between the individuals of a group. Then the difference between thee groups can then be seen as the reason for the difference.  Non-experimental vs. experimental The choice of an experimental or a non-experimental approach largely depends on what a researcher wants to know. Experimental research is a type of research that uses controlled experiments and statistical manipulations. It aims to decompose complex processes into parts that can be studied and manipulated in a controlled environment. This method provides “hard” evidence. It is useful when studying phenomena that require controlled conditions. Non-experimental research relies on more interpretative methods. It is suited for studying aspects that are better understood through observation and analysis of natural, spontaneous behaviour. Process research vs. product research Product research is a type of research that works with outcomes and variables at one moment in time. The conclusions relate to the outcomes of a process, rather than the process itself. Process research emphasizes the change of development over time and focuses on the dynamic interaction of factors affecting the language system, or the overall system in general, over time. Longitudinal vs. Cross-sectional Longitudinal research is research in which individuals’ development over time is studied. In many longitudinal studies, there are more moments of measurement with smaller time intervals. They are small but have major impact on the field, due to the time it takes. Due to time/money problem, cross-sectional design is used. Here, individuals in different phases of development are compared at one moment in time. Both designs have their issues. Longitudinal studies: ☁ The number of participants is in general very small because a large corpus of data is generated ☁ Small numbers mean that the findings may be highly idiosyncratic and difficult to generalize ☁ Subject mortality. Individuals may drop out or die. Cross-sectional studies ☁ Assuming the selected groups for comparison behave like one another ○ Solved by cohort effect. By looking at a certain group in different years  Qualitative vs. quantitative Qualitative: Research in which the focus is on naturally occurring phenomena, and data are primarily recorded in non-numerical form. It is holistic and tries to integrate as many aspects that are relevant into one study. It is interpretative. Issues: Lack of objectivity → interpretation of researchers Quantitative: Research in which variables are manipulated to test hypotheses, and in which there is usually quantification of data and numerical analyses. Issues: Clashing with quantitative: because “they try to explain complex phenomena by calculating the influence of a limited number of factors”. It is not always clear what participants actually do. In-situ/naturalistic research vs. laboratory research Naturalistic research refers to research that studies a phenomenon in its normal, natural setting and in normal everyday tasks. More ecologically valid, it focuses on the tasks in their normal setting. Laboratory research refers to both isolating a phenomenon from its normal setting and to the use of data that are an artifact of the procedures used. It aims to find “pure” effects that are not tainted by the messiness of everyday life. Issue: It doesn’t reflect reality Chapter 2: Variables Operationalization: converting the abstract phenomenon, or construct we want to investigate, into a measurable variable. Variable: any observation that can take on different values Attribute: a specific value on a variable The translation of a construct into a variable is always the researcher’s own choice; and the validity of the outcomes of the investigation may strongly depend on it. If the variable resulting from the operationalization does not adequately represent the underlying construct, the entire study may be questionable. Variable type is also called variable scale.  Variable scales Name (in R) Explanation Calculable? Example Interval (numeric) Numbers Yes Temperature (0 is not 0) Nominal (factors) Words No Male / Female Ordinal (integer (no decimal)) Ranked levels, but no certain No Hotness in peppers interval Ratio Numbers with fixed interval Yes Weight in Kilograms (0 is no AND fixed zero point weight) It is important to categorize the variables, since it influences the choice of operationalizing a construct. Because a particular type of variable will have significant consequences for the calculations that can be performed in the analysis of the results of a study. Questions to ask to help find the category 1. Is there a ranking? a. No → nominal b. Yes → Ordinal or interval/ratio 2. Can you identify the interval? How much? a. No → Ordinal b. Yes → Interval/ratio Findings of studies only count if other researchers can also find the same results if the study is reproduced. Two key requirements: Validity: closely related to operationalization. The link between the statistical study and the real world. A valid study incorporates measures that actually measure what the study intended to measure. Is the procedure really measuring what you think it’s measuring? Reliability: the internal consistency of a study. Closely related to the issue of replications. Will your measures generate similar results again? Chapter 3: Descriptive statistics Descriptive statistics: describes the characteristics of a particular dataset or to describe patterns of development in longitudinal analyses. Descriptives are used to describe observations for two different kinds of perspectives on data: means and relationships. Inductive statistics: helps in decision-making about the results of the study. Hypothesis testing, in which the results of a small sample are generalized to a larger population. If we want to make generalizations about groups, we want the groups to be as homogeneous as possible. Calculating the mean Mean: used when comparing different groups. Gives insight to the characteristics of that dataset. It constitutes a summary of the real dataset. It does not give the full picture of the full dataset. Relationships: used when you want to say something about the relationship between variables measured within a group. Σ𝑋 𝑋= 𝑁 𝑋 = 𝑚𝑒𝑎𝑛 Σ = 𝑠𝑢𝑚 𝑋 = 𝑖𝑡𝑒𝑚𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑑𝑎𝑡𝑎𝑠𝑒𝑡 𝑁 = 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑖𝑡𝑒𝑚𝑠 Mean calculation cannot happen for ordinal or nominal datasets. Boxplot Boxplot: summarizes the data points for a group of individuals taken together. Shows the median, extremevalues and the interquartile range of a dataset. Scatterplot: shows a relationship between all available data points Mode: The value that occurs most Median: Middle point of the dataset. Range: take the highest value and subtract the lowest value. Interquartile range: Sometimes the range is not very representative of the dataset. The last 25% is then cut off on either side of the dataset. Step 1: Divide the data into four equal parts. Step 2: find the median for all the four parts Lower Quartile: Median of the lower half Upper quartile: median of upper half Interquartile range: range between the lower and upper quartile.  Standard deviation Standard deviation represents the amount of dispersion for a dataset. From the SD and the mean, a clear picture of that dataset will emerge. It shows how far the variables are away from the mean. Or how similar or different the individual values of the dataset are. It can be used to provide information about how an individual item in the dataset is related to the whole dataset. 2 Σ(𝑋−𝑋) 𝑆𝐷 = (𝑁−1) 𝑋 = 𝑚𝑒𝑎𝑛 Σ = 𝑠𝑢𝑚 𝑋 = 𝑖𝑡𝑒𝑚𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑑𝑎𝑡𝑎𝑠𝑒𝑡 𝑁 = 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑖𝑡𝑒𝑚𝑠 Calculating the standard deviation: Step 1: Calculate the mean Step 2: Determine how far each value is away from the mean Step 3: Square all numbers determined in step 2 Step 4: Add all numbers from step 3 Step 5: Divide the outcome of step 4 by the amount of items -1. Step 6: Take the square root of step 5. You can calculate how many SD’s an individual score is away from the mean. Z-score: the number of SD’s for an idndivual item. It gives immediate insight into how an individualscore must be valued relative to the entire dataset. 𝑧 − 𝑠𝑐𝑜𝑟𝑒 = (𝑠𝑐𝑜𝑟𝑒 − 𝑡ℎ𝑒 𝑚𝑒𝑎𝑛)/𝑆𝐷 Min-max graph Min-max graph visualizes the degree of variability by plotting the minimum observed value and the maximum observed value of many observations of single individuals over time. Instead of comparing the means of the groups, we look at how the mean value of a variable changes over time. Frequency distribution The mean and the SD do not tell us how often certain values occur and how they are distributed. Graphically, a frequency distribution can be displayed as a histogram or line graph. If there are many data points, the frequency distribution will typically result in the same bell-shaped line graph that is commonly referred to as a normal distribution (Gaussian distribution). Normal distribution Common characteristics: mean, mode and median largely coincide. The extreme low and high values occur much less frequently than the scores around the mean. The number of scores that occur in the section related to the standard deviations is always approximately the same. 1 SD away from mean: 34.13% 1 and 2 SD away from mean: 13.59% Higher or lower than 2SD away from the mean: 2.14% Skewness: normal distribution is offset to the left or right Kurtosis: how narrow or wide the distribution is. Chapter 4: Statistical logic Inductive statistics: helps generalziing the findings for a sample to a larger population.  Independent and dependent variables Independent variable (grouping variables, factors, predictos varibales): variables that are systematically changed to investigate the effect of the independent variable on a dependent variable. Dependent variable: variable that is measured at that is expected to be affected by the independent variable. The dependent variable changes as a result of the independent variable. What is measured is normally the dependent variable. Control variable: The variable that does not change. Constant variable. Errors Statistics is all about estimating the change of making the wrong decision, and about comparing a limited dataset to a larger group of people. ⍺-error: the error of assuming that something is true which in reality is not true. Incorrectly rejecting the H0. Ꞵ-error: the error of assuming something is not true which in reality is true. Yes No Excluded OK ⍺-error Not excluded Ꞵ-error OK Hypotheses Form a hypothesis about the relationship between the dependent and independent variable. The main hypothesis that is tested is the null hypothesis. Null hypothesis (H0): there is no difference between the dependent and independent variable. Alternative hypotheses (H1 and H2): Other possible hypotheses. It is impossible to prove a hypothesis right, however it is possible to prove a hypothesis wrong. Principle of falsification: proving a hypothesis wrong. To avoid the possibility that we reject the H0 incorrectly, we try to calculate the degree of chance that might have been involved in obtaining these scores by chance. We want to calculate the exact probability of committing an ⍺-error. Yes No  Excluded OK ⍺-error Not excluded Ꞵ-error OK Significance (p value) p-value: quantifying the probability of making an alpha error or significance. ⍺-error has to be 5% or less. (⍺ = 0.05). (Unless you’re doing medicine, than 5% can be way too much 😅 ) It is up to the researcher to set the alpha level. A significant result is one that is acceptable within the scope of the statistical study. Example: “a statiscitical analysis showed that the difference between the two groups was significant”,p = 0.02. Interpret it as “I accept that there is a difference between the two groups, and the chance that I have made an error in assuming this is approximately 2%”. You can never fully say that a hypothesis is true; instead you can express the chance of making the wrong decision. If it is smaller than 5% (or 1%) it is taken to be acceptable. One or two-tailed One-tailed: only one alternative hypothesis is needed Two-tailed: there can be more than one alternative hypothesis. One-tailed is easier, since you can ignore an entire side of the normal distribution. Keep in mind that allowing for a 5% chance of incorrectly rejecting the null hypothesis, since it is all on one side. Two-tailed hypothesis are preferred, except when it is impossible to do so. Population sample A sample needs to be representative for the population that we want to investigate. Two ways to ensure that the sample is representative. 1. The sample is selected purely at random from the entire population. This is very difficult to achieve 2. Focussing on a very specific subpopulation (stratum) by including all possibly interfering factors as control conditions. Be aware that it may not always be completely representative of the population. quasi -experimental: studies that are done without a randomly picked sample. It is crucial to take a sufficiently large enough sample. The stronger the effect (the bigger difference between groups or the stronger the relationship between variables), the smaller the sample needed to demonstrate that effect in a statistical study. Power of a study: The strength of the effect in relation to the sample size. It is related to the number of participants in the sample. Greek symbols are used for the population σ = standard deviation μ = mean ⍺ = chance of making the wrong decision Latin is used for samples M = mean 𝑋 = 𝑚𝑒𝑎𝑛 P = chance of making the wrong decision SD = Standard deviation Standard error: expressing how much a sample deviates from a population. σ 𝑆𝐸 = 𝑁 σ = 𝑚𝑒𝑎𝑛 𝑁 = 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑖𝑡𝑒𝑚𝑠 Degrees of freedom Degrees of freedom (df): a necessary correction that is needed for the calculation of the probability of the sample and is related to the number of “free choices” that can be made. Degrees of freedom is related to the total size of the sample. The bigger the sample size, the larger the df, the more lenient we can be in rejecting the H0 and the more relevant is the value. The df is determined by taking the total numer of participants in the group and subtracting the numbers of levels of the independent variable (the number of groups). Parametric or non–parametric Parametric test consists of numeric data. Non-parametric tests are stricter and take data in account that are not normally distributed or of ordinal nature. It can only be allowed when the data is:  Consists of an interval dependent variable and independent observations Approximate the normal distribution Comply with homogeneity of variance ○ Refers to the similarity of variation in the data. Linearity when assessing relationships. Variance is similar to the SD: it is the total sum of the deviation from the mean. The nation for variance is s2 or σ2 the variance across the dataset and the different groups must be about the same. If one SD is more than twice as big as that for another group, then you know that the variance is non homogeneous. Make sure the relationship between variables is linear. It does not have to follow a straight line, but it should be monotonic. It should go up or down, but not both. If the data does not match one or more of the points mentioned above, we will need to apply non-parametric tests. They do not calculation the mens. They are less powerful and harder to interpret. Effect sizes Conventionally accepted beta error is 20%. We want to be 80% certain that there that effect is really to be found in the experiment. The smaller the effect, the larger the sample needs to be in order to actually find the effect. Trying to find a small effect with a limited number of participants may be a waste of time. Looking for a large effect is a waste of time trying to find it in a large sample. Null Hypothesis Significance testing (NHST) Issues with NHST: All-or-nohting approach of significance. ○ A result is either significant or not. ○ A more realistic method is to report the confidence interval (CI) in addition to the p-value. ○ Use the P-value in research reports with caution or in combination with the relevant CI. The interpretation of alpha. ○ While we want to test if H0 is true, we are testing with NHST is the probability of finding the data in the sample if the H0 is true prior to the experiment. The probability pof H0 can change from 5% to 60% when the probability prior the the experiment is included in the calculation. Two solutions:  1. Include the chance of finding H0 prior to the experiment in the calculations. This is done by the Bayesian statistics. They allow for continuously updating the probabilyes of an effect really existing as soon as new information is added. 2. Replicating studies with different samples and in different contexts. When something is “significant” it is not necessarily true. A higher significance does not mean that it is more true. Important to remember: ！ Always report on descriptives combined with visualizations of the data and use common sense in interpreting them. ！ Always report on the effect size of a study, which should be leading in the interpretation of the result. Steps in statisticical studies 1. Operationalize Operationalize the constructs into variables and determine the scale for each variable. Determine which is the independent variable and which is the dependent variable and the control variable if there is one. 2. Hypothesis formulation Form your hypotheses. Apart from the H0 also set up the alternative hypothesis. 3. Sample selection Select a representative sample and subsequently collect data. Be wary of dangers: Self-selection: when participants voluntarily choose to participate in the study, these may be volunteers who already excel in the tested variable/skill. Researcher expectancy: researchers influencing the procedure due to the expectations they have Subject expectancy: participants want to please the researcher and respond in a desirable way rather than being objective. Observer’s paradox: when participants know what is being researched they may adjust themselves to be more disable. Thus it is better not to reveal the purpose of the study. 4. Conduct and interpret statistics First consider the descriptive statistics (calculate the mean, range, SD, etc.) and create visualizations. Then, using the outcomes, we can determine the probability of finding the effect if the H0 is true. Chapter 5: Assessing relationships and comparing groups *the APA citation format explicitly requires researchers to include effects sizes in their scientific papers. Relationships between interval and/or ordinal variables normally distributed Correlations: relationship between two variables. Correlation coefficient: the coefficient that tells us how much they differ. Measured on an interval scale, using Pearson r or ryx. ryx runs from -1 to 1. If ryx = 0 than there is no correlation. 0.00 - 0.19: very weak 0.20 - 0.39: weak 0.40 - 0.59: moderate 0.60 - 0.79: strong 0.80 - 1.0: very strong A positive correlation means that if one variable goes up, the other does too. Negative correlation means that if one variable goes up, the other goes down. Pearson r also reports on the significance of the correlation, meaning the estimated chance of incorrectly rejecting the null hypothesis and the CI 95%. The CI provides two values showing that - if we were to repeat the correlation analysis with different samples - the correlation coefficient would lie somewhere between the estimated upper and lower bound CI values approximately 95% of the time. Pitfall of correlation studies: a correlation between two variables is seen as a causal relation. The correlated variables are not in a dependency relation. The distinction between dependent and independent variable is not relevant or correlation studies.  Relationships between interval and/or ordinal variables not normally distributed When the numbers are not normally distributed, there can still a correlation calculated that is baes on mean rank orders. Spearmans Rho (⍴): similar to Pearson r; for larger samples. Kendall’s Tau (τ): useful for smaller samples with identical scores. Correlation is useful when assessing the reliability of an experiment or exam. Meaning: does your experiment or exam what it claims to do? Three ways to test. 1. Test-retest reliability: when students would perform the same if they had to take the exam 2. When students who are equal receive the same score. 3. When better participants outperform the lesser ones on every question Measuring using split-half correlations, using Cronbach’s Alhpa. It takes all the items separately and relates all scores on each item to the scores on the other items. High outcome: items are highly correlated Low outcome: low correlated. Correlation and effect sizes R-value: an expression of the strength of the relationship. Sometimes squared to obtain an effect size that is related to the amount of variance that can be explained by the variables in our experiment. 2 Small effect 𝑟 = 0. 1 (𝑟 = 1% ) 2 Medium effect 𝑟 = 0. 30 (𝑟 = 9% ) 2 Large effect 𝑟 = 0. 50 (𝑟 = 25% ) Associations between nominal variables Analysis design when there are no interval variables at all. Spearman or Kendall correlation can be carried out when using nominal scales and does not have a normal distribution or straight line. In all other cases, statistics cannot be used and therefore we count number of occurrences. Frequency analysis: counting number of occurrences. It is used to measure the strength of the potential association between nominal variables.  2 Chi-square (χ ) analysis:calculates the number of occurrences in a particular cell relative to the margins of that cell. The total number of instances of each construction and the total number of 2 participants in each group. The value of χ runs from 0 to 10. Three assumptions for chi-square 1. Nominal variables only 2. Requires the assumption of independence of observations to be met 3. The expected frequencies should not be too low Reported in Phi or Cramers V as a measure of effect size. Phi (φ): is used for variables with 2 levels Cramers V: variables with more than 2 levels. Are measures between 0 and 1. The closer it is to 1 the stronger the associations between the two nominal variables. The t-test It can be used only to evaluate interval data of two groups representing one nominal independent variable. It uses the following descriptive statistics Number of participants per group Mean scores per group standard deviation per group The outcome of this calculator expresses the magnitude of the difference between the groups, taking into account the variation within the group and the group sizes. Large difference between groups = higher t values. Large variance = smaller t. The closer the to zero t is = weak difference between the groups  Chapter 6: simple and multiple linear regression analyses Regression analysis Sometimes we want to test the difference between groups or determine rhe relationship s between two variables, but sometimes we want to predict the outcome based on one or various endependent variables. We laso might want to know what the exact influence is of the independent variable and, in case of multiple predict variables, which of these contributes most to the outcome. This is called regression analysis. Regression: can establish the strenght of a relationship between two numeric variables. It allows us to compare the means of two or more groups. It allows us to combine what we can do with correlations and mean comparisons and can be seen as a technique use to predict the vlaue of a certain dependent variable as a fucntion or one or more nominal and/or continuous independnt varibales. Mulptiple regresssion allows us to examine which of these variables contributes most to the outcome by controlling for the effect of the other variables. Assessing relationships and simple regression Slope: steepness of a line. Finding the number of the slope is the aim of performing regression analysis. It tells us how much a change in a predictor variable (x) affect the value of the outcome variable (y). When performing a regression analysis, the H0: the slope = 0. Ha: the slope is not 0. Intercept: the point at which the line crosses the y-axis. Helps us characterize and summarize a straight line. The straight line in a scatterplot is used to summarize the effect of a predctor variable (x) on an outcome (y). The idea of regression analysis is that it can be used to predict the outcome as a function of one varibale (simple regression) or multiple variables (multiple regression). Find out how much this predictor variable contributes to the variabence in our dependent, or outcome, varibale. How to create a regression line 1. We need the startinpoint (constant or b0). 2. Understanding the steepness of the line (b1) Deviation of a particular person from the model is referred to as the error. 𝑌 = 𝑏0 + 𝑏1𝑥 + ε Or in words: outcome = startingpoint + slope + deviation Residuals: the difference between the actual data and the data predicted by the model. the error-term in the equation abvoe, which are siple the deviations of each data pint from the model and very similar to the deviations away from the mean that we used to calcuate the SD. Important as they tell us something about ho w well the model fits the data. Thelower the better fit. r2 : gives us a measure of how much of the variation in the dependent varibale can be explained by variation in the independent variables and it is provided in the output. For multiple regression models it is better to look at the adjusted R-scquared value instead. This squalred version has been adjusted for the number of predcots that we added. Regression model: the straight lines plotted through the data points representing the best fitted lines. The sum of differences is often used to assess how well the linear line fits the data. Least squares approach: The smaller the squared differences, the better and more representative a model is. Goodness of fit Information can be found in the coefficitents table. It provides: Estimate of the intercept The slope T-value ○ Corresponding p-value ○ It shows whether the intercept is significantly different fom 0. Multipe linear regression In multiple regression analysis we find out what the most important predictor variables are in our study and howmuch they contribute to the variabece of our data.. We can add multiple independent variables that are interval, but we can also add categorical independent variables to predict the value of our dependet variable. We can compare the size of the effects directly by looking at standardized coeffictientcs. These standardiezed (beta coefficients) reveeal how many SD’s the outcmoe changes if the predictor varibale changes with one SD as well. They are beased on standaradezid values of the variables involved and measures in dhe same unit, SD’s, the can be directly compared; the higher the standardized beta coefficitent the strong the effect of that specifc variable. Assumptions for regression and nonparametric alternatives If the data points all reoughtly follow a straight line it can be assumeds that the relationship is linear. Two other assumptions can be made: the homoscedasticy of variance and multicollinearity. Homoscedasticy : assumes that the residuals or errors (the differences between the observed values and the ones fitted by the models)vary constantly. Multicollinearity: a situation in which multiple varibales relate to the same underlying varible. Errors in regressino whould be normally distributes. Chapter 7: Additional statistics for Group Comparisons When comparing three or more groups, you cannot use the t-test. You will have to use ANOVA. Always run a post-hoc test (Turkey Honest Significnat Differences (HSD)). THis provides corrected p-values. Factorial ANOVA Two way factorial ANOVA: you are looking for differences between the levels of each of the independent variables. Main effect: the efffect that one independent variable has on the dependent variable without looking at the differences caused by any of the other independent variables. Interaction effects: The different effects of one independent variable on different levels of another independent variable. Can only interpreted on the basis of graphs.  Kruskal-wallis H test INstead of using means, these non-parametric alternative often base their p-values on the ranked data instead. Using the ranking system, we could also number all ranks form lowest to highest and then calculate the sum of ranks. If there is nodfference, on eor two of the groups should have a hgigher sum of ranks than the others. In many designs we inlcude so-called within-subjects, where the independent variable is called the within-subjects factor. We are testing that variable are violating the assumption of independence of the observatsions. With the repeated measures ANOVA or the mixed tdesign you can account for dependence among measurements and this ANOVA can be seen as an extension of the paried or dependent samples t-test.