Survey Design & Analysis in Psychology - 3003PSY PDF

3003PSY Survey Design and Analysis in Psychology FUNDAMENTALS OF LONGITUDINAL DESIGNS  Determinations of causation are based on three criteria: SURVEYS AND 1.The constructs need to covary; CAUSATION 2.There must be temporal precedence, and; 3.No confoun...

3003PSY Survey Design and Analysis in Psychology FUNDAMENTALS OF LONGITUDINAL DESIGNS  Determinations of causation are based on three criteria: SURVEYS AND 1.The constructs need to covary; CAUSATION 2.There must be temporal precedence, and; 3.No confounding factors (i.e. alternative explanations need to be ruled out).  In cross-sectional research, temporal precedence can never be directly empirically tested. The ‘causal’ direction is inferred by TEMPORAL allocating a variable as the criterion (DV), PRECEDENCE and others as predictors (IVs). This suggests the IVs ‘cause’ the DV. Theory and previous empirical findings are used to justify the direction of the statistical analysis.  Panel designs are the gold- standard technique in longitudinal survey research. In a panel design, the same PANEL participants complete the same questionnaire over DESIGNS multiple time points. This ensures that both the predictor and criterion variable are measured at multiple time points. STABILITY AND CHANGE The degree of consistency in scores, means, Stability or rank orders from one time point to another. The degree of fluctuation in scores, means, Change or rank orders from one time point to another. In what ways and how much do things stay the same over time versus how much do they change. Often tested as baseline (the initial measurement) to re-test. DIRECTION OF THE ASSOCIATION Uni-directional relationship: There is a clear direction in the relationship between the predictor and criterion variable. A uni-directional relationship in a well-designed longitudinal study provides support for temporal precedence Bi-directional relationship: Occurs when the predictor variable is related to the criterion variable, and the criterion variable is related to the predictor variable. In this instance, it is not possible to conclude that one variable occurred prior to the other, so temporal precedence cannot be determined. Rather both variables ‘cause’ one another… Analysis of Longitudinal Survey Data using Regression OVERVIEW OF ANALYTICAL TECHNIQUES Simplex models Mini-lecture 1 Longitudinal Correlations Residualised Longitudinal Regression Mini-lecture 2 Cross-lagged models Mini-lecture 3  Simplex designs, also known as autoregressive designs, involve regressing a variable on itself across time. In other words, the measurement of a variable at time 1 should predict time 2 (i.e., stability). SIMPLEX This type of model is called DESIGNS autoregressive as the values of a scale are automatically regressed onto the same scale. This type of analysis allows researchers to explore the stability and change in one construct.  SIMPLEX DESIGNS A perfect association between the two time-points indicates that individual’s relative standings on the construct have not changed. The rank-order of the participants remains the same. Participants with high T1 scores, will have high T2 scores (i.e., there is a high degree of stability from T1 to T2) SIMPLEX DESIGNS: STABILITY A large autoregressive coefficient can mean one of two things*: 1. Individuals do not change over time. 2. Individuals uniformly increase or decrease over time. *There are other, more complex and less common explanations, that are not covered in this course. 1. Individuals do not change over time ID Time 1 Time 2 1 2 2 2 5 5 3 4 4 4 3 3 5 1 1 6 1 1 7 2 2 8 3 3 2. Individuals uniformly increase or decrease over time. Scale has increased, denoting participants have higher scores at T2. ID Time 1 Time 2 1 2 4 2 5 7 3 4 6 4 3 5 5 1 3 6 1 3 7 2 4 8 3 5 SIMPLEX DESIGNS: STABILITY Across each of these explanations, the rank-order of the participants have not changed. Therefore, on a between-participant level (inter-individual), there is stability Participant T1 T2Example 1 T2Example 2 Rank Order 2 5 5 7 1 3 4 4 6 2 4 3 3 5 3 8 3 3 5 3 1 2 2 4 4 7 2 2 4 4 5 1 1 3 5 6 1 1 3 5  SIMPLEX DESIGNS A small or zero association between the two time-points indicate that individual’s relative standings on the construct have changed dramatically (i.e., there is a lot of change from T1 to T2). SIMPLEX DESIGNS: CHANGE Time 1 Time 2 2 16 5 5 4 19 3 2 1 8 1 13 2 23 3 13  SIMPLEX DESIGNS: CHANGE With a low autoregressive association, it indicates a rank-order change in the participant’s scores from time 1 to time 2. Thus, there is change between the individuals. Participant T1 T1 Rank T2 T2 Rank Order Order 1 2 4 16 3 2 5 1 5 6 3 4 2 19 2 4 3 3 2 7 5 1 5 8 5 6 1 5 13 4 7 2 4 23 1 8 3 3 13 4  SIMPLEX DESIGNS Time 1 Time 2  SIMPLEX DESIGNS Stability Change Time 1 Time 2  SIMPLEX DESIGNS Strong relationship, temporal Weak relationship, temporal stability. change. Stability Stability Change Change SIMPLEX DESIGNS: EXAMPLE 2384 children were followed from Grade 1 to 6 and had their literacy grades measured. VO stands for vocabulary. Verhoeven & Leeuwe (2008). Prediction of the development of reading comprehension: A longitudinal Study. Applied Cognitive Psychology, 22, 407-423  SIMPLEX DESIGNS: EXAMPLE The strong and significant associations between each grade indicates that the relative standing of the children remains the same. In other words, children with higher vocabulary in Grade 1 compared to their classmates had high vocabulary across every subsequent grade in comparison to their classmates. Based upon this information alone, the researchers couldn’t conclude why the relative standing is stable. 1. Children’s VO doesn’t change across primary school. 2. Every child’s VO increases uniformly.  SIMPLEX DESIGNS: EXAMPLE In addition to examining the autoregressive coefficients, the researchers also examined the average score on the VO measure. Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 VO 38.59 43.13 93.38 104.97 108.91 114.01 By exploring the average for all participants in the sample, you can examine group-level growth. The increase in average VO scores indicates that average vocabulary levels are increasing over time. This rules out the possibility that the high auto-regressive coefficients indicates no growth. It does not distinguish between the possibility that there is uniform growth, or that there is intra-individual differences in growth.  LONGITUDINAL CORRELATIONS Longitudinal correlations examine the relationship between the IV at time 1 and the DV at time 2. Similar to the correlations and bivariate regression techniques covered in this course, only two variables are analysed. LONGITUDINAL CORRELATIONS Time 2 Time 1 LONGITUDINAL CORRELATIONS In order to examine temporal precedence, researchers can run two separate longitudinal correlations. If the relationship If the between relationship IV atIVtime between 1 and at time DV DV 1 and at time 2 is 2significant at time is ANDsignificant the relationship between AND the the DVbetween relationship at time 1the with DVthe at IV at 1 time time with2 is not significant, the IV at ittime can2be is argued that temporal not significant, it canprecedence has temporal be argued that been found. precedence has been found. If both analyses are significant, it may indicate a bi-directional relationship. LONGITUDINAL CORRELATIONS If the relationship between IV at time 1 and DV at time 2 is significant AND the relationship between the DV at time 1 with the IV at time 2 is not significant, it can be argued that temporal precedence has been found. LONGITUDINAL CORRELATIONS Weaknesses This analysis does not account for correlations between variables at each time point. Does not account for the stability in a construct. There is no indication of the stability or change in the construct over time. The combination of these limitations ensure researchers cannot rule out that the relationship between T1 and T2 is simply due to a cross-sectional relationship.  SUMMARY uWe use longitudinal designs when collecting survey data in order to test for temporal precedence uWe collect data for our variables at each time point uSimplex designs allow us to establish stability and change uStrong autoregressive associations show stability; weak autoregressive associations show change uSimplex designs do not establish temporal precedence uLongitudinal correlations have been used to try to establish temporal precedence but not stability nor change