PSY2801 Nov 14 - Measurement PDF

Week 11: Measurement www.5lovelanguages.com/quizzes/  “Love Languages” Sample Items It’s more meaningful to me when: I’m complimented by my partner on my appearance My partner takes the time to listen to me and really understand my feelings It’s more mea...

Week 11: Measurement www.5lovelanguages.com/quizzes/  “Love Languages” Sample Items It’s more meaningful to me when: I’m complimented by my partner on my appearance My partner takes the time to listen to me and really understand my feelings It’s more meaningful to me when: I receive a loving note/text/email for no special reason from my loved one My partner and I hug  Love Languages as a Likert Scale It’s meaningful to me when: 1. Not at 2 3 4 5 6 7. Very All True True I’m complimented by my partner on my appearance My partner takes the time to listen to me and really understand my feelings I receive a loving note/text/email for no special reason from my loved one My partner and I hug  Love Languages as a Likert Scale It’s meaningful to me when: 1. Not at 2 3 4 5 6 7. Very All True True I’m complimented by my partner on X my appearance My partner takes the time to listen to X me and really understand my feelings I receive a loving note/text/email for X no special reason from my loved one My partner and I hug X  Love Languages as a Likert Scale It’s meaningful to me when: 1. Not at 2 3 4 5 6 7. Very All True True I’m complimented by my partner on X my appearance My partner takes the time to listen to X me and really understand my feelings I receive a loving note/text/email for X no special reason from my loved one My partner and I hug X  Measurement Operationalizing Your Variable How do you choose your measurement? Reliability Does your measurement consistently measure the same thing? Validity Does your measurement consistently measure the right thing? Operationalizing Your Variables What are we measuring? Variables, and particularly constructs Defining your variables Conceptually vs. operationally Types of measurement Self-report, behavioral, physiological How do you choose? Previous research, theory, methodological advances, and feasibility  We are measuring variables: things that vary E.g., age, gender, shoe size, extraversion, aggression, depression, intelligence, the number of sheep in the Scotland highlands… What are We Measuring?  Variables that cannot be observed directly E.g., traits, emotions, attitudes, abilities Constructs “Our kids are sponges and when they leave here, what they listen to now will affect them the rest of their lives” Relevant Constructs Predictor variable: “What they listen to” Let’s call it musical exposure Dependent variable: “Will affect them” More difficult. Wellbeing? Curiosity? Creativity? Musical taste in adulthood? Let’s go with Openness to Experience Musical exposure in preschool predicts openness to experience in adulthood Conceptually Define Your Construct What do you think this construct is? Openness to experience: the personality trait of being intellectually curious, creative, and imaginative. Musical exposure: ??  Poll Time Hypothesis: Musical exposure in preschool predicts openness to experience in adulthood. How would you define “musical exposure”, conceptually?  How will this construct can be measured? To operationalize a variable is to concretely specify how it will be measured or manipulated Operationally Define Your Construct Every Variable in Your Study Must Be Operationalized Openness to experience has an established scale we can use  The Big Five Inventory (BFI; Goldberg, 1993) ___ Is inventive ___ Is ingenious, a deep thinker ___ Is original, comes up with new ideas ___ Likes to reflect, play with ideas ___ Is curious about many different things ___ Has few artistic interests ___ Has an active imagination ___ Values artistic, aesthetic experiences ___ Prefers work that is routine ___ Is sophisticated in art, music, or literature  A person’s mean score on the 10 openness items from the BFI scale (two items reverse-scored) Operational Definition of Openness Every Variable in Your Study Must Be Operationalized Musical exposure does not have an established measure So, we must produce one ourselves based on our conceptual definition Operationalizing “Musical Exposure” Amount of exposure? Number of hours spent listening to music per week Range of exposure? Number of different musical genres available in household Method of exposure? Time spent listening to musical recordings vs. live performances Musical knowledge? Performance on a music recognition quiz Operationalizing Your Variables What are we measuring? Variables, and particularly constructs Defining your variables Conceptually vs. operationally Ways to measure your variables Self-report, behavioral, physiological Research question: Is Black Mirror more stressful than a nature documentary?  After each clip, ask participants: How do you currently feel? (1 = Not at all, 5 = A lot) Tense At Ease On Edge Etc. Some Types of Measurement Self-report measures Interviews or questionnaires People report their beliefs, behavior, history, etc… Watch video-recordings of people watching the two shows, code for stress cues Nail-biting Tense expressions Putting their hands over their faces Etc. Some Types of Measurement Behavioral measures Observations of behavior Could be naturally occurring (e.g., flirting in a bar) …or lab-induced Have people wear heart monitors; record their heart rate as they watch each show Some Types of Measurement Physiological measures Assessment of bodily states E.g., brain imaging (fMRI, PET); heart rate Operationalizing Your Variables What are we measuring? Variables, and particularly constructs Defining your variables Conceptually vs. operationally Types of measurement Self-report, behavioral, physiological How do you choose? Previous research, methodological advances, feasibility Research Tradition How was this variable measured in previous studies? Part of why doing a literature review is so important Methodological Advances New measurement options can become available with new technology Feasibility Resource limitations (e.g., time, money) may constrain your choice  Measurement Operationalizing Your Variable How do you choose your measurement? Reliability Does your measurement consistently measure the same thing? Accuracy No measure is going to be completely accurate E.g., scale will be slightly off, questionnaire scores won’t be identical… True score – the “real” score on the variable Obtained score – the score the measure gives Measurement error – difference between true score and obtained score. Want to minimize measurement error  Does your measure give consistent results under the same conditions? E.g., if nothing changes: Scales should give the same weight Questionnaire results shouldn’t change if taken twice Reliability How do you test the Test-retest reliability reliability of a self-report Parallel-forms reliability (if applicable) measure? Internal consistency  Test it with test-retest correlation Same test is given twice with some time in between. Good for stable qualities (e.g., personality), not temporary states (e.g., mood) Test-Retest Reliability  Different forms of the same test used Parallel-Forms Reliability  Test it with split-half correlation Top-half of questionnaire is compared to the bottom-half Test it with Chronbach’s alpha Tests how all the items are intercorrelated Internal Consistency Chronbach’s Alpha: A Score Between 0 and 1 Chronbach’s α Internal Consistency Higher than.90 Excellent.90 to.80 Good.80 to.70 Acceptable.70 to.60 Questionable Lower than.60 Poor  “I have trouble “My mind goes “I panic before and “I get superstitious sleeping the night blank during a test” during a test” about tests” before a test” Student 1 4 5 4 1 Student 2 2 1 1 3 Student 3 1 2 1 4 Student 4 3 4 4 2 Student 5 5 5 4 3 Reliability of all four items: α =.60 Reliability of first three items: α =.96 How do you test the Interrater reliability reliability of observational measures?  “Student exhibited signs of nervousness (e.g., fidgeting)” Student 1 Yes Only One Rater Student 2 No Student 3 No Student 4 No Student 5 Yes  “Student exhibited signs of nervousness (e.g., fidgeting)” Rater #1 Rater #2 Student 1 Yes Yes Multiple Raters = Student 2 No No Reliability is Testable Student 3 No Yes Student 4 No No Student 5 Yes Yes  “Student’s feet were large” Rater #1 Rater #2 Student 1 Yes Yes Reliable Student 2 No No Measure of Test Anxiety? Student 3 No No Student 4 No No Student 5 Yes Yes  Measurement Operationalizing Your Variable How do you choose your measurement? Reliability Does your measurement consistently measure the same thing? Validity Does your measurement consistently measure the right thing? How do you test the validity of a measure? Face Validity Content Validity Criterion Validity Convergent Validity Predictive Validity Discriminant Validity  Does the measure look like it measures the thing it’s meant to measure? Face Validity  “I am depressed” Depression Items Which is face-valid? “I wake up several hours earlier than I used to and cannot get back to sleep”  Does the measure capture all the important facets of the construct? Content Validity 9 Key Symptoms of Depression Depressed mood Markedly diminished interest or pleasure Weight/appetite problems (increase or decrease) Sleep problems (increase or decrease) Fatigue or loss of energy Feelings of worthlessness and guilt Diminished ability to think; concentrate; make decisions Recurrent thoughts of death  Convergent validity Does it correlate with similar variables? Predictive validity Does it predict expected outcomes? Criterion Validity  Poll Time How could we test the convergent validity of a depression measure? What other measures could we try to correlate it with? Predictive Validity of A Depression Measure Does it predict longer-term outcomes? E.g., physical illness, missed school or work, relationship strain, substance use  Your measure should NOT correlate with theoretically different variables Discriminant Validity  Poll Time How could we test the discriminant validity of a depression measure? What measures should not be associated with depression? Discriminant Validity Also referred to as “divergent validity” If you don’t achieve discriminant validity, your measure is likely too general or broad Grit Measure “You have a certain amount of intelligence, and you can’t really do much to change it” Your talent is something about you that you can’t change very much” “You can learn new things, but you can’t really change your basic intelligence” What If You Put Nonsense? “You have a certain amount of gavagai, and you can’t really do much to change it” “No matter who you are, you can significantly change your gavagai level” “You can learn new things, but you can’t really change your basic gavagai” Maul, 2017 Is Gavagai Valid? Face validity? No Content validity? No Reliability? Yes! Alpha =.91 Convergent validity? Yes! Modest correlations with agreeableness (r =.22, p <.01) and openness (r =.24, p <.01). Measurement Schmeasurement Takeaways Measurement is important Careful validation is very tricky Qualitative validation methods should not be underestimated Next Week: Interpreting Graphs