Identifying Good Measurement

• Identifying Good Measurement

Abstract vs. Concrete Constructs

• An abstract construct refers to a mental or theoretical concept (e.g., love, hunger, intelligence).
• A concrete construct can be directly observed or measured (e.g., number of items recalled, reaction time, height).
• "Love" is an example of an abstract concept.
• "Blood pressure" can be a concrete physiological index of stress.

Conceptual Definitions

• The conceptual definition of a variable, also known as the construct definition, clarifies the researcher's meaning when referring to that variable.
• A conceptual definition of "hunger" could be "the subjective feeling of needing food."

Operationalization

• Operationalization, also called an operational definition, is how a conceptual variable gets measured or manipulated in a study.
• "Hunger" might be operationalized as "hours since last meal" or "total calories consumed in a day.”

Self-Report Measures

• These measures gather data through participants' verbal or written responses, like surveys, questionnaires, or interviews.
• Parent or teacher reports can also be included when studying children.
• A questionnaire asking, "On a scale of 1–5, how hungry are you right now?" is an example.

Observational Measures (Behavioral)

• These measures are based on direct observation of behavior or physical traces.
• Physical traces, like counting wrappers in a trash can, can indicate snacking behavior.
• Counting the number of times someone opens a refrigerator gives an indication of hunger through observable behavior.

Physiological Measures

• These measures record biological data, such as fMRI, EEG, hormone levels, and heart rate.
• Measuring salivary cortisol indicates stress levels, while saliva production can operationalize hunger.

Reliability

• Reliability refers to the consistency or repeatability of a measure.
• It encompasses test-retest, interrater, and internal reliability.

Test-Retest Reliability

• This is the consistency of a measure across two or more testing occasions for constructs expected to remain stable (e.g., intelligence).
• Measuring someone's intelligence in January and June should yield strongly correlated scores if the measure is reliable for "IQ".

Interrater Reliability

• Interrater reliability indicates the degree to which two or more observers or coders agree on their observations of the same behavior.
• If two observers count instances of aggression on a playground, their tallies should be very similar if interrater reliability is high.

Internal Reliability

• Internal reliability is the extent to which multiple items on the same measure or scale consistently measure the same construct.
• Cronbach's alpha is often used to assess it.
• Scores should be correlated if a depression questionnaire with 10 questions targeting depression symptoms is internally reliable.

Scales of Measurement (Levels of Measurement)

• Nominal scales are categorical with no numerical meaning (e.g., types of soda like Coke, Sprite, Pepsi).

• Assigning "1" to Coke, "2" to Sprite, and so on, is for labeling only.

• Ordinal scales indicate ranked order, though intervals are not necessarily equal (e.g., finishing places in a race: 1st, 2nd, 3rd).

• The time gap between 1st and 2nd place may differ from the gap between 2nd and 3rd.

• Interval scales include numeric scales that have equal intervals between values, but no true zero (zero doesn't mean "none" of the construct).

• For example, 0°C doesn't mean “no temperature.”

• Ratio scales feature numeric scales with equal intervals and a true zero (i.e., zero means "nothing" of that variable).

• Examples include height, weight, and reaction time (0 ms signifies no reaction time).

Scatterplots

• Scatterplots are graphs that display the relationship between two variables, often to visualize correlation.

Correlation Coefficients

• A statistical measure, like Pearson's r, that indicates the direction and strength of a relationship between two variables, ranging from -1.0 to +1.0.
• An r of +0.80 indicates a strong positive correlation, while r = -0.05 indicates nearly no relationship.

Validity (of a Measure)

• This refers to the extent to which a measure assesses what it is intended to measure.

Face Validity

• Face validity assesses whether a measure appears, on the surface, to measure what it claims to measure.
• A survey titled "Depression Questionnaire" which asks about mood and energy has face validity since it looks like it's measuring depression.

Content Validity

• This refers to the extent to which a test or measure includes all parts of the construct it aims to assess.
• A math test covering algebra, geometry, and calculus (not just algebra) captures the entire domain.

Criterion Validity

• Criterion validity is the extent to which a measure predicts or relates to an outcome it should theoretically predict.
• SAT scores serving as a measure to predict college performance is an example.

Convergent Validity

• Convergent validity refers to the degree to which a measure correlates strongly with other measures of the same construct.
• A new anxiety scale correlating highly with an established anxiety questionnaire demonstrates convergent validity.

Discriminant (Divergent) Validity

• This refers to the extent to which a measure does not correlate strongly with measures of different constructs.
• A depression scale shouldn't correlate strongly with a measure of physical fitness.

Internal Validity (as a measurement term)

• In measurement contexts, "internal validity" can mean the measure is free from confounding factors.
• It assures that the measure alone captures the construct.

Between-Groups Designs (Independent-Groups or Between-Subjects Designs)

• Different groups of participants are placed into different levels of the independent variable.
• Group 1 studies with classical music, and Group 2 studies with no music to compare test performance.

Equivalent Groups, Posttest-Only Design

• Participants are randomly assigned to groups to ensure equivalence, then tested once (posttest) on the dependent variable.
• Randomly assigning people to watch either a funny or a serious video, then measuring their mood afterward, exemplifies a posttest-only design.

Equivalent Groups, Pretest/Posttest Designs

• Participants are randomly assigned to at least two groups, measured on the DV before exposure to the IV (pretest) and then again after exposure (posttest).
• When two groups both take a mood pretest, later Group 1 gets the "funny" video and Group 2 gets the "serious" video, followed by a mood posttest.

Control Variable (and Control Variables)

• Any variable the experimenter holds constant across conditions to isolate the independent variable's effect on the dependent variable.
• Maintaining consistent room temperature, time of day, and lighting for all participants.

Dependent Variable

• This is the variable measured to determine whether it is affected by the independent variable.
• Test scores, reaction time, and mood ratings are examples of the dependent variable.

Independent Variable

• The variable that the experimenter manipulates; it has distinct levels/conditions.
• Classical music versus no music in a memory study is an example.

Design Confounds

• A second variable systematically varies with the IV, providing an alternative explanation for the results.

Effect Size

• Effect size is a quantitative measure of the strength or magnitude of the relationship (e.g., Cohen's d, Pearson's r) between variables.
• A large Cohen's d (e.g., 0.80) indicates a big difference between groups.

Internal Validity (in Experiments)

• This is the degree to which a study rules out alternative explanations for a causal relationship between the IV and DV.

• Also it establishes that the IV alone caused changes in the DV.

• A well-controlled study with no confounds and random assignment ensures high internal validity.

Matched-Groups Design

• Participants are matched on a particular characteristic (e.g., IQ, age) and then randomly assigned to different conditions.
• This design is often used to reduce individual difference confounds in small samples.
• Matching on reading level and then assigning matched pairs to different teaching-method conditions.

Order Effects (Carryover & Practice Effects)

• A within-subjects phenomenon: being exposed to one condition changes how participants respond to subsequent conditions.

Types of Order Effects:

• Carryover Effects: Residual influence from a previous condition (e.g., drug still in the system).
• Practice (or Fatigue) Effects: Participants get better (practice) or worse (fatigue) over repeated tasks.

Pretest/Posttest Design

• This is a type of between-groups design where participants are measured on the DV both before and after exposure to the IV.

Selection Effects

• Selection effects occur when participants in one IV level systematically differ from those in another IV level.
• This often arises when participants self-select or when assignment isn't properly random.
• Volunteers for the "stressful condition" may already be more thrill-seeking compared to those placed in a neutral condition.

Systematic Variability

• This is when, in experiments, a variable's levels coincide in a predictable (non-random) manner with the IV, potentially creating confounds if uncontrolled.
• When all enthusiastic research assistants run the "treatment" group while all bored assistants run the "control" group.

Unsystematic Variability

• Random or haphazard fluctuations across conditions contribute noise to data but don't provide a systematic alternative explanation.
• Individual differences in mood fluctuate daily, adding random variation unrelated to the IV group.

Within-Groups Designs (Within-Subjects Designs)

• The same group of participants experiences all levels/conditions of the independent variable.

Concurrent Measures

• Participants are exposed to all IV levels (or multiple stimuli) simultaneously, then a single attitudinal or behavioral preference is the DV.
• Infants view two faces simultaneously (male vs. female), and the time spent looking at each face is measured.

Repeated Measures

• Participants are measured on the DV multiple times, after each distinct level of the IV.
• The same group rates the taste of Cookie A, then Cookie B, then Cookie C.

Attrition Threats

• Attrition threatens occur when certain participants (often extreme scorers) drop out of a study systematically.
• This threatens internal validity if one condition loses more participants than another.
• Participants with severe anxiety dropping out of the "candy therapy" condition while calmer individuals stay.

Blind Studies (and Double-Blind Studies)

• In a single-blind (masked) study: Participants or researchers are unaware of which condition participants are in.
• In a double-blind study: Neither the participants nor the researchers evaluating them know who's in which condition.
• In a drug trial: neither participants nor experimenters know who receives the real drug versus a placebo.

Ceiling and Floor Effects

• Ceiling effect: occurs when all scores cluster at the top of the scale because a task is too easy or the measurement is capped.
• Floor effect: occurs when all scores cluster at the bottom because a task is too hard or the measurement starts above zero.
• A math test that is far too easy could produce near-100% scores for everyone (ceiling effect).

Combined Threats

• Combined threats occur when two or more internal validity threats overlap.
• For example, selection-history, where a historical event only affects participants in one condition.
• When only the experimental group experiences a campus wellness event, leading them to change their behavior differently from the control group.

Demand Characteristics

• These are cues in the research setting that allow participants to guess the hypothesis or expectations, potentially altering their behavior to "help" the study.
• A participant notices the researcher's excitement about one condition and starts trying to please them in that condition.

History Threats

• External events happening during the study affect all (or most) participants in the treatment group.
• A campus-wide stress-reduction campaign might reduce everyone's stress, not just the group receiving "therapy.”

Instrumentation Threats

• These occur if the measurement instrument (e.g., coding guidelines, calibration) changes over time.
• The changes make pretest/posttest scores not directly comparable.
• A researcher becoming more lenient in scoring anxiety over the course of the study.

Maturation Threats

• A natural change in participants (like spontaneous improvement) occurs over time, not due to the IV.
• Students becoming less nervous by the end of the semester simply because they've adjusted to school demands.

Measurement Error

• Measurement error consists of factors that inflate or deflate a person's true score on the DV.
• Using a poorly calibrated scale to weigh participants introduces random error in weight measurements.

Observer Bias

• Observer bias occurs when researcher expectations influence how they interpret outcomes or record behaviors.
• A researcher unconsciously rates participants they "expect" to improve as more improved.

Placebo Effects

• Improvement or change occurs simply because participants believe they are receiving a valid treatment, not from the treatment's "active" ingredients.
• Taking a sugar pill for headache relief and feeling better purely due to the belief that the pill is real medicine.

Precision & Power

• Power is the likelihood of finding a statistically significant effect when one truly exists.
• Precision refers to how narrow the estimate (e.g., a confidence interval) is around an effect.
• Larger sample sizes and well-controlled methods increase power by reducing random noise.

Regression Threats (Regression to the Mean)

• This is a tendency for extreme scores at one measurement to be less extreme (closer to average) at the next measurement.
• Students with unusually high nervousness scores on the first day might naturally score closer to average by the second day, regardless of any treatment.

Reverse Confounds

• When an unaccounted-for variable pushes the effect of the IV in the opposite direction, potentially masking a real difference.
• If the "no coffee" group accidentally all had intense sugar rushes, it might cancel out the caffeine advantage in the "coffee" group.

Sample Sizes

• The number of participants in each condition.
• Larger sample sizes reduce random error and increase the precision of estimates, thereby boosting power.
• A study with 200 participants per group typically has higher power than one with only 20 participants per group.

Situation Noise

• Any external distractions or variability in the testing environment that increase unsystematic variability within each group.
• Running one group of participants in a noisy hallway and the other group in a quiet lab could add extra "noise" to the data.