Test Validity and Reliability Overview

Establishing Test Validity and Reliability

• Desired Learning Outcomes: Explain procedures and statistical analysis for establishing test validity and reliability; and decide if a test is valid or reliable.

Test Reliability

• Definition: Reliability is the consistency of responses under three conditions: retesting the same person; retesting on the same measure; and similarity of responses across items measuring the same characteristic.

• Conditions:

  • Consistent response when the test is given to the same participants.
  • Reliability is achieved if responses to the same or equivalent test (or another test measuring the same characteristic) are consistent when administered at different times
  • Reliability exists when the person responds consistently across items measuring the same characteristic.

• Factors Affecting Reliability:

  • Number of items: More items generally lead to higher reliability due to a larger item pool.
  • Individual differences: Characteristics like fatigue, concentration, innate ability, perseverance and motivation, which can change over time, impact consistency.
  • External environment: Factors like room temperature, noise level, instruction quality, and exposure to materials can affect examinee responses.

Methods of Establishing Test Reliability

• 1. Test-Retest:

  • Administer a test to a group once, then again to the same group later (with a 6-month maximum interval). Responses should be similar. Suitable for aptitude tests.
    • Time interval is 30 mins minimum.
  • Suitable for stable variables such as aptitude and psychomotor measures (typing, physical education tasks).
  • Use Pearson Product Moment Correlation (or Pearson r) for analysis; significant positive correlation indicates temporal stability.

• 2. Parallel Forms:

  • Create two versions of the same test (called "forms") measuring the same skill.
  • Administer one form to a group, then the other form to the same group at a later time. Responses on both forms should be similar.
  • Applicable for repeatedly used tests (like entrance exams).
  • Use Pearson r for analysis; significant positive correlation indicates consistency in the different forms.

• 3. Split-Half:

  • Administer a test to a group. Divide the test into two halves (usually odd-numbered and even-numbered questions).
  • Correlate the scores of the two halves.
  • Applicable for tests with many items.
  • Use Pearson r to correlate the halves; then Spearman-Brown Coefficient to determine internal consistency reliability. Scores from each set should be consistent.

• 4. Test of Internal Consistency:

  • Determine if responses to each item are consistent.
  • Administer the test, record scores for each item.
  • Responses should be consistent amongst items; useful for assessments with many items and scales (e.g., Likert scales).
  • Use Cronbach's alpha or Kuder-Richardson for analysis; value of 0.60 or above indicates internal consistency.

• 5. Inter-rater Reliability:

  • Determine the consistency of multiple raters using rating scales or rubrics to judge performance.
  • Multiple raters should produce similar/consistent ratings.
  • Useful when assessments involve multiple raters.
  • Use Kendall's tau coefficient for analysis; a significant value indicates agreement among raters.

Test Validity

• Definition: A measure is valid if it measures what it's supposed to.
  • A valid quarterly exam directly measures the curriculum's objectives.
  • A personality scale with 5 factors should have highly related items.
  • A valid entrance exam predicts first-semester grades.

Types of Validity

• 1. Content Validity:

  • Items represent the entire domain being measured.
  • Items are compared to objectives of the program; a reviewer checks alignment.

• 2. Face Validity:

  • Test appears to measure what it's intended to.
  • Items, instructions, grammar, and vocabulary must be understandable to the test takers.
  • Small group of respondents are checked.

• 3. Predictive Validity:

  • Measures ability to predict future performance.
  • Example: Entrance exam predicting first-semester grades; correlation between test scores and later grades.

• 4. Construct Validity:

  • Measures the underlying theoretical constructs it's designed to measure.
  • Items are correlated with each factor (correlation is done for the factors of the test). - Example: factor analysis to determine how items load (belong) into specific constructs or domains or areas.

• 5. Concurrent Validity:

  • Examines whether a new measure correlates with established measures of the same characteristic.
  • Scores should be correlated to other measures of the same characteristic.

• 6. Convergent Validity:

  • Measures whether multiple tests designed to measure similar constructs correlate highly.
  • Correlation is conducted for the factors of the test

• 7. Divergent Validity:

  • Establishes that a construct doesn't correlate with measures of unrelated constructs.
  • Correlation is conducted for factors the test

This quiz focuses on the principles of test validity and reliability. You will explore statistical analysis methods and procedures to determine if a test meets the desired criteria for consistency and accuracy. By the end, you'll be equipped to evaluate the reliability of various assessments.