Psychology Lecture 2: Reliability and Validity

Questions and Answers

What is the difference between reliability and validity?

Reliability: The test measures one and only one thing (precisely). Validity: The test measures what it is supposed to measure.

What are three things that need consistency for reliability?

Consistency across items (internal consistency, alternate forms, split-half reliability), 2) Consistency across time (test-retest), 3) Consistency across other sources (inter-rater reliability).

What is the Generalizability Theory?

Generalizability Theory (G-Theory) looks at all the different sources as part of the same analysis - it examines the amount of inconsistency due to each source of error.

What are the five sources of evidence for validity?

<ol> <li>Evidence from item content, 2) Evidence from process/manipulations, 3) Evidence from internal structure, 4) Evidence from relationship to other variables (criterion and construct), 5) Evidence from consequences of test use.</li> </ol> Signup and view all the answers

What are the Test Standards?

Recommendations for using and interpreting test scores, developed by the American Psychological Association (APA), American Educational Research Association (AERA), and National Council on Measurement in Education (NCME). Signup and view all the answers

What are the three parts of the test standards?

Part I: Foundations, Part II: Operations, Part III: Testing Applications. Signup and view all the answers

What are the 3 tenets of professional practice for using and interpreting test scores?

<ol> <li>Validity, 2) Reliability/precision and errors of measurement, 3) Fairness in testing.</li> </ol> Signup and view all the answers

Why are the test standards important?

Test standards provide a framework that represents the current consensus and operational guidelines for testing. Signup and view all the answers

What is a major problem with the 'new' standards?

They are currently very difficult to get. Signup and view all the answers

What is validity?

Validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests. Signup and view all the answers

What is the over-arching standard of validity?

A rationale should be presented for each recommended interpretation and use of test scores, along with evidence and theory supporting the intended use. Signup and view all the answers

What was the 1954 Standards definition of validity?

Criterion-based view - 'A test is valid for anything with which it correlates.' Signup and view all the answers

What was the 1966 Standards definition of validity?

Tripartate view including content validity, criterion validity (concurrent & predictive), and construct validity (discriminant & convergent). Signup and view all the answers

What was the 1985 Standards definition of validity?

Tripartate + outcomes (intended or unintended consequences of the test). Signup and view all the answers

What was the 1999 Standards definition of validity?

A unitary form of validity based on evidence from multiple sources to support an argument for what the test scores actually mean. Signup and view all the answers

What is the 2014 Standards definition of validity?

A unitary form of validity, based on evidence from multiple sources to support an argument for what the test scores actually mean. Signup and view all the answers

Study Notes

Reliability vs. Validity

Reliability: Measures consistency in what is tested; focuses on precision.
Validity: Assesses whether a test measures what it is intended to measure.

Consistency in Reliability

Across Items: All test items must measure the same construct (assesses internal consistency).
Across Time: Consistent test performance over different administrations (test-retest reliability).
Across Raters: Consistency across different evaluators (inter-rater reliability).

Generalizability Theory (G-Theory)

Analyzes various sources of error within the same framework.
Provides a statistical method for evaluating the reliability of measurements under specific conditions.

Sources of Validity Evidence

Evidence from item content.
Evidence from process/manipulations.
Evidence from internal structure.
Evidence from relationships to other variables (includes criterion and construct validity).
Evidence from consequences of test use.

Test Standards

Established by the American Psychological Association (APA), the American Educational Research Association (AERA), and the National Council on Measurement in Education (NCME).
Updated versions published in 1954, 1966, 1974, 1985, 1999, with the latest in 2014.

Parts of Test Standards

Part I: Foundations.
Part II: Operations.
Part III: Testing Applications.

Professional Practice Tenets

Key principles for test score interpretation include validity, reliability/precision, and fairness in testing.

Importance of Test Standards

Provide a framework representing current consensus and operational guidelines.
Include multiple viewpoints and psychometric models for evaluating validity, reliability, and bias.

Challenges with New Standards

Access to updated test standards is currently problematic.

Definition of Validity (2014 Standards)

Validity is defined as the extent to which evidence and theory support the intended uses of test scores, emphasizing empirical observations and their interpretations.

Overarching Validity Standard

Requires rationale for each recommended interpretation of test scores, including evidence and theory summary.

Historical Definitions of Validity

1954 Standards: Criterion-based view correlating tests with outcomes.
1966 Standards: Tripartite view encompassing content, criterion, and construct validity.
1985 Standards: Extended to include the consequences of testing.
1999 Standards: Unitary validity based on multiple evidence sources supporting score interpretations.
2014 Standards: Maintained the unitary form of validity from 1999.

Criterion View of Validity

Originated in the early 1950s by theorists like Guilford and Cureton, focusing on correlating test results with relevant criteria.

Studying That Suits You

Use AI to generate personalized quizzes and flashcards to suit your learning preferences.

Description

This quiz focuses on the key concepts of reliability and validity in psychological testing. It covers essential definitions and the importance of consistency in test standards. Explore how these concepts impact the effectiveness of assessments.