Understanding Test Validity

The Concept of Validity

• Validity involves a judgment or estimate of how well a test measures what it intends to measure in a specific context.
• It's based on evidence concerning the appropriateness of inferences drawn from test scores, relying on logical results or deduction.
• Validity assesses how useful an instrument is for a specific purpose and population.
• It concerns what an instrument measures, its accuracy, and the meaningfulness of inferences from its results.
• Validity is the degree to which evidence and theory support the interpretation of test scores for proposed uses of tests, following testing standards.
• The validity of test scores is based on accumulated evidence supporting their interpretation and uses
• The validity of inferences (hypotheses) based on test scores can be enhanced or diminished.
• The evidentiary basis for test score interpretations can come from various methods.
• Validation is the process of gathering and evaluating evidence about validity.
• Validation studies compare a measure's accuracy with a gold standard (established) measure.
• Both the test developer and user have roles in validating a test for a specific purpose.
• The test developer is responsible for providing validity evidence in the test manual.
• The test user should conduct their own validation studies with their group of test takers.
• Samuel Messick significantly reshaped the concept of validity, influencing current testing standards.
• Validity integrates evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other assessment modes.
• Local validation studies are needed when the test user alters the test's format, instructions, language, or content.
  • These studies show the correlation between two variables (test scores and performance) across a large group, which requires large sample sizes.
  • They may also provide insights into a particular population of test takers compared to the norming sample in a test manual.

Three Categories of Validity

• Content validity is based on evaluating the subjects, topics, or content covered by the test items.
• Criterion-related validity is obtained by evaluating the relationship of scores on the test to scores on other tests or measures.
• Construct validity is determined through comprehensive analysis by relating test scores to other scores and measures.
  • It also involves understanding how test scores fit within a theoretical framework of the construct being measured.

Trinitarian View

• This approach considers criterion-oriented (predictive), content, and construct validity for assessing test validity.

Validity as an Umbrella Concept

• Construct validity acts as the "umbrella validity," encompassing other forms.
• The three aspects of validity (criterion-related, content, and construct) are examined from a dual perspective to understand a construct.
  • This establishes a basis for comparison between evaluations of measurement validity and evaluations of hypothesis validity.

Approaches to Test Validation

• Includes:
  • Content validation strategies
  • Criterion-related validation strategies
  • Construct validation strategies
• Trinitarian approaches to validity assessment are not mutually exclusive.
• Each of the three conceptions of validity provides evidence that contributes to a judgment about a test's validity.
• All three types of validity evidence contribute to a unified picture of a test's validity.
• Ecological validity refers to a judgment of how well a test measures what it intends to measure at the time and place where the variable is actually emitted.

Face Validity

• Face validity is what a test appears to measure to the person being tested.
• It involves a judgment about the relevance of the test items.
  • A test with high face validity seems valid "on the face of it."
  • A lack of face validity can reduce confidence in the test's perceived effectiveness

Content Validity

• Content validity describes a judgment of how adequately a test samples behavior representative of what the test was designed to sample.
• When a test has content validity, the items represent the entire range of possible items the test should cover and may be drawn from a large pool covering a broad range of topics.
  • Test developers include key components of the construct targeted for measurement and exclude irrelevant content

Educational and Achievement Tests

• Educational achievement tests should have a proportion of material covered by the test that approximates the proportion of material covered in the course.
• A test blueprint is the evaluation's "structure" and includes a plan detailing the types of information to be covered by the items, the number of items for each area, and the organization of the items.

Culture and the Relativity of Content Validity

• A history test considered valid in one classroom at one time and place may not be considered so in another classroom, time, or place.
• Politics can also influence perceptions and judgments about the validity of tests and test items.

Criterion-Related Validity

• Criterion-Related Validity involves a judgment of how well a test score can infer an individual's most probable standing on a measure of interest (the criterion).
• It indicates the effectiveness of an instrument in predicting an individual's performance on a specific criterion.
• A test has criterion-related validity when it effectively predicts indicators of a construct.

Concurrent Validity and Predictive Validity

• The two types of Criterion-Related Validity
  • Concurrent Validity
  • Predictive Validity

What is a Criterion?

• A criterion is a standard on which a judgment or decision may be based.
• For discussion purposes, it is the standard against which a test or test score is evaluated and should be relevant, valid, and uncontaminated.
  • Relevant: it is pertinent or applicable to the matter at hand.
  • Valid: if test X is used to validate test Y, then evidence should exist that test X is valid.
  • Uncontaminated: criterion contamination occurs when a criterion measure is based, at least in part, on predictor measures.
  • Concurrent validity is concerned with the relationship between an instrument's results and another currently obtainable criterion.
  • It refers to the extent to which a measure's results correlate with the results of an established measure of the same or a related construct assessed within a similar time frame.
  • Its statement indicates how well test scores estimate an individual's present standing on a criterion.

Predictive Validity

• Predictive validity examines the relationship between an instrument's results collected now and a criterion collected in the future.
  • It is the degree to which test scores accurately predict scores on a criterion measure; ex. college admissions test scores predicting college GPA.
  • Researchers consider several factors
    • Base rate: the extent to which a particular trait, behavior, characteristic, or attribute exists in the population.
    • Hit rate: the proportion of people a test accurately identifies as possessing or exhibiting a particular trait, behavior, characteristic, or attribute.
    • Miss rate: the proportion of people the test fails to identify as having, or not having, a particular trait, behavior, characteristic, or attribute.
    • False Positive: A test result that incorrectly indicates that a person has a specific disease or condition
    • False Negative: A test result that incorrectly indicates that a person does not have a specific disease or condition.
  • The validity coefficient is a correlation coefficient that measures the relationship between test scores and the criterion measure using Pearson r or Spearman rho.
  • The test developer reports validation data in the test manual.
  • Test users should carefully read the description of the validation study and evaluate the suitability of the test for their specific purposes.
  • There are no set rules for the minimum acceptable size of a validity coefficient/ Cronbach and Gleser (1965) cautioned against such rules.
  • Validity coefficients need to be large enough to enable the test user to make accurate decisions within the unique context of the test's use.
  • Incremental validity is how much an additional predictor explains about a criterion beyond what is already explained by existing predictors.

Construct Validity

• Construct validity involves a judgment about the appropriateness of inferences drawn from test scores regarding individual standings on a construct.
  • A construct is an informed, scientific idea developed or hypothesized to describe or explain behavior.
  • Constructs are unobservable underlying traits that a test developer may use to describe test behavior or criterion performance.
  • Construct validity has been viewed as the unifying concept for all validity evidence.

Integrative Function of Constructs in Test Validation

• To designate the traits, processes, knowledge stores, or characteristics whose presence and extent the test aims to ascertain through specific behavior samples.
• A construct is a hypothetical entity derived from psychological theory, research, or observation of behavior.
• To designate the inferences that may be made based on test scores.
• Construct refers to a specific interpretation of test data or other behavioral data based on a network of pre-established theoretical and empirical relationships between scores and other variables.

Various Techniques of Construct Validation

• Evidence of homogeneity indicates how uniform a test is in measuring a single concept.
  • The Pearson r is used to correlate average subtest scores with the average total test scores
• Evidence of changes with age demonstrates the changes that occur with age.
• Evidence of pretest-posttest changes demonstrates that test scores change following the experiences someone undertakes between a pretest and a posttest.
• Evidence from distinct groups
  • If a test measures a particular construct, test scores should differ between groups presumed to differ with respect to that construct.
• Convergent evidence reveals that test scores tend to correlate highly in the predicted direction with scores on older, validated tests measuring the same or a similar construct.
• Discriminant evidence, a validity coefficient showing a statistically insignificant relationship between test scores and other variables should not theoretically be correlated.
• Factor analysis: a set of mathematical procedures designed to identify factors or variables on which people differ.

TEST BIAS

• This is an error in testing that prevents accurate, impartial measurement.
• Rating error is a judgment resulting from the intentional or unintentional misuse of a rating scale.
• Leniency error (Generosity error) is an error in testing that shows a tendency to be lenient (lazy) in scoring, marking, and/or grading.
• Severity error, shows a tendency to be harsh or too generous in rating.
• Central tendency error, shows the tendency of raters to just select the middle road.
• Halo effect, describes the tendency of some raters to wrongly rate an area.

TEST FAIRNESS

• The extent to which a test is used impartially, justly, and equitably.
• An example is that the norms used for most psychological tests are from the western population and can create a cultural bias for the norms used in the test.
  • A solution to this is Validation studies / Local Validation studies.