week5-1_testing.ppt

Validity I Chapter 8: Content and Construct-oriented Validation Strategies 01/18/24 1 What is Validity? Validity Defined Traditional- Test measures what it is intended to/ claims to/designed to/purpose to measure. Current- ”the degree to which evidence and theory support the interpretations o...

Validity I Chapter 8: Content and Construct-oriented Validation Strategies 01/18/24 1 What is Validity? Validity Defined Traditional- Test measures what it is intended to/ claims to/designed to/purpose to measure. Current- ”the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests” 01/18/24 2 What is Validity? It is not the test itself is validated, but rather the inferences and conclusions that you reach on the basis of test scores. Validity refers to the “interpretation of test scores.” So, the goal is to understand the meaning and the implications of test scores. For example, “Is the interpretation of performance on the WISC-IV (Wechsler Intelligence Scale for Children) as reflecting intelligence valid?” 01/18/24 3 What is Validity? E.g. ALES If an undergraduate student receives a higher scores on ALES, a graduate admissions committee might use that score to predict or infer that s/he will do well in graduate school. If that student do well in grad school and if students with lower scores do less well, they drew valid conclusions from scores on this test. 01/18/24 4 2 Major Types of Validity Validity of Measurement: Does the test measure what it is supposed to measure and how well? Content validity Construct validity Validity of Decisions: How accurate are the decisions based on the test? Predictive validity Concurent validity 01/18/24 5 2 Major Types of Validity A company use an inventory called Leadership Skills Profile to select managers. 1st: does this inventory tell you anything about a person’s leadership skills? 2nd: do people who got higher scores on this test become a good manager? 01/18/24 6 Content Validity Extent to which the items on a test are representative of the construct the test measures. Is established by examining the test itself. If your test is supposed to be on chapters 1-5, this is representative of content validity. 01/18/24 7 Content Validity I want to measure X, let’s say mathematical ability e.g. Questions about sports A content domain has boundaries. 01/18/24 8 Content Validity 01/18/24 9 Content Validity No exact statistical method to assess content validity The process consists of 3 steps: Describe the content domain. Determine the domain areas that are measured by each test item. Compare the structure of the test with the structure of the domain. Tests with high content validity should cover all parts of the domain and most important aspects should be covered with longest number of items. 01/18/24 10 Content Validity 01/18/24 11 Content Validity 01/18/24 12 Construct Validity How well the test measures the theoretical construct or trait. Defined as the extent to which the test measures a theoretical construct. Deals with the assumed relationships between and among hypothetical constructs. We test construct validity by looking at the patterns of relationships of measures of constructs (i.e., are the tests scores correlated the way we think they would be?) 01/18/24 13 What is a Construct? Construct is abstract, hypothetical trait that summarizes some regularity in nature. related to Concrete-behaviors (activities that are observable and measurable) In psychology, a sample of behaviors. leadership ability, overcontrolled hostility, depression, and intelligence, beauty, love, selfesteem, honesty, etc.  Think of gravity, we can not see gravity, but we can see an apple fall from a tree. 01/18/24 14 Construct Explication Translating the abstract construct into concrete, behavioral terms. Identify the behaviors that relate to the construct. Identify other constructs that may be related to the construct being explained. Identify behaviors related to similar constructs and determine whether these behaviors are related to the original construct. 01/18/24 15 Construct Explication depression Crying eating disorder obsessive Self-esteem Successful 01/18/24 anxiety sleep disturbance being honesty worthy Provides a definition of what we mean by depressio 16 Construct Explication 01/18/24 17 Construct Explication 01/18/24 18 Construct Explication 01/18/24 19 Gathering Psychometric Evidence for Construct Validity Reliability Correlations to other relevant tests. Multitrait-multimethod matrix (MTMM) Convergent Validity Discriminant Validity Factor Analysis Items correlations to the factor (construct) 01/18/24 20 Factor Analysis Describe variability among observed, correlated variables in terms of a potentially lower number of unobserved variables called factors.  For example, it is possible that variations in six observed variables mainly reflect the variations in two unobserved (underlying) variables. 01/18/24 21 Factor Analysis Factor analysis is a specialized statistical technique that is particularly useful for investigating construct validity. The goal is to find a smaller set of factors that can account for the observed array of intercorrelations among individual tests. 01/18/24 22 Factor Analysis Spatial ability and verbal ability. These 2 abilities are independent. RC and VOCAB are highly correlated because they measure the same factor. 01/18/24 23 Factor Analysis So, there are obviously 2 distinct factors, factor loadings. 01/18/24 24 Factor Analysis We are creating new factors (components) 01/18/24 based on original variables 25 Factor Analysis 01/18/24 26 Factor Analysis Types Explatory factor analysis: to explore the possible underlying factor structure of a set of observed variables Confirmatory factor analysis: to test how well the measured variables represent the number of constructs researchers specify the number of factors researcher uses knowledge of the theory, empirical research for the number of factors 01/18/24 27 MTMM (Multitrait-multimethod matrix) Combines evidence of reliability, convergent validity and discriminant validity into one study. Investigators choose 2 or more constructs that are unrelated in theory and 2 types of different methods (self-report, observation, peer rating, etc)  An example could be the 5 different personality traits assessed by self, peer, and observer.  The honesty, the aggressiveness and intelligence of a group of schoolchildren can be measured using 3 methods: teacher ratings, paper-pencil tests, ratings from outside observers. 01/18/24 28 Method biasCorrelations might be high in a particular way of measuring. 01/18/24 29 Convergent validity Establishes construct by comparing it to same construct collected by another method. Convergent validity is demonstrated when a test correlates highly with other variables or tests with which it shares an overlap of constructs. Example: compare a new measure of anxiety with an old, established measure of anxiety.  Correlate new and old measure  If correlation is strong, positive = good convergent validity 01/18/24 30 Discriminant Validity Establishes construct by differentiating it from separate constructs. Discriminant validity is demonstrated when a test does not correlate with variables or tests from which it should differ. Compare construct to unrelated constructs Example: compare extraversion to neuroticism To analyze: correlate construct with other unrelated constructs. Expect negative correlations. Pearson’s r should be low or 0. 01/18/24 31 Convergent and Discriminant validity ______: Different measures of the same construct should correlate highly. ______:theoretically independent constructs should not be correlated. 01/18/24 32 Convergent and Discriminant validity 01/18/24 33 Characteristics of a good test Consistent scores of the same construct from different tests. Test scores do not correlate with unrelated constructs (when the same method is used) Method of measurement used by the test show little evidence of bias. 01/18/24 34 Face Validity Not technically a form of validity. A test has face validity if it looks valid to test users, examiners, and especially the examinees. How a test taker perceives that appropriateness of a test. Important because it can influence how test takers approach the test. 01/18/24 35 Differences between content and construct validity 1. Content validity is assessed by checking if the test provides representative sample of the content domain. Construct validity is showed when the pattern of correlations is the same as hypothesized. 01/18/24 36 Differences between content and construct validity 2. Content validity is established if a test looks like a valid measure. Construct validity is established if a test acts like a valid measure. 01/18/24 37