Psychological Assessment: Scorer Differences

Psychological Assessment

Error: Scorer Differences

• Evaluates the degree of agreement between two or more scorers on a particular measure
• Calculated by determining the percentage of times two individuals assign the same scores to the performance of examinees
• Variations: having two examiners test the same client using the same test and determining the closeness of their scores or ratings
• Measures of scorer differences:
  • Fleiss Kappa: determines the level of agreement between two or more raters on a categorical scale
  • Cohen's Kappa: used for two raters only
  • Krippendorff's Alpha: used for two or more raters, correcting for chance agreement

Tests Designed

• Homogenous tests: designed to measure one factor, expected to have a high degree of internal consistency
• Dynamic tests: measure traits, states, or abilities that are fast-changing as a function of situational and cognitive experience
• Static tests: measure traits, states, or abilities that are relatively unchanging
• Restriction of range or variance: when the variance of either variable in a correlational analysis is restricted, resulting in a lower correlation coefficient

Power Tests

• Designed to allow test-takers to attempt all items within a time limit
• Measures a test-taker's ability to complete a task accurately and efficiently

Speed Tests

• Contain items of uniform difficulty with a time limit
• Reliability should be based on performance from two independent testing periods using test-retest, alternate-forms, or split-half reliability

Criterion-Referenced Tests

• Designed to provide an indication of where a test-taker stands with respect to a criterion
• As individual differences decrease, traditional measures of reliability also decrease, regardless of individual performance stability

Classical Test Theory

• Assumes that everyone has a "true score" on a test
• True score reflects an individual's ability level as measured by a particular test
• Random error affects the observed score

Domain Sampling Theory

• Estimates the extent to which specific sources of variation contribute to test scores
• Considers problems created by using a limited number of items to represent a large construct

Test Reliability

• Conceived as an objective measure of how precisely a test score assesses a domain
• Reliability is a function of the proportion of total variance attributed to true variance

Generalizability Theory

• Based on the idea that test scores vary due to variables in the testing situation
• Universe: the test situation
• Facets: number of items, amount of review, and purpose of test administration
• Given the same conditions, the same test score should be obtained (universe score)

Decision Study

• Examines the usefulness of test scores in helping test users make decisions

Systematic Error

• Factors inherent in a test that prevent accurate, impartial measurement

Item Response Theory

• The probability of a person with a certain ability level performing at a certain level on a test
• Focuses on item difficulty

Latent-Trait Theory

• A system of assumptions about measurement and the extent to which items measure a trait
• Computers are used to focus on the range of item difficulty that helps assess an individual's ability level
• If a person answers easy items correctly, the computer will move to more difficult items
• Item attributes: difficulty, discrimination, and dichotomousness

Construct Validity (Umbrella Validity)

• Covers all types of validity
• Logical and statistical
• Judgment about the appropriateness of inferences drawn from test scores regarding individual standing on a variable called a construct

Criterion Validity

• More statistical than logical
• Judgment about the adequacy of test scores in inferring an individual's standing on a criterion measure
• Criterion: a standard on which a judgment or decision may be made
• Characteristics: relevant, valid, uncontaminated
• Types of criterion validity: concurrent, predictive, and incremental validity

Factor Analysis

• Designed to identify factors or variables that are typically attributes, characteristics, or dimensions on which people may differ
• Developed by Charles Spearman
• Employed as a data reduction method
• Used to study the interrelationships among a set of variables
• Types of factor analysis: explanatory, confirmatory, and factor loading

Cross-Validation

• Validation of a test to a criterion based on a different group from the original group
• Validity shrinkage: a decrease in validity after cross-validation
• Co-validation: validation of more than one test from the same group
• Co-norming: norming more than one test from the same group

Bias

• Factors inherent in a test that systematically prevent accurate, impartial measurement
• Prevention: during test development through procedures such as estimated true score transformation

Rating

• Numerical or verbal judgment that places a person or attribute along a continuum identified by a scale
• Rating error: intentional or unintentional misuse of the scale
• Types of rating error: leniency, severity, central tendency, and halo effect
• One way to overcome rating errors is to use rankings### Discriminant Evidence
• Definition: A validity coefficient showing little relationship between test scores and/or other variables
• Importance: Used in Psychological Assessment

Measures of Central Tendency

Mode

• Definition: The most frequently occurring score in a distribution
• Use: For ordinal data, and for nominal scales and discrete variables
• Characteristics: Can be used in analyses of qualitative or verbal nature, and gives an indication of the shape of the distribution

Measures of Spread or Variability

• Definition: Statistics that describe the amount of variation in a distribution
• Use: Gives an idea of how well the measure of central tendency represents the data
• Characteristics: Large spread of values means large differences between individual scores

Measures of Spread or Variability

Range

• Definition: The difference between the highest and lowest score
• Use: Provides a quick but gross description of the spread of scores
• Characteristics: Can be affected by extreme scores in the distribution

Variance

• Definition: The square root of the average squared deviations about the mean
• Use: Equal to the square root of the variance, and measures the distance from the mean
• Characteristics: Equal to the arithmetic mean of the squares of the differences between the scores in a distribution and their mean

Measures of Location

Percentile or Percentile Rank

• Definition: Expressed in terms of the percentage of persons in the standardization sample who fall below a given score
• Use: Indicates the individual's relative position in the standardization sample
• Characteristics: Not linearly transformable, converged at the middle, and the outer ends show large intervals

Quartile

• Definition: Dividing points between the four quarters in the distribution
• Use: Specific point in the distribution

Correlation

Spearman Rho

• Definition: Used for ordinal + ordinal data
• Use: Measures the correlation between two variables

Biserial

• Definition: Used for true dichotomous + interval/ratio data
• Use: Measures the correlation between two variables

Point Biserial

• Definition: Used for nominal (true dichotomous) + nominal (true/artificial dichotomous) data
• Use: Measures the correlation between two variables

Phi Coefficient

• Definition: Used for artificial dichotomous + artificial dichotomous data
• Use: Measures the correlation between two variables

Tetrachoric

• Definition: Used for 3 or more ordinal/rank data
• Use: Measures the correlation between two variables

Kendall's Rank Biserial Differences

• Definition: Used for two separate groups, random assignment
• Use: Measures the correlation between two variables

T-Test

• Definition: Used for comparing means between two groups
• Use: One group, two scores (e.g., blood pressure before and after the lecture)

T-Test Dependent (Paired T-test)

• Definition: Used for comparing means between two groups
• Use: One group, measured at least three times

One-Way ANOVA

• Definition: Used for comparing means between three or more groups
• Use: One group, measured at least three times

One-Way Repeated Measures

• Definition: Used for comparing means between three or more groups
• Use: Three or more groups, tested for two variables

Two-Way ANOVA

• Definition: Used for comparing means between two or more groups, controlling for an additional variable
• Use: Used when you need to control for an additional variable that may be influencing the relationship between your independent and dependent variable

Utility Gain

• Definition: Estimate of the benefit of using a particular test
• Use: In Psychological Assessment

Productivity Gains

• Definition: An estimated increase in work output
• Use: In Psychological Assessment

Cut Score

• Definition: Reference point derived as a result of a judgment and used to divide a set of data into two or more classifications
• Use: In Psychological Assessment

Relative Cut Score

• Definition: Reference point based on norm-referenced considerations, not fixed per se
• Use: In Psychological Assessment

Fixed Cut Scores

• Definition: Set with reference to a judgment concerning minimum level of proficiency required
• Use: In Psychological Assessment

Multiple Cut Scores

• Definition: Refers to the use of two or more cut scores with reference to one predictor for the purpose of categorization
• Use: In Psychological Assessment

Multiple Hurdle

• Definition: Multi-stage selection process, a cut score is in place for each predictor
• Use: In Psychological Assessment

Compensatory Model of Selection

• Definition: Assumption that high scores on one attribute can compensate for lower scores
• Use: In Psychological Assessment

Angoff Method

• Definition: Setting fixed cut scores
• Use: In Psychological Assessment

Known Groups Method

• Definition: Collection of data on the predictor of interest from groups known to possess and not possess a trait of interest
• Use: In Psychological Assessment

IRT-Based Methods

• Definition: Cut scores are typically set based on test-taker's performance across all the items on the test
• Use: In Psychological Assessment

Item-Mapping Method

• Definition: Arrangement of items in a histogram, with each column containing items deemed to be equivalent in value
• Use: In Psychological Assessment

Bookmark Method

• Definition: Expert places a "bookmark" between the two pages that are deemed to separate test-takers who have acquired the minimal knowledge, skills, and/or abilities from those who are not
• Use: In Psychological Assessment

Method of Predictive Yield

• Definition: Took into account the number of positions to be filled, projections regarding the likelihood of offer acceptance, and the distribution of applicant scores
• Use: In Psychological Assessment

Discriminant Analysis

• Definition: Used to shed light on the relationship between identified variables and two naturally occurring groups
• Use: In Psychological Assessment, used to analyze data when the criterion or dependent variable is categorical and the predictor or independent variable is interval in nature

Reliability and Validity

Reliability

• Definition: The consistency of test scores
• Interpretation:
  • Excellent: 0.90 and up
  • Good: 0.80-0.89
  • Adequate: 0.70-0.79
  • Limited applicability: below 0.70

Validity

• Definition: The degree to which a test measures what it claims to measure
• Interpretation:
  • Very beneficial: above 0.35
  • Likely to be useful: 0.21-0.35
  • Depends on circumstances: 0.11-0.20
  • Unlikely to be useful: below 0.11

Item Analysis

Item-Validity Index

• Definition: Designed to provide an indication of the degree to which a test is measuring what it purports to measure
• Use: In Psychological Assessment

Item-Discrimination Index

• Definition: Measures the difference between the proportion of high scorers answering a question correctly and the proportion of low scorers answering it correctly
• Use: In Psychological Assessment

Extreme Group Method

• Definition: Compares people who have done well with those who have done poorly
• Use: In Psychological Assessment

Discrimination Index

• Definition: The difference between those proportions
• Use: In Psychological Assessment

Point-Biserial Method

• Definition: Correlation between a dichotomous variable and continuous variable
• Use: In Psychological Assessment

This quiz assesses your knowledge of scorer differences in psychological assessment, including measures of agreement and variability. Topics include evaluating scorer differences, types of measures, and the Fleiss Kappa coefficient.