Psychological Assessment: Scorer Differences

Psychological Assessment

Error: Scorer Differences

• Evaluates the degree of agreement between two or more scorers on a particular measure
• Calculated by determining the percentage of times two individuals assign the same scores to the performance of examinees
• Variations: having two examiners test the same client using the same test and determining the closeness of their scores or ratings
• Measures of scorer differences:
  • Fleiss Kappa: determines the level of agreement between two or more raters on a categorical scale
  • Cohen's Kappa: used for two raters only
  • Krippendorff's Alpha: used for two or more raters, correcting for chance agreement

Tests Designed

• Homogenous tests: designed to measure one factor, expected to have a high degree of internal consistency
• Dynamic tests: measure traits, states, or abilities that are fast-changing as a function of situational and cognitive experience
• Static tests: measure traits, states, or abilities that are relatively unchanging
• Restriction of range or variance: when the variance of either variable in a correlational analysis is restricted, resulting in a lower correlation coefficient

Power Tests

• Designed to allow test-takers to attempt all items within a time limit
• Measures a test-taker's ability to complete a task accurately and efficiently

Speed Tests

• Contain items of uniform difficulty with a time limit
• Reliability should be based on performance from two independent testing periods using test-retest, alternate-forms, or split-half reliability

Criterion-Referenced Tests

• Designed to provide an indication of where a test-taker stands with respect to a criterion
• As individual differences decrease, traditional measures of reliability also decrease, regardless of individual performance stability

Classical Test Theory

• Assumes that everyone has a "true score" on a test
• True score reflects an individual's ability level as measured by a particular test
• Random error affects the observed score

Domain Sampling Theory

• Estimates the extent to which specific sources of variation contribute to test scores
• Considers problems created by using a limited number of items to represent a large construct

Test Reliability

• Conceived as an objective measure of how precisely a test score assesses a domain
• Reliability is a function of the proportion of total variance attributed to true variance

Generalizability Theory

• Based on the idea that test scores vary due to variables in the testing situation
• Universe: the test situation
• Facets: number of items, amount of review, and purpose of test administration
• Given the same conditions, the same test score should be obtained (universe score)

Decision Study

• Examines the usefulness of test scores in helping test users make decisions

Systematic Error

• Factors inherent in a test that prevent accurate, impartial measurement

Item Response Theory

• The probability of a person with a certain ability level performing at a certain level on a test
• Focuses on item difficulty

Latent-Trait Theory

• A system of assumptions about measurement and the extent to which items measure a trait
• Computers are used to focus on the range of item difficulty that helps assess an individual's ability level
• If a person answers easy items correctly, the computer will move to more difficult items
• Item attributes: difficulty, discrimination, and dichotomousness

Construct Validity (Umbrella Validity)

• Covers all types of validity
• Logical and statistical
• Judgment about the appropriateness of inferences drawn from test scores regarding individual standing on a variable called a construct

Criterion Validity

• More statistical than logical
• Judgment about the adequacy of test scores in inferring an individual's standing on a criterion measure
• Criterion: a standard on which a judgment or decision may be made
• Characteristics: relevant, valid, uncontaminated
• Types of criterion validity: concurrent, predictive, and incremental validity

Factor Analysis

• Designed to identify factors or variables that are typically attributes, characteristics, or dimensions on which people may differ
• Developed by Charles Spearman
• Employed as a data reduction method
• Used to study the interrelationships among a set of variables
• Types of factor analysis: explanatory, confirmatory, and factor loading

Cross-Validation

• Validation of a test to a criterion based on a different group from the original group
• Validity shrinkage: a decrease in validity after cross-validation
• Co-validation: validation of more than one test from the same group
• Co-norming: norming more than one test from the same group

Bias

• Factors inherent in a test that systematically prevent accurate, impartial measurement
• Prevention: during test development through procedures such as estimated true score transformation

Rating

• Numerical or verbal judgment that places a person or attribute along a continuum identified by a scale
• Rating error: intentional or unintentional misuse of the scale
• Types of rating error: leniency, severity, central tendency, and halo effect
• One way to overcome rating errors is to use rankings### Discriminant Evidence
• Definition: A validity coefficient showing little relationship between test scores and/or other variables
• Importance: Used in Psychological Assessment

Measures of Central Tendency

Mode

• Definition: The most frequently occurring score in a distribution
• Use: For ordinal data, and for nominal scales and discrete variables
• Characteristics: Can be used in analyses of qualitative or verbal nature, and gives an indication of the shape of the distribution

Measures of Spread or Variability

• Definition: Statistics that describe the amount of variation in a distribution
• Use: Gives an idea of how well the measure of central tendency represents the data
• Characteristics: Large spread of values means large differences between individual scores

Measures of Spread or Variability

Range

• Definition: The difference between the highest and lowest score
• Use: Provides a quick but gross description of the spread of scores
• Characteristics: Can be affected by extreme scores in the distribution

Variance

• Definition: The square root of the average squared deviations about the mean
• Use: Equal to the square root of the variance, and measures the distance from the mean
• Characteristics: Equal to the arithmetic mean of the squares of the differences between the scores in a distribution and their mean

Measures of Location

Percentile or Percentile Rank

• Definition: Expressed in terms of the percentage of persons in the standardization sample who fall below a given score
• Use: Indicates the individual's relative position in the standardization sample
• Characteristics: Not linearly transformable, converged at the middle, and the outer ends show large intervals

Quartile

• Definition: Dividing points between the four quarters in the distribution
• Use: Specific point in the distribution

Correlation

Spearman Rho

• Definition: Used for ordinal + ordinal data
• Use: Measures the correlation between two variables

Biserial

• Definition: Used for true dichotomous + interval/ratio data
• Use: Measures the correlation between two variables

Point Biserial

• Definition: Used for nominal (true dichotomous) + nominal (true/artificial dichotomous) data
• Use: Measures the correlation between two variables

Phi Coefficient

• Definition: Used for artificial dichotomous + artificial dichotomous data
• Use: Measures the correlation between two variables

Tetrachoric

• Definition: Used for 3 or more ordinal/rank data
• Use: Measures the correlation between two variables

Kendall's Rank Biserial Differences

• Definition: Used for two separate groups, random assignment
• Use: Measures the correlation between two variables

T-Test

• Definition: Used for comparing means between two groups
• Use: One group, two scores (e.g., blood pressure before and after the lecture)

T-Test Dependent (Paired T-test)

• Definition: Used for comparing means between two groups
• Use: One group, measured at least three times

One-Way ANOVA

• Definition: Used for comparing means between three or more groups
• Use: One group, measured at least three times

One-Way Repeated Measures

• Definition: Used for comparing means between three or more groups
• Use: Three or more groups, tested for two variables

Two-Way ANOVA

• Definition: Used for comparing means between two or more groups, controlling for an additional variable
• Use: Used when you need to control for an additional variable that may be influencing the relationship between your independent and dependent variable

Utility Gain

• Definition: Estimate of the benefit of using a particular test
• Use: In Psychological Assessment

Productivity Gains

• Definition: An estimated increase in work output
• Use: In Psychological Assessment

Cut Score

• Definition: Reference point derived as a result of a judgment and used to divide a set of data into two or more classifications
• Use: In Psychological Assessment

Relative Cut Score

• Definition: Reference point based on norm-referenced considerations, not fixed per se
• Use: In Psychological Assessment

Fixed Cut Scores

• Definition: Set with reference to a judgment concerning minimum level of proficiency required
• Use: In Psychological Assessment

Multiple Cut Scores

• Definition: Refers to the use of two or more cut scores with reference to one predictor for the purpose of categorization
• Use: In Psychological Assessment

Multiple Hurdle

• Definition: Multi-stage selection process, a cut score is in place for each predictor
• Use: In Psychological Assessment

Compensatory Model of Selection

• Definition: Assumption that high scores on one attribute can compensate for lower scores
• Use: In Psychological Assessment

Angoff Method

• Definition: Setting fixed cut scores
• Use: In Psychological Assessment

Known Groups Method

• Definition: Collection of data on the predictor of interest from groups known to possess and not possess a trait of interest
• Use: In Psychological Assessment

IRT-Based Methods

• Definition: Cut scores are typically set based on test-taker's performance across all the items on the test
• Use: In Psychological Assessment

Item-Mapping Method

• Definition: Arrangement of items in a histogram, with each column containing items deemed to be equivalent in value
• Use: In Psychological Assessment

Bookmark Method

• Definition: Expert places a "bookmark" between the two pages that are deemed to separate test-takers who have acquired the minimal knowledge, skills, and/or abilities from those who are not
• Use: In Psychological Assessment

Method of Predictive Yield

• Definition: Took into account the number of positions to be filled, projections regarding the likelihood of offer acceptance, and the distribution of applicant scores
• Use: In Psychological Assessment

Discriminant Analysis

• Definition: Used to shed light on the relationship between identified variables and two naturally occurring groups
• Use: In Psychological Assessment, used to analyze data when the criterion or dependent variable is categorical and the predictor or independent variable is interval in nature

Reliability and Validity

Reliability

• Definition: The consistency of test scores
• Interpretation:
  • Excellent: 0.90 and up
  • Good: 0.80-0.89
  • Adequate: 0.70-0.79
  • Limited applicability: below 0.70

Validity

• Definition: The degree to which a test measures what it claims to measure
• Interpretation:
  • Very beneficial: above 0.35
  • Likely to be useful: 0.21-0.35
  • Depends on circumstances: 0.11-0.20
  • Unlikely to be useful: below 0.11

Item Analysis

Item-Validity Index

• Definition: Designed to provide an indication of the degree to which a test is measuring what it purports to measure
• Use: In Psychological Assessment

Item-Discrimination Index

• Definition: Measures the difference between the proportion of high scorers answering a question correctly and the proportion of low scorers answering it correctly
• Use: In Psychological Assessment

Extreme Group Method

• Definition: Compares people who have done well with those who have done poorly
• Use: In Psychological Assessment

Discrimination Index

• Definition: The difference between those proportions
• Use: In Psychological Assessment

Point-Biserial Method

• Definition: Correlation between a dichotomous variable and continuous variable
• Use: In Psychological Assessment