Test Scores: Norms and Standardization

• Two key concepts are needed to interpret test/scale scores: norms and reliability
• Norms help make sense of an individual's score using comparative frames of reference
• Reliability indicates whether test scores are consistent or repeatable

Part I: Norms

• Norms represent the distribution of scores on a test from a well-established group
• Norms indicate an examinee's standing relative to others of the same age, gender, etc.
• A representative, large, and heterogeneous sample is selected to develop the norms

Norms and Test Standardization

• Raw scores
• Criterion-referenced tests
• Essential statistical concepts
• Raw score transformations
• Selection of a norm group

Raw Scores

• Raw scores are the basic level of information from a psychological test
• Raw scores are meaningless in isolation
• Raw scores becomes meaningful when compared with others' scores
• Raw scores becomes meaningful when related to norms obtained by a representative sample
• Norms help determine if an obtained score is low, average, or high
• All norms statistically summarize a large body of scores

Norm vs. Criterion-Referenced Tests

• Norm-Referenced Tests classify examinees from low to high
• Norm-Referenced Tests use a representative sample of individuals
• Items are chosen to provide maximal discrimination
• Example: IQ tests, which determine if a test taker is more intelligent than others
• Criterion-Referenced Tests compare accomplishments to a performance standard (%)
• Content is selected based on relevance in the curriculum
• Criterion-Referenced Tests identify an examinee's mastery or non-mastery of specific behavior
• Example: a driving test

Essential Statistical Concepts

• Measures of Central Tendency: mean, median, mode
• Measures of Variation: standard deviation (SD), variance
• Raw data are difficult to display,so summarizing and organizing it in a meaningful pattern is needed
• Frequency tables, histograms, and frequency polygons visually represent the data

Frequency Tables & Histograms

• A frequency table is needed to build a histogram
• A frequency table specifies a small number of equal-sized intervals, counting how many scores fall within each interval
• Histograms display the developed frequency tables
• Histograms are bar graphs: bars represent the interval of the frequency table, the height of each bar indicates the frequency

Constructing Frequency Polygons

• A frequency table that includes interval midpoints and frequencies is needed
• Dots are placed above each interval midpoint at the height of the class frequency
• Two dots are put on the horizontal axis, one before the first point and one after the last point
• These two new points are not in the table so the frequency of each is 0
• Dots are connected with straight lines

Measures of Relative Standing

• Used to provide information about where a score falls in relation to other scores in a distribution
• Commonly used measures include: Z-Scores, Normal Distribution, Stanine, T Scores

Z-Scores

• A measure of an observation's distance from the mean, measured in standard deviation units
• Z = (X - X̄) / SD
• Z-scores are advantageous when grades of two students in different classes needs to be compared
• To find a z-score, subtract the mean and divide by the standard deviation
• If a z-score is zero, it's on the mean
• If a z-score is positive, it's above the mean
• If a z-score is negative, it's below the mean
• A z-score of 1 is 1 SD above the mean
• A z-score of -2 is 2 SDs below the mean
• The mean of all z-scores for a sample is 0, and the SD is 1

Normal Distribution

• The scores of a quiz of 30 students is measured and graphed
• As the scores continue to plot, a pattern emerges
• As scores become larger or smaller, there are fewer people with that measurement
• Most measurements tend to fall in the middle, with fewer approaching the high and low extremes
• Smoothing the lines creates a bell-shaped curve
• This bell-shaped curve is known as the "Bell Curve" or the "Normal Curve."term scores for 51 students in research skills created a curve
• Mean, mode, and median will fall on the same point
• Normal distributions are a family of distributions that have the same general shape
• Normal distributions are symmetric, the left side is a mirror of the right
• Scores are more concentrated in the middle than in the tails
• Normal distributions differ in how spread out they are, but the area under each curve is the same
• If data fits a normal distribution, about 68% of subjects will fall within one standard deviation from the mean
• 95% will fall within two standard deviations
• Over 99% will fall within three standard deviations

Standard Deviation

• The mean and standard deviation describe a set of scores
• Scores grouped closely together have a smaller standard deviation than scores spread farther apart
• If raw scores distribute normally (like midterm score data with a mean of 17 and an SD of 2.24), the data has the predicted properties
• A subject's raw score and the mean/SD, can determine their standardized score if the distribution of scores is normal
• Standardized scores are useful when comparing a student's performance across different tests, or when comparing students
• The number of points that one standard deviation equals varies from distribution to distribution
• Skew refers to the tail of the distribution
• In a negative skew, the tail is on the negative (left) side of the graph,
• In a positive skew, the tail is on the positive (right) side of the graph

Using the Normal Distribution

• Skewed data do not possess the characteristics of the normal curve
• In skewed data the mean, mode, and median do not fall on the same score
• The mode is represented by the highest point, mean is toward the side with the tail, and the median falls between the mode and mean
• Fatima's' score standing can be determined on the normal distribution, to compare results
• Six categories cover 99% of students: very weak, weak, below average, above average, good, very good

Stanine

• Stanine (STAndard NINE) scales test scores on a 9-point standard scale
• Stanines are similar to normal distributions, where scores are a bell curve sliced into 9 pieces
• The mean of Stanines is 5 and the SD is 2
• Stanines assign a number to a member of a group, relative to all members in that group
• Whole positive numbers are used
• 9 ratings: Bottom 4%, next bottom 7%, next bottom 12%, next bottom 17%, middle 20%, next top 17%, next top 12%
• A person with a score of 9 is in the top 4% of scorers, while a person with a score of 1 is in the bottom 4%

Z-Scores vs Stanine

• Z-scores can be expressed with decimals while stanines are always positive whole numbers from 0 to 9
• Two scores in a stanine can be farther apart than two scores in adjacent stanines which reduces their value

T Scores

• T scores are normally distributed with a mean of 50 and an SD of 10
• T scores can be found if the distribution of scores is normal
• Converting each score to a Z score
• Multiplying the Z by 10
• Adding or subtracting (depending on the sign of Z) to or from 50

Converting Z-Scores

• Once there is a set of z-scores, converting to any other scale can be done
• New score = Z-score (SD of new scale) + mean of the new scale

Percentile Ranks

• Percentile Rank tells the percentage of scores in a reference group that fall below a particular raw score
• 93rd percentile means that 93 percent of the scores in the reference group fall equal or below the score
• Percentile ranks have ordinal measurement properties and can be calculated using SPSS

Types of Standard Scores

• Mean and SD are used to determine the type of standard scores

Selecting a Norm Group

• Age and grade norms
• Local and subgroup norms

Age and Grade Norms

• Age norms facilitate same-aged comparisons
• Examinee performance is compared with standardization subjects of the same age
• The age span can vary from a few months to a decade
• Grade norms describe the level of performance for each grade, useful for school settings

Local and Subgroup Norms

• Local norms are derived for representative local examinees as opposed to a national sample (state norms vs. national norms
• Subgroup norms consist of scores from an identified subgroup (African American, Females,...)

Part II: Reliability

• Reliability is consistency in measurement
• A measure has high reliability if it produces similar results under consistent conditions
• Continuum ranges from minimal consistency to near perfect
• Reliability indicates accuracy which is related to measurement error
• Classical test theory of measurement

Classical Test Theory

• Factors that contribute to consistency- factors related to the attribute being measured
• Factors that contribute to inconsistency- factors related to the individual or the test (but NOT the attribute being measured)
• X = T + e (X is observed score, T is true score, e is error)
• e = X-T
• e cannot be 0 and T value is unknown

Sources of Measurement Error

• Item Selection: Items selected in tests might not be fair to all examinees
• Test Administration: uncomfortable room temperature, dim lighting, noise, examinees' motivation, anxiety, concentration, and fatigue
• Test Scoring: non-multiple choice scoring requires judgment, especially with essays
• Systematic Measurement Error

Types of Error

• All discussed errors were unsystematic measurement error (random error)
• Systematic error: a test consistently measures something other than the trait intended
• X = T + es + eu where es is the systematic error and eu is the unsystematic error
• Random error is caused by any factor that randomly affects measurement of the variable across the sample
• Random error does not have any consistent effects across the entire sample.
• Random error pushes scores up or down randomly
• In a distribution they would have
• Systematic error affects the average (bias)