Test Validity and Reliability

• Use procedures and statistical analysis to establish test validity and reliability.
• Decide whether a test is valid or reliable.
• Decide which test items are easy and difficult.

Significant Culminating Performance Task

• Demonstrate knowledge and skills in determining whether tests and their items are valid and reliable.

Specific Performance Tasks and Success Indicators

• Use appropriate procedure in determining test validity and reliability.
  • Provide detailed steps, decision, and rationale in using appropriate validity and reliability measures.
• Show the procedure on how to establish test validity and reliability.
  • Provide detailed procedure from the preparation of the instrument, procedure in pretesting, and analysis in determining the test's validity and reliability.
• Provide accurate results in the analysis of item difficulty and reliability.
  • Make appropriate computation, use of software, reporting of results, and interpretation of the results for the tests of validity and reliability.
• Should have prepared a test following the proper procedure with clear learning targets (objectives), table of specifications, and pretest data per item.
• Assessment becomes valid when test items represent a good set of objectives in the table of specifications.

Test Reliability

• In order to establish the validity and reliability of an assessment tool, you need to know the different ways of establishing test validity and reliability.
• Reliability is the consistency of responses to measure under three conditions:
  • When retested on the same person.
  • When retested on the same measure.
  • Similarity of responses across items measuring the same characteristic.
• Consistent response is expected when the test is given to the same participants.
• Reliability is attained if the responses to the same test are consistent with the same test (or its equivalent) or another test that measures the same characteristic when administered at a different time.
• There is reliability when a person responds in the same way or consistently across items that measure the same characteristic.

Factors Affecting Reliability

• Number of Items in a Test: More items increase the likelihood of high reliability due to a larger pool of items.
• Individual Differences of Participants: Factors like fatigue, concentration, innate ability, can affect test performance and the consistency of answers.
• External Environment: Room temperature, noise level, depth of instruction, exposure to materials can affect examinee responses in a test.

Ways to Establish Test Reliability

• There are different ways in determining the reliability of a test.
• The specific kind of reliability depends on the variable, type, and the number of versions of the test.

Test-Retest Reliability

• Administer the same test to the same group of examinees at two different times.
• Maintain a time interval of no more than 6 months between administrations, especially for tests measuring stable characteristics.
• A short post test with at least a 30 minute interval can also be given.
• Responses should be similar across both administrations.
• Applicable for tests measuring stable variables (aptitude, psychomotor skills).
• Correlate test scores from the first and second administrations.
• A significant and positive correlation indicates temporal stability over time.
• Correlation refers to a statistical procedure where a linear relationship is expected for two variables.
• Pearson Product Moment Correlation can be used because test data are usually in an interval scale.

Parallel Forms Reliability

• Use two versions of a test measuring the same skills.
• Administer both forms to the same group of participants.
• Responses on both forms should be similar.
• Applicable when there are two versions of the test, especially for repeated administrations to different groups.
• Correlate test results for the first and second forms.
• A significant and positive correlation coefficient is expected, indicating the versions are consistent.
• Pearson r is usually used for this analysis.

Split-Half Reliability

• Administer a test to a group of examinees.
• Split the test into two halves (usually odd vs. even items).
• Correlate the scores from the item halves.
• Each examinee will have two scores coming from the same test.
• Split-half is applicable if the test has a large number of items.
• Correlate the two sets of scores using Pearson r.
• Apply the Spearman-Brown Coefficient.
• The correlation coefficient obtained should be significant and positive to mean that the test has internal consistency reliability.

Internal Consistency Reliability Using Kuder-Richardson and Cronbach's Alpha Method

• Determine if scores for each item are consistently answered by examinees.
• Administer the test and record scores for each item.
• Works well with assessment tools having a large number of items and scales/inventories.
• Use Cronbach's alpha or Kuder-Richardson to determine internal consistency of the items.
• A Cronbach's alpha of 0.60 or above indicates internal consistency.

Inter-Rater Reliability

• Determine the consistency of ratings by multiple raters using rating scales or rubrics.
• Focuses on the similarity or consistency of ratings provided by multiple raters using the same assessment tool.
• Applicable when the assessment requires multiple raters.
• Use Kendall's tau coefficient of concordance to determine if ratings by multiple raters agree.
• A significant Kendall's tau value indicates that raters concur.

Statistical Analysis and Linear Regression

• Statistical analysis, particularly linear regression, is required to determine test reliability.

Linear Regression

• Linear regression is demonstrated with two variables measured, such as two sets of scores in a test taken at two different times by the same participants.
• When the two scores are plotted in a graph (with X- and Y-axis), they tend to form a straight line.
• The straight line formed for the two sets of scores can produce a linear regression.
• When a straight line is formed, we can say that there is a correlation between the two sets of scores.
• The graph that represents this correlation is called a scatterplot.

Computation of Pearson r Correlation

• Correlation coefficient is the index of the linear regression.
• Strong correlations occur when points in a scatterplot closely follow a linear line.
• Correlation coefficient is calculated as:
  • r = [N(ΣXY) – (ΣX)(ΣY)] / √{[N(ΣX²) – (ΣX)²][N(ΣY²) – (ΣY)²]}

Positive and Negative Correlation

• A positive correlation coefficient means the higher the scores in X, the higher the scores in Y, and vice versa.
• A negative correlation coefficient means the higher the scores in X, the lower the scores in Y, and vice versa.
• A positive correlation indicates reliability or consistency.

Strength of Correlation

• The value of the correlation coefficient indicates the strength of the reliability of the test
• The closer the value to 1.00 or -1.00 means the stronger the correlation.
• 0.80-1.00: Very strong relationship
• 0.6-0.79: Strong relationship
• 0.40-0.59: Substantial/marked relationship
• 0.2-0.39: Weak relationship
• 0.00-0.19: Negligible relationship

Significance of the Correlation

• The correlation obtained between two variables could be due to chance.
• In order to determine if the correlation is free of certain errors, it is tested for significance.
• When a correlation is significant, it means that the probability of the two variables being related is free of certain errors.
• When the value computed is greater than the critical value, it means that the information obtained has more than 95% chance of being correlated & is significant.

Cronbach's Alpha procedure

• Another statistical analysis mentioned to determine the internal consistency of test is the Cronbach's alpha.
• The consistency of ratings can also be obtained using a coefficient of concordance.
• The Kendall's w coefficient of concordance is used to test the agreement among raters.
• In the formula, m is the numbers of raters.

Test Validity

• A measure is valid when it measures what it is supposed to measure, such as achievement or objectives.
• Measures should have items that contain factors that are highly correlated.
• If an entrance exam is valid, it should predict success in the students` grades.

Types of Validity

• Content Validity: items represent the domain being measured.
  • Procedure: The items are compared with the objectives of the program. Reviewer conducts the checking.
• Face Validity: the test is well-presented, free of errors, and well-administered.
  • Procedure: The test items and layout are reviewed and tried out on a small group of respondents.
• Predictive Validity: a measure should predict a future criterion.
  • Procedure: A correlation coefficient is obtained where the X-variable is used as the predictor and the Y-variable as the criterion.
• Construct Validity: the components or factors of the test should contain items that are strongly correlated.
  • Procedure: The Pearson r can be used to correlate the items for each factor.
• Concurrent Validity: when two or more measures are present for each examinee that measure the same characteristic.
  • Procedure: The scores on the measures should be correlated.
• Convergent Validity: When the components or factors of a test are hypothesized to have a positive correlation
  • Procedure: Correlation is done for the factors of the test.
• Divergent Validity: When the components or factors of a test are hypothesized to have a negative correlation
  • Procedure: Correlation is done for the factors of the test.

Determining item difficulty

• Determine accuracy and discrimination
• Item is difficult: If majority of students are unable to provide the correct answer.
• Items is easy: If majority of the students are able to answer correctly.
• Item can discriminate: If the examinees who score high in the test can answer more the items correctly than examinees who got low scores.

Procedures to determine item difficulty and discrimination

• Get the total score of each student and arrange scores from highest to lowest.
• Obtain the upper and lower 27% of the group.
  • Multiply 0.27 by the total number of students, and you will get a value of 2.7. The rounded whole number value is 3.0
  • Get the top three students and the bottom three students based on their total scores.
• Obtain the proportion correct for each item.
  • This is computed for the upper 27% group and the lower 27% group
  • Divide the total number of students by the summated total per item.
• The item difficulty is obtained using the following formula: Item difficulty = (PH + PL) / 2

Ranges to interpret item difficulty

• 0.76 or higher: Easy Item
• 0.25 to 0.75: Average Item
• 0.24 or lower: Difficult Item
• The index of discrimination is obtained using the formula: Item discrimination = pH - pL

Ranges for Index discrimination

• 0.40 and above: Very good item
• 0.30-0.39: Good item
• 0.20-0.29: Reasonably Good item
• 0.10-0.19: Marginal item
• Below 0.10: Poor item
• When developing a teacher-made test, good to have items that are easy, average, and difficult with positive discrimination indices.
• If you are developing a standardized test, the rule is more stringent as it aims for average items or not so easy nor difficult items with a discrimination index of at least 0.3.