week5-2_testing.ppt

Validity II Chapter 9: Validity for Decisions: Criterion-related Validity 01/18/24 1 Construct and CriterionRelated Validity 01/18/24 2 Construct and CriterionRelated Validity  e.g.success  These are scores both ‘predictor’ and ‘criterion’ sides, construct and measures 01/18/24 3 e needed...

Validity II Chapter 9: Validity for Decisions: Criterion-related Validity 01/18/24 1 Construct and CriterionRelated Validity 01/18/24 2 Construct and CriterionRelated Validity  e.g.success  These are scores both ‘predictor’ and ‘criterion’ sides, construct and measures 01/18/24 3 e needed. Are our decisions valid? Validity of Decisions: How accurate are the decisions based on the test? Criterion-related validity: Do the test scores correlate with criterion, in other words, predict outcome? Can test scores predict performance on a criterion? (e.g., SAT predict college GPA) Validity coefficients: correlation coefficient between test scores and criterion. 01/18/24 4 Are our decisions valid? On the basis of person’s score on measurement, you try to estimate which applicants would perform well on the job. A good test allows you to make reasonably accurate predictions. 01/18/24 5 Are our decisions valid? The variable of primary interest is the outcome measure, called a criterion. Criterion is a measure that could be used to determine the accuracy of decision. (e.g. Work performance on the job) Criteria must be relevant, uncontaminated and representative of the domain to be predicted. 01/18/24 6 Are our decisions valid? Evidence of criterion-related validity, when the test demonstrates that its scores are systematically related to relevant criterion. Two types; Predictive: Test scores are used to estimate outcomes to be measured at a later date. Concurrent: Test scores and criterion information are obtained simultaneously. 01/18/24 7 Predictive studies involve a time interval between test and criterion. In concurrent studies, the test and criterion are measured at the same time. 01/18/24 8 Predictive validation Test scores are obtained before making decisions. 1.Obtain test scores, but do not use the test, either directly or indirectly, in making hiring decisions. 2.At some later time, obtain performance measures for those people hired, and correlate these measures with test scores to obtain the predictive validity coefficient. 01/18/24 9 Predictive validation Obtain criterion (e.g. performance) Measure and correlate test scores Needs a random sample. Theoretically the best strategy, but has many practical and ethical problems. Not a realistic one.  It is impractical to hire people, admit them to school on a random basis.  Decisions are made about applicants without test scores.  Failure on the job is a very negative experience. Have substantial losses in terms of training costs and lost productivity.  01/18/24 10 Concurrent validation (the practical alternative) Test and criterion scores are obtained in the same time in a preselected sample. The most fundamental difference between predictive and concurrent validity is not time interval. Concurrent validity coefficient is obtained in a preselected sample (e.g. Present employees, students already accepted) The sample is preselected, not a random sample. Correlation between test and criteria. (e.g. Correlation between test scores and school grades) 01/18/24 11 Concurrent validation (the practical alternative) Adv: practical, quick (test and criterion scores obtained simultaneously, no time interval), easy (no random sampling) Disadv: range restriction. Range is smaller. Caused by selection because people are selected according to their test scores (e.g. Bad performers drop out) So, only those with high test scores are selected.  In a restricted sample, test measures the difference between moderate and good workers. The worst end of the distribution is missing.  01/18/24 12 Validity Coefficient The relationship between test scores and criterion measures rxy 01/18/24 13 In the population of applicants, the correlation between test and criterion. 01/18/24 14  those with test  scores above 60 When people are selected on the basis of their test scores, the range of the predictor is directly restricted. 01/18/24 15 Validity Coefficient Usually quite small (r=.30-.50) The lower reliability of the tests, the lower validity coefficients. Correlation between test scores and criterion does not give the full picture. Another problem is about population. For example, in a work setting, there is a number of workers who have extensive experience on the job as well as new workers. They might have completely different abilities. So, a test that predicts the performance of experienced ones may not be useful in predicting performance of new ones. 01/18/24 16 Tests and Decisions We need to evaluate the accuracy of decisions. So compare predictions with the outcome of decisions. 01/18/24 17 Decision Theory 01/18/24 18 Tests and Decisions True positive. True negative  represent accurate decisions. False positive. False negative decision error. 01/18/24 19 The cost of false negatives Deciding that someone is not suicidal because he/she is below the cutoff score when, in fact, he or she is suicidal may allow a preventable suicide. If the cost of a false negative is high, then a test developer might lower the cutting score. With a lower cutting score, the test will make more but safer errors. 01/18/24 20 The cost of false positive someone is selected for a job on the basis of a test. Once on the job, the person does poorly and gets fired. High costs sometimes accompany this type of error. For instance, time and money might be invested to train a person who cannot really do the job. If the costs of a false positive are high, then you may want to raise the cutting score. 01/18/24 21 Decision Theory Base ratio=proportion of the population who meet the criterion. (e.g. if 50% pass a training course, base ratio is .50) High base ratio means many true positive and some false negatives. Low base ratio means many true negative and some false positive decisions. Base ratio of .50 is the best for test use in decisions. 01/18/24 22 Decision Theory Selection ratio= rating of positions to applicants. (e.g. 12 applicants for 10 positions. 0.83) The lower the selection ratio is, more a test’s validity influences the proportion of success. 100 applicants but only 1 position. In this case, the test should be valid to select the perfect 1 applicant! The selection rate, base rate and the validity influence the outcomes of decisions. 01/18/24 23 Tests and Decisions When decisions are made on a random basis… A base rate of .60 and a selection ratio of .50, 30% of the decisions made at random will be true positives. 01/18/24 24 Tests and Decisions True positive. True negative  represent accurate decisions. False positive. False negative decision error. 01/18/24 25 Tests and Decisions When decisions are made on a valid test… When the validity of the test is equal to 0.0, the probability of TP is exactly same as when decisions are made at random. When validity coefficient gets higher, we observe an increment in the likelihood of TP decisions. 01/18/24 26 Tests and Decisions Tests are used for making BETTER decisions than without the tests. But, how much better? Utility Theory suggests 2 things to consider a test’s impact on decisions: It’s ability to increase the number of correct decisions (true positives and true negatives). (who can be a good pilot?) The value of correct decision. (which is more important and hard to decide: value of choosing a good pilot) 01/18/24 27 Utility Theory When base rate, selection ratio and coefficient are known, the effect of test can be determined easily. What about determining the value? Productivity gain: ‘the amount money gained if a test is used.’ 01/18/24 28