Lesson 9 Utility Analysis PDF

Psychological Assessment 1 Utility  What is Test Utility? Practical value of Usefulness of a using a test to aid test in decision making  Utility-related Questions How useful is this test in ter...

Psychological Assessment 1 Utility  What is Test Utility? Practical value of Usefulness of a using a test to aid test in decision making  Utility-related Questions How useful is this test in terms of cost efficiency? How useful is this test in terms of savings in time? What is the comparative utility of this test? That is, how useful is this test as compared to another test? What is the clinical utility of this test? That is, how useful is it for purposes of diagnostic assessment or treatment? What is the diagnostic utility of this neurological test? That is, how useful is it for classification purposes? How useful is this medical school admissions test used in assigning a limited number of openings to an overwhelming number of applicants? How useful is the addition of another test to the test battery already in use for screening purposes?  Utility-related Questions How useful is this personnel test as a tool for the selection of new employees? Is this particular personnel test used for promoting middle-management employees more useful than using no test at all? Is the time and money it takes to administer, score, and interpret this personnel promotion test battery worth it as compared to simply asking the employee’s supervisor for a recommendation as to whether the employee should be promoted? How useful is the training program in place for new recruits? How effective is this particular clinical technique? Should this new intervention be used in place of an existing intervention?  What is Test Utility? Factors That Affect a Test’s Utility – Psychometric Soundness – Costs – Benefits  Psychometric Soundness Reliability Validity  Costs Expenditures associated with testing or not testing Disadvantages Losses Expenses *both considering economic and non-economic terms  Refers to profits, gains, or advantages – an increase in the quality of workers’ performance; – an increase in the quantity of Benefits workers’ performance; – a decrease in the time needed to train workers; – a reduction in the number of accidents; – a reduction in worker turnover.  What is a Utility Analysis? How is a Utility Analysis Conducted? Expectancy Data Utility The Brogden-Cronbach-Gleser formula Analysis Some Practical Considerations The pool of job applicants The complexity of the job The cut score in use  What is Utility Analysis? A utility analysis may be broadly defined as a family of techniques that entail a cost–benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment. Utility analysis is not one specific technique used for one specific objective. Rather, utility analysis is an umbrella term covering various possible methods, each requiring various kinds of data to be inputted and yielding various kinds of output.  Expectancy Data An expectancy table can provide an indication of the likelihood that a test taker will score within some interval of scores on a criterion measure—an interval that may be categorized as “passing,” “acceptable,” or “failing.” For example, with regard to the utility of a new and experimental personnel test in a corporate setting, an expectancy table can provide vital information to decision-makers. An expectancy table might indicate, for example, that the higher a worker’s score is on this new test, the greater the probability that the worker will be judged successful. The test is working as it should and, by instituting this new test on a permanent basis, the company could reasonably expect to improve its productivity.  The Taylor-Russell tables provide an estimate of the extent to which inclusion of a particular test in the selection system will improve selection. More specifically, the tables provide an estimate of the percentage of employees hired Taylor-Russell by the use of a particular test tables who will be successful at their jobs, given different combinations of three variables: the test’s validity, the selection ratio used, and the base rate. Assists in judging the utility of a particular test by determining the increase over current procedures  The Naylor-Shine tables entails obtaining the difference between the means of the selected and unselected groups to derive an index of what the test (or some other tool of assessment) is adding to Naylor-Shine already established procedures tables Assists in judging the utility of a particular test by determining the increase in average score on some criterion measure through concurrent validation procedures Brogden- Cronbach- Gleser (BCG) formula Brogden- Cronbach- Gleser (BCG) formula Brogden- Cronbach- Gleser (BCG) formula  Question: Would it be wise for a company to make an investment of $24,000 to receive a return of about $300,000?  Modification of the BCG When might it be better to present utility gains in productivity terms rather than financial terms? Productivity gain refers to an estimated increase in work output. In this modification of the formula, the value of the standard deviation of productivity, SDp, is substituted for the value of the standard deviation of performance in dollars, SDy (Schmidt et al., 1986, as cited by Cohen, 2018). The result is a formula that helps estimate the percent increase in output expected through the use of a particular test. The revised formula is: productivity gain = (N)(T)(rxy)(SDp)(Zm) − (N)(C)  Decision Theory and Test Utility Cronbach and Gleser (1965) presented 1) a classification of decision problems; 2) various selection strategies ranging from single-stage processes to sequential analyses; 3) a quantitative analysis of the relationship between test utility, the selection ratio, cost of the testing program, and expected value of the outcome; and 4) a recommendation that in some instances job requirements be tailored to the applicant’s ability instead of the other way around (a concept they refer to as adaptive treatment). *Integrate base rate, hit rate, miss rate, false positive, and false negative Methods for Setting Cut Scores The Known The Angoff IRT – Based Groups Method Methods Method Edward L. Thorndike Other Methods Method of predictive yield  Cut Score A cut score is a (usually numerical) reference point derived as a result of a judgment and used to divide a set of data into two or more classifications, with some action to be taken or some inference to be made on the basis of these classifications. Relative cut score (reference point or norm-referenced cut score) Fixed cut score (absolute cut score with reference to minimum level of proficiency)  Cut Score Multiple cut scores refers to the use of two or more cut scores with reference to one predictor for the purpose of categorizing test takers. At every stage in a multistage (or multiple hurdle) selection process, a cut score is in place for each predictor used. The cut score used for each predictor will be designed to ensure that each applicant possess some minimum level of a specific attribute or skill.  The Angoff Method Devised by William Angoff (1971), the Angoff method for setting fixed cut scores can be applied to personnel selection tasks as well as to questions regarding the presence or absence of a particular trait, attribute, or ability. When used for purposes of personnel selection, experts in the area provide estimates regarding how test takers who have at least minimal competence for the position should answer test items correctly. Issue: inter-rater reliability Known Groups Method The known groups method entails collection of data on the predictor of interest from groups known to possess, and not to possess, a trait, attribute, or ability of interest. Based on an analysis of this data, a cut score is set on the test that best discriminates the two groups’ test performance.  Item Response Theory (IRT-) Based Methods In the IRT framework, each item is associated with a particular level of difficulty. In order to “pass” the test, the test taker must answer items that are deemed to be above some minimum level of difficulty, which is determined by experts and serves as the cut score. Item-mapping method (used in licensure exams; with TOS) Bookmark method (academic applications; issues on training of experts; possible floor and ceiling effects; optimal length of item booklets) Predictive Yield (Edward Thorndike) A norm-referenced method The method of predictive yield was a technique for setting cut scores which took into account the number of positions to be filled, projections regarding the likelihood of offer acceptance, and the distribution of applicant scores.  Discriminant Analysis Another approach to setting cut scores employs a family of statistical techniques called discriminant analysis (also referred to as discriminant function analysis). These techniques are typically used to shed light on the relationship between identified variables (such as scores on a battery of tests) and two (and in some cases more) naturally occurring groups (such as persons judged to be successful at a job and persons judged unsuccessful at a job).  Reference Cohen, Ronal Jay & Swerdlik, Mark E. (2018). Psychological testing and assessment Ninth Edition: McGraw-Hill Education.