week7-1_testing.ppt

Item Analysis Chapter 10 01/18/24 1 Items and Item Analysis Items are the questions in test. Item analysis focuses on the functioning of individual test items. It helps to find bad items and remove them to improve tests. It helps to explain low reliability and validity. The aim is to devel...

Item Analysis Chapter 10 01/18/24 1 Items and Item Analysis Items are the questions in test. Item analysis focuses on the functioning of individual test items. It helps to find bad items and remove them to improve tests. It helps to explain low reliability and validity. The aim is to develop better tests. If you improve quality of individual items, you improve overall quality of the test 01/18/24 2 Item Analysis Does an item do a good job to measure the behavior or not? 3 main types; Distractor analysis Item difficulty analysis Item discrimination analysis 01/18/24 3 Distractor Analysis In a multiple choice test, there is only one correct answer, the rest are referred to as ‘distractors’. Q: What is the first name of Freud? a)Sigism They should be confusing so that b)Sigmund people who do not know the c)Sinan answer can choose randomly 1 among 4. the answers should be d)Siam close to each other. The aim is to examine the frequency of each incorrect response chosen by examinees. 01/18/24 4 Distractor Analysis 01/18/24 5 Distractor Analysis In a good test, people who know the right answer should be able to choose it and others should guess randomly among the possible responses. It should get people away from the correct answer if they don’t know the correct one precisely. Good distractor distracts equally. N=50 a)10 b)20 c)10 d)10 Right answer20 Wrong30 6 Distractor Analysis What is the capital city of Denmark? a)Stockholm b)Copenhagen c)Helsinki d)Oslo If n=50, a.1 b.47 c.1 d.1 bad distractors. a very easy question 7 Distractor Analysis 8 Distractor Analysis # of people expected = # of people answering incorrectly to choose each distractor # of distractors =39/3 9 Item Difficulty Analysis ‘How many people answered the item correctly?’ Only applicable to maximum performance tests Item difficulty = percentage if people who answer the items correctly p for item = # of people answering correctly # of people taking the test p is a behavioral measure. p can range from 0 (no one answers correctly) to 1 (everyone answers correctly). 01/18/24 10 Item Difficulty Analysis Item difficulty index can range from 0.0 to 1.0 Easier items have a larger decimal number Harder items have a smaller decimal number p values of .0 OR 1.0 does not contribute to measuring individual differences so it means the item is useless. Extreme p values directly restrict the variability of test scores. So, p values of .5 being optimum. Maximum variance. It’s ideal. .1 means almost noone/everyone _______ answered correctly, .9 means almost noone/everyone _______ answered correctly. Both situation declares that test items are really bad. 01/18/24 11 Item Difficulty Analysis If question is a true-false item, what is the percentage of answering the question by guessing? %50 So p value of .5 is not optimum for true-false questions. 01/18/24 12 Item Difficulty Analysis According to Kaplan and Saccuzzo (2017), the optimal difficulty level for items is usually about halfway between 100% of the respondents getting the item correct and the level of success expected by chance alone. Thus, the optimum difficulty level for a four-choice item is approximately .625. (1+0.25)/2 Optimum difficulty level for a true-false item is approximately .75. (1+0.5)/2 01/18/24 13 Item Discrimination Analysis ‘Are responses to this item related to responses to other items on the test?’ Aim is; to find out the extent to which responses to each item discriminate those who receive high vs. low scores on the test. to discover which items best measure the construct or attribute. 01/18/24 14 Item Discrimination Analysis 3 strategies to measure the discriminating power of an item. 1.Discrimination index (D) 2.Item-total correlation 3.Interitem correlation 01/18/24 15 Item Discrimination Analysis 1. Discrimination index (D) -every test item is ‘an individual test’ so people doing well in the whole test should also perform well in individual items. -extreme groups (e.g. lowest and highest group 27%) should differ in # of correct answers. -formula for D= 01/18/24 16 Item Discrimination Analysis 1. Discrimination index (D)  For example: U/nu = 0.80 (80% of the examinees in the top group answered the item correctly) L/nl = 0.30 (30% of the examinees in the bottom group answered the item correctly) D = 0.80 − 0.30 = 0.50 01/18/24 17 Item Discrimination Analysis 1. Discrimination index (D) Item 1 & 2 seems difficult to lower group. GOOD! Item 3 does not show much discriminating power. This item is equally difficult for the upper and lower groups. (-) D indicates that the item is easier for lower people who do poorly on the test, which is bad. Although item discriminates, but it’s in the wrong direction. 18 Item Discrimination Analysis 2. Item-total correlation -correlation between the total test score and item score. range: -1 to 1, zero indicating that the item does NOT discriminate between low and high scores. (+) correlation indicates that item successfully discriminates between those who do well and those who do poorly. (-) correlation indicates that the scores on the item and scores on the test disagree. Those who do well on an individual item with a (-) item-total r do poorly on the test. 01/18/24 19 Item Discrimination Analysis 3. Interitem correlation -correlations for all test item. All items should measure the same thing. -if an item doesn’t correlate with any other items, it should be rewritten or discarded altogether. If an item correlates with some items but not all items, it might show that test measures different dimensions/factors. 01/18/24 20 Interactions among item characteristics The difficulty of an item depends on the plausibility of the distractors. Item difficulty can be change by rewriting some distractors. 01/18/24 21 Interactions among item characteristics p=1.0 if everyone chooses the correct response, P=.0 if everyone chooses an incorrect response So, such items do not discriminate good and bad performers. 01/18/24 22