Test Construction and TOS Making PDF

TEST CONSTRUCTION AND TOS MAKING RO B E RTO J. M O N TERO Assessment of students is based on observations, oral conversations (conferences) and analyzing students’ outputs. Evaluation is the final judgment about the quality or worth of something, such as the educational program, teacher’s performanc...

TEST CONSTRUCTION AND TOS MAKING RO B E RTO J. M O N TERO Assessment of students is based on observations, oral conversations (conferences) and analyzing students’ outputs. Evaluation is the final judgment about the quality or worth of something, such as the educational program, teacher’s performance, and students’ achievements. A WELL DEFINED SYSTEM OF EVALUATION ❖Enable one to clarify goals ❖Check upon each phase of development ❖Diagnose learning difficulties ❖Plan carefully for remediation Domains of Objective Affective Cognitive Psychomotor Bloom’s Taxonomy of Objective Knowledge – requires recognition and/or recall of basic facts and terms as well as specifics and details. This is the least complex type. Locate Record Tell Name List Recall state Reproduce KNOWLEDGE Example: It is a process of changing water into water vapor. A. Condensation B. Evaporation C. Precipitation D. Solidification Comprehension – represents most basic form of understanding, requiring the ability to restate or summarize information without altering its meaning. Convert distinguish Explain generalize Restate summarize defend estimate express illustrate translate reconstruct discuss expand extend predict outline review COMPREHENSION Example: A pendulum consists of a sphere hanging from a string. What will happen to the period of the pendulum if the mass of the sphere is doubled? A. It will increase B. It will decrease C. It will remain unchanged. D. More formation is needed to determine what will happen Application – application of general principles and techniques toward the solution of clearly stated problem. The examinee will apply the appropriate principle, techniques, or theoretical approach toward solution. Apply associate solutions manipulate Solve write demonstrate construct Illustrate put in compute practice perform dramatize APPLICATION Example: Carlos had some cookies in a bag. He gave 9 cookies to each of 7 children, and kept the remaining 4. Which of the following number expressions tells how many cookies Carlos had it first? A. (7x9) + 4 B. (7 X 9) – 4 C. (7 x 9) x 4 D. (7 x 9) / 4 Analysis – be able to extract the salient features from a block of information and determine their implications. Analyze conclude discriminate experiment appraise criticize determine Test infer discover discern ANALYSIS Example: [The poem is included here.] The chief purpose of stanza 9 is to: A. Delay the ending to make the poem symmetrical. B. Give the reader a realistic picture of the return of the cavalry. C. Provide material for extending the simile of the bridge to a final point. D. Return the reader to the scene established in stanza 1. Synthesis – assembly of information, knowledge, and skills into a form which constitutes an explanation, plan or unique set of operations or constructs. Arrange assemble categorize compose formulate Modify predict propose theorize produce Generalize manage rearrange hypothesize forecast EX. OF SYNTHESIS 1. What hypothesis could be formulated that would adequately explain why man should man should limit the population? 2. What conclusion can you formulate after studying on the causes of floods in our locality? Evaluation – provides examinees with information, cases or applications which must be evaluated either in terms of criteria which are independent of the content of the information or criteria which are determined by the content. Judge summarize critique justify compare explain measure discriminate criticize Conclude develop criteria revise Assess evaluate contrast EX. OF EVALUATION 1. Is it proper for a woman to ask a man for a date? 2. Should Catholic schools accept Non-Catholic teachers? 3. Burning of garbage in your backyard is dangerous. Why? Ex. 1. Direction: Indentify the following statements, questions, phrase according to the level of skills. Write K for knowledge, C for Comprehension, Ap for Application on the space provided before the number. 1. Memorize the prayer “ Our Father”. 2. Identify medial vowel sound. 3. Solve a 2 step word problem involving addition and subtraction of whole numbers. 4. Construct plane figures using a ruler and compass. What is the meaning of a particular statement, diagram, or a formula? 6. What scientific principle is applicable in a given situation? 7. What extrapolation from the given quantitative data is reasonable. 8. What effect would specific change in a part of the system have on another part of the system or on the system as a whole. 9. How can the graph be expressed in another form? 5. Recite the Act of Contrition. 11. Indentify initial consonant blends/clusters. 12. Pronounce words with “ue” sound. 13. Identify soft and loud sound. 14. What applications illustrate a scientific principle that is not stated though it may be named. 10. Ex. 2 Direction: Identify the following statements, questions, phrase according to the level of skills. Write K, for Knowledge, C for Comprehension, AP for Application, An for Analysis, Syn for Synthesis and Ev for Evaluation on the space provided before the number. 1. Students will know the dates and countries of origin of the major immigrant groups to the US. Students will explain the motives of Filipino migrants in going to the U.S. 3. Students will design an immigration policy that is fair to future immigrant groups. 4. Students will decide whether the current U.S. immigration policy is fair to various immigrant groups, tell why? 5. Which of several suggested procedures is not acceptable for achieving a stated goal but also meets and added criterion such as faster, cheapest, most accurate. 2. Nakikilala ang mga matatalinghagang salita at idyoma. 7. Nakasusulat ng liham paanyaya o pagtanggap ng paanyaya. 8. If gasoline consumption needs to be curtailed, do you favor direct or indirect means of doing so? In your answer, include at least three historical precedents. 9. Design a survey form to measure student’s attitude toward the English rule. 10. From the lessons learned volcanoes, design a volcano model. 6. What is an effect will terrorism have in the budget of the Philippines government. 12. Why did the budget for education increase in the last three years? In answering the question relate it to the performance of the students in the NAT. 13. 15. Locate Davao City in the map. 11. BASIC TEST ITEM FORMATS Multiple Choice Matching True- False Completion/Short answer Essay/Demonstration ANATOMY OF A MULTIPLE – CHOICE ITEM Basic Parts Stem – problem Either in a form of question or an incomplete statement Alternatives – list of suggested solutions Contains: Best alternative ( answer) Distractors ( Inferior alternatives) - Purpose of distracters is to appear as plausible to the problem for those students who have not achieved the objective being measured by the test item. - The distracters must appear as implausible solutions for those students who have achieved the objective. Example: Which of the following is best measured in cubic Stem meter? Distractor ----- A. ice cream Distractor ----- B. rice Alternatives Answer ----- C. sand Distractor ----- D. sugar GUIDELINES IN CONSTRUCTING MULTIPLE ITEM TEST 1. Avoid repeating words. Poor: Entomology is A. the study of birds B. the study of fish c. the study of insects Better: Entomology is the study of A. birds B. fish C. insects ELIMINATE UNNECESSARY WORDS. Poor: There were many different theories about the origin of mankind. The man associated with the theory of evolution was A. Darwin B. Galileo C. Freud Better: The man associated with theory of evolution was A. Darwin B. Galileo C. Freud 3. State the problem in the stem Poor: Abraham Lincoln A. chopped down the cherry tree. B. signed the Declaration of Independence C. wrote the Emancipation Proclamation 3. State the problem in the stem Better: The president who wrote the Emancipation Proclamation was A. George Washington B. Thomas Jefferson C. Abraham Lincoln 4. AVOID GIVING CLUES IN THE QUESTION (GRAMMATICAL) Poor: A long, winding, gravel crest of glacial origin is a A. kame. B. ridge C. esker 4. AVOID GIVING CLUES IN THE QUESTION (GRAMMATICAL) Better : A long, winding, gravel crest of glacial origin is a/an A. kame B. ridge C. esker 5. USE PLAUSIBLE OPTIONS. Poor: Sino ang mayor ng Kidapawan? A. Fidel B. Ramos B. Jesse Robredo C. Rodolfo Gantuangco D. Rodrigo Duterte. 5. USE PLAUSIBLE OPTIONS. Better: Sino ang mayor ng Kidapawan? A. Philbert Malaluan B. Joseph Evangelista C. Rodolfo Gantuangco D. Jun Cabiles. AVOID THE USE OF ALL OF THE ABOVE Poor: Which of the following factors must be considered in computing basal energy requirements? A. Age B. Height C. Weight D. All of the above AVOID THE USE OF ALL OF THE ABOVE Better: Which of the following factors must be considered in computing basal energy requirements? A. Weight only B. Age only C. Height and Weight only D. Age, height and weight AVOID AMBIGUITY IN THE QUESTION. INCLUDE ONLY ONE CORRECT ANSWER. Poor: A color – blind boy inherits the trait from a A. male parent B. Female parent C. Maternal grandparent D. Paternal grandparent AVOID AMBIGUITY IN THE QUESTION. INCLUDE ONLY ONE CORRECT ANSWER. Better: A color-blind boy most probably inherited the trait from his A. father B. mother B. paternal grandfather D. paternal grandmother WRITE OPTIONS THAT HAVE PARALLEL GRAMMATICAL CONSTRUCTION. Poor: An electric transformer can be used A. for storing up electricity B. to increase the voltage of alternating current C. it converts electrical energy into mechanical D. alternating current is changed to direct current WRITE OPTIONS THAT HAVE PARALLEL GRAMMATICAL CONSTRUCTION. Better: An electric transformer can be used to A. store up electricity B. increase the voltage of alternating current C. convert electrical energy into mechanical energy D. change alternating current to direct current AVOID NEGATIVE IN THE STEM. Poor: Which one of the following is not safe driving practice on icy roads? A. Accelerating slowly B. Jamming on the brakes C. Holding the wheel firmly D. Slowing down gradually AVOID NEGATIVE IN THE STEM. Better: All of the following are safe driving practices on icy roads EXCEPT A. accelerating slowly B. jamming on the brakes C. holding the wheel firmly D. slowing down gradually ARRANGE RESPONSE CHOICES IN LOGICAL ORDER- ALPHABETICALLY AND NUMERICALLY Poor: The principal keyboard instrument in 16th century Europe was the A. Organ B. Harpsichord C. Clavichord D. Pianoforte ARRANGE RESPONSE CHOICES IN LOGICAL ORDERALPHABETICALLY AND NUMERICALLY Better: The principal keyboard instrument in 16th century Europe was the A. Clavichord B. Harpsichord C. Organ D. Pianoforte GUIDELINES IN CONSTRUCTING MULTIPLE ITEM TEST Make alternatives approximately equal in length. Undesirable: The most general cause of low individual incomes in the United States is *a.lack of valuable productive services to sell. b.unwillingness to work. c.automation. d.inflation. GUIDELINES IN CONSTRUCTING MULTIPLE ITEM TEST Make alternatives approximately equal in length. Desirable: What is the most general cause of low individual incomes in the United States? *a.A lack of valuable productive services to sell. b.The population's overall unwillingness to work. c.The nation's increased reliance on automation. d.an increasing national level of inflation. ADVANTAGES IN USING MULTIPLE -CHOICE Multiple-choice items can provide 1. versatility in measuring all levels of cognitive ability. 2. highly reliable test scores. 3. scoring efficiency and accuracy. 4. objective measurement of student achievement or ability. 5. a wide sampling of content or objectives. 6. a reduced guessing factor when compared to truefalse items. DISADVANTAGES Multiple-choice items... 1. are difficult and time consuming to construct. 2. lead an instructor to favor simple recall of facts. 3. Encourage guessing. MATCHING ITEMS Matching item consists of two columns: one column of stems or problems to be answered, and another column of responses from which the answers are to be chosen. Traditionally, the column of stems is placed on the left and the column of responses is placed on the right. Directions: Match the data gathering procedures in the item column on the left with the name of the data gathered in the response column on the right. Place your answer in the blank to the left of each procedure. Each answer may be used only once. Data Gathering Procedure 1. Administer two forms of a test 2. Estimate reliability on the basis of item data 3. Obtain Type of Data a. Coefficient of equivalence odd-item and even-item scores 4. Test then retest in one month b. Coefficient of stability c. Internal consistency d. Rational equivalence MATCHING ITEMS Include Homogeneous Material in Each Exercise 1. Measures factual knowledge a. Matching item 2. One stem and several options b. Multiple choice item 3. Spread of scores around the mean c. Reliability 4. Susceptible to specific determiners d. Standard deviation 5. Test-retest score correlation e. True-false item 6. Test score-criterion correlation f. Validity MATCHING ITEMS Include at Least Three to Five but no More than Eight to Ten Items in a Matching Set 3. Eliminate Irrelevant Clues 4. Place Each Set of Matching Items on a Single Page 5. Reduce the Influence of Clues and thereby Increase Matching Item Difficulty 6. Compose the Response List of Single Words or Very short Phrases 7. Arrange the Responses in Systematic Order: Alphabetical, Chronological, etc. 2. ADVANTAGES OF USING MATCHING ITEMS 1. require short periods of reading and response time, allowing you to cover more content. 2. provide objective measurement of student achievement or ability. 3. provide highly reliable test scores. 4. provide scoring efficiency and accuracy. LIMITATIONS IN USING MATCHING ITEMS 1. have difficulty measuring learning objectives requiring more than simple recall of information. 2. are difficult to construct due to the problem of selecting a common set of stimuli and responses. TRUE OR FALSE SUGGESTIONS FOR WRITING TRUE-FALSE TEST ITEMS 1.Base true-false items upon statements that are absolutely true or false, without qualifications or exceptions. Undesirable: Nearsightedness is hereditary in origin. TRUE OR FALSE SUGGESTIONS FOR WRITING TRUE-FALSE TEST ITEMS 1.Base true-false items upon statements that are absolutely true or false, without qualifications or exceptions. Desirable: Geneticists and eye specialists believe that the predisposition to nearsightedness is hereditary. TRUE OR FALSE Express the item statement as simply and as clearly as possible. Undesirable: When you see a highway with a marker that reads, "Interstate 80" you know that the construction and upkeep of that road is built and maintained by the state and federal government. TRUE OR FALSE Express the item statement as simply and as clearly as possible. Desirable: The construction and maintenance of interstate highways is provided by both state and federal governments. TRUE OR FALSE 3.Express a single idea in each test item. Undesirable: Water will boil at a higher temperature if the atmospheric pressure on its surface is increased and more heat is applied to the container. TRUE OR FALSE 3.Express a single idea in each test item. Desirable: Water will boil at a higher temperature if the atmospheric pressure on its surface is increased. and/or Water will boil at a higher temperature if more heat is applied to the container. TRUE OR FALSE 4.Include enough background information and qualifications so that the ability to respond correctly to the item does not depend on some special, uncommon knowledge. Undesirable: The second principle of education is that the individual gathers knowledge. TRUE OR FALSE 4.Include enough background information and qualifications so that the ability to respond correctly to the item does not depend on some special, uncommon knowledge. Desirable: According to John Dewey, the second principle of education is that the individual gathers knowledge. TRUE OR FALSE 5.Avoid lifting statements from the text, lecture or other materials so that memory alone will not permit a correct answer. Undesirable: For every action there is an opposite and equal reaction. TRUE OR FALSE 5.Avoid lifting statements from the text, lecture or other materials so that memory alone will not permit a correct answer. Desirable: If you were to stand in a canoe and throw a life jacket forward to another canoe, chances are your canoe would jerk backward. TRUE OR FALSE 6.Avoid using negatively stated item statements. Undesirable: The Supreme Court is not composed of nine justices. TRUE OR FALSE 6.Avoid using negatively stated item statements. Desirable: The Supreme is composed of nine justices. TRUE OR FALSE 7. Avoid the use of unfamiliar vocabulary. Undesirable: According to some politicians, the raison d'etre for capital punishment is retribution. TRUE OR FALSE 7. Avoid the use of unfamiliar vocabulary. Desirable: According to some politicians, justification for the existence of capital punishment is retribution. TRUE OR FALSE 8. Avoid the use of specific determiners which would permit a test-wise but unprepared examinee to respond correctly. Specific determiners refer to sweeping terms like "all," "always," "none," "never," "impossible," "inevitable," etc. Statements including such terms are likely to be false. On the other hand, statements using qualifying determiners such as "usually," "sometimes," "often," etc., are likely to be true. When statements do require the use of specific determiners, make sure they appear in both true and false items. TRUE OR FALSE Undesirable: All sessions of Congress are called by the President. (F) The Supreme Court is frequently required to rule on the constitutionality of a law. (T) An objective test is generally easier to score than an essay test. (T) TRUE OR FALSE Desirable: (When specific determiners are used reverse the expected outcomes.) The sum of the angles of a triangle is always 1800. (T) Each molecule of a given compound is chemically the same as every other molecule of that compound. (T) The galvanometer is the instrument usually used for the metering of electrical energy used in a home. (F) TRUE OR FALSE 9.False items tend to discriminate more highly than true items. Therefore, use more false items than true items (but no more than 15% additional false items). ADVANTAGES IN USING TRUE-FALSE ITEMS True-False items can provide... 1. the widest sampling of content or objectives per unit of testing time. 2. scoring efficiency and accuracy. 3. versatility in measuring all levels of cognitive ability. 4. highly reliable test scores. 5. an objective measurement of student achievement or ability. LIMITATIONS IN USING TRUE-FALSE ITEMS True-false items... 1. incorporate an extremely high guessing factor. For simple true-false items, each student has a 50/50 chance of correctly answering the item without any knowledge of the item's content. 2. can often lead an instructor to write ambiguous statements due to the difficulty of writing statements which are unequivocally true or false. 3. do not discriminate between students of varying ability as well as other item types. 4. can often include more irrelevant clues than do other item types. 5. can often lead an instructor to favor testing of trivial knowledge. COMPLETION ITEM SUGGESTIONS FOR WRITING COMPLETION TEST ITEMS 1. Omit only significant words from the statement. Undesirable: Every atom has a central (core) called a nucleus. COMPLETION ITEM SUGGESTIONS FOR WRITING COMPLETION TEST ITEMS 1. Omit only significant words from the statement. Desirable: Every atom has a central core called a(n) (nucleus). COMPLETION ITEM 2.Do not omit so many words from the statement that the intended meaning is lost. Undesirable: The ___________were to Egypt as the____________were to Persia and as __________were to the early tribes of Israel. Desirable: The Pharaohs were to Egypt as the__________were to Persia and as ____________were to the early tribes of Israel. COMPLETION ITEM 3.Avoid grammatical or other clues to the correct response. Undesirable: Most of the United States' libraries are organized according to the (Dewey) decimal system. COMPLETION ITEM 3.Avoid grammatical or other clues to the correct response. Desirable: Which organizational system is used by most of the United States' libraries? (Dewey decimal) COMPLETION ITEM 4.Be sure there is only one correct response. Undesirable: Trees which shed their leaves annually are (seed-bearing, common). COMPLETION ITEM 4.Be sure there is only one correct response. Desirable: Trees which shed their leaves annually are called (deciduous). COMPLETION ITEM 5. When possible, delete words at the end of the statement after the student has been presented a clearly defined problem. Undesirable: (122.5) is the molecular weight of KClO3. Desirable: The molecular weight of KClO3 is (122.5). COMPLETION ITEM 7. Avoid lifting statements directly from the text, lecture or other sources. 8. Limit the required response to a single word or phrase. ESTIMATING THE DIFFICULTY OF THE TEST Fig. 1. The normal distribution of test scores, or “ bell-shaped” curve. In constructing test, there should be 15% difficult questions, 70% average questions and 15% easy questions. TABLE OF SPECIFICATION Objective Skills Knowled Applicati Synthesis ge on Evaluatio Compreh Analysis n ension Identify the parts Of the digestive System Explain the process of digestion Predict what happens to coral feefs are destroyed Identify the parts of the respiratory system 1 1 Item Place ment Per cen t I 1,2.05 I3.05 I 34 5 II 1-5 12.5 ADVANTAGES IN USING COMPLETION ITEMS Completion items 1. can provide a wide sampling of content. 2. can efficiently measure lower levels of cognitive ability. 3. can minimize guessing as compared to multiple-choice or true-false items. 4. can usually provide an objective measure of student achievement or ability. LIMITATIONS IN USING COMPLETION ITEMS Completion items 1. are difficult to construct so that the desired response is clearly indicated. 2. have difficulty measuring learning objectives requiring more than simple recall of information. 3. are more time consuming to score when compared to multiple-choice or true-false items. 4. are more difficult to score since more than one answer may have to be considered correct if the item was not properly prepared. ESSAY TEST ITEMS The essay test is probably the most popular of all types of teacher-made tests. In general, a classroom essay test consists of a small number of questions to which the student is expected to demonstrate his/her ability to (a) recall factual knowledge, (b) organize this knowledge and (c) present the knowledge in a logical, integrated answer to the question. ESSAY TEST ITEMS Sample Extended-Response Essay Item Explain the difference between the S-R (StimulusResponse) and the S-O-R (Stimulus-OrganismResponse) theories of personality. Include in your answer (a) brief descriptions of both theories, (b) supporters of both theories and (c) research methods used to study each of the two theories. (10 pts. 20 minutes) ESSAY TEST ITEMS Sample Short-Answer Essay Item Identify research methods used to study the S-R (Stimulus-Response) and S-O-R (StimulusOrganism-Response) theories of personality. (5 pts. 10 minutes) ESSAY TEST ITEMS 1.Prepare essay items that elicit the type of behavior you want to measure. Learning Objective: The student will be able to explain how the normal curve serves as a statistical model. Undesirable: Describe a normal curve in terms of: symmetry, modality, kurtosis and skewness. Desirable: Briefly explain how the normal curve serves as a statistical model for estimation and hypothesis ESSAY TEST ITEMS 2.Phrase each item so that the student's task is clearly indicated. Undesirable: Discuss the economic factors which led to the stock market crash of 1929. ESSAY TEST ITEMS 2.Phrase each item so that the student's task is clearly indicated. Desirable: Identify the three major economic conditions which led to the stock market crash of 1929. Discuss briefly each condition in correct chronological sequence and in one paragraph indicate how the three factors were inter-related. ESSAY TEST ITEMS 3.Indicate for each item a point value or weight and an estimated time limit for answering. Undesirable: Compare the writings of Bret Harte and Mark Twain in terms of settings, depth of characterization, and dialogue styles of their main characters. ESSAY TEST ITEMS 3.Indicate for each item a point value or weight and an estimated time limit for answering. Desirable: Compare the writings of Bret Harte and Mark Twain in terms of settings, depth of characterization, and dialogue styles of their main characters. (10 points 20 minutes) ESSAY TEST ITEMS 4.Ask questions that will elicit responses on which experts could agree that one answer is better than another. 5.Avoid giving the student a choice among optional items as this greatly reduces the reliability of the test. 6.It is generally recommended for classroom examinations to administer several shortanswer items rather than only one or two extended-response items. ADVANTAGES IN USING ESSAY ITEMS Essay items 1. are easier and less time consuming to construct than are most other item types. 2. provide a means for testing student's ability to compose an answer and present it in a logical manner. 3. can efficiently measure higher order cognitive objectives (e.g., analysis, synthesis, evaluation). LIMITATIONS IN USING ESSAY ITEMS Essay items 1. cannot measure a large amount of content or objectives. 2. generally provide low test and test scorer reliability. 3. require an extensive amount of instructor's time to read and grade. 4. generally do not provide an objective measure of student achievement or ability (subject to bias on the part of the grader). SUGGESTIONS FOR SCORING ESSAY ITEMS 1. Choose a scoring model. Two of the more common scoring models are ANALYTICAL SCORING and GLOBAL QUALITY. ANALYTICAL SCORING: Each answer is compared to an ideal answer and points are assigned for the inclusion of necessary elements. Grades are based on the number of accumulated points either absolutely (i.e., A=10 or more points, B=6-9 pts., etc.) or relatively (A=top 15% scores, B=next 30% of scores, etc.) ANALYTICAL SCORING: Examples Essay Item and Grading Models "Americans are a mixed-up people with no sense of ethical values. Everyone knows that baseball is far less necessary than food and steel, yet they pay ball players a lot more than farmers and steelworkers." WHY? Use 3-4 sentences to indicate how an economist would explain the above situation. ANALYTICAL SCORING Necessary Elements to be Included Points in Response Salaries are based on demand 3 relative to supply of such services. Excellent ball players are rare. 2 Ball clubs have a high demand 2 for excellent players. Clarity of Response 2 SUGGESTIONS FOR SCORING ESSAY ITEMS GLOBAL QUALITY: Each answer is read and assigned a score (e.g., grade, total points) based either on the total quality of the response or on the total quality of the response relative to other student answers. SUGGESTIONS FOR SCORING ESSAY ITEMS. Try not to allow factors which are irrelevant to the learning outcomes being measured affect your grading (i.e., handwriting, spelling, neatness). 3.Read and grade all class answers to one item before going on to the next item. 2 4.Read and grade the answers without looking at the students' names to avoid possible preferential treatment. SUGGESTIONS FOR SCORING ESSAY ITEMS 5.Occasionally shuffle papers during the reading of answers to help avoid any systematic order effects (i.e., Sally's "B" work always followed Jim's "A: work thus it looked more like "C" work). 6.When possible, ask another instructor to read and grade your students' responses.