Final Exam 2025 Strategies for Measurement Tools

Topic 1. Strategies for Designing Measurement Tools and Procedures 2. Measurement Reliability 3. Validity 4. Guidelines in Scale Development 5. Standardized Approaches to Measurement 6. Questionnaire ...

Topic 1. Strategies for Designing Measurement Tools and Procedures 2. Measurement Reliability 3. Validity 4. Guidelines in Scale Development 5. Standardized Approaches to Measurement 6. Questionnaire  4 Strategies for Designing Measurement Tools and Procedures Rosemarie DIMauro Satyshur As noted in Chapter 1, the two major frameworks for measurement are the norm-­ referenced and criterion-referenced approaches. This chapter focuses on the design and interpretation of each of these types of measures. Norm-referenced measures are employed when the interest is in evaluating a subject’s performance relative to the performance of other subjects in some well-defined comparison group. For ­example, Benkert, Hollie, Nordstrom, and Wickson (2009) employed norm-­referenced ­measures in their study of trust, mistrust, racial identity, and patient satisfaction in urban African American primary care patients of nurse practitioners; and Rew, Grady, Whittaker, and Bowman (2008) employed a norm-referenced measure to study the effects of duration of homelessness and gender on personal and social resources, cognitive-perceptual factors, and sexual health behaviors among ­homeless youths. Balashov et al. (2008) employed norm-referenced equations that predict a 6-­minute- walk distance according to age, height, weight, and gender patterns for healthy patients to explore whether or not these equations apply to patients with c­ ongestive heart ­failure. Spaulding (2012) evaluated the consistency in severity classifications for children with language impairment on tests of child language. How well a subject’s performance compares with the performance of other subjects is irrelevant when a criterion-referenced approach is used. Criterion-referenced measures are employed when the interest is in determining a subject’s performance relative to a predeter- mined set of target behaviors. For example, Davison, Davenport, Chang, Vue, and Su (2015) applied a criterion-related profile analysis to assess whether subscores of a test or a test battery account for more variance than does any single variable and illustrated this application using Scholastic Aptitude Test data. Aiken and Poghosyan (2009) employed a criterion measure in their study to determine the extent to which direct care professional nurses working in inpatient units in four hospitals in Russia and Armenia agreed that certain characteristics that typify Magnet hospitals were present in their work setting. Valentine and Cooper (2008) developed an instrument, the Study Design and Implementation Assessment Device (DIAD), to assess the quality of intervention effectiveness research study designs and implementation on multiple 113 114   II Understanding Measurement Design dimensions that results in a study quality profile. For this reason, d ­ ifferent strategies are used when designing norm-referenced and criterion-referenced measures. DESIGNING NORM-REFERENCED MEASURES Essential steps in the design of a norm-referenced measure are (a) selection of a ­conceptual model for delineating the nursing or health care aspects of the measure- ment process; (b) explication of objectives for the measure; (c) development of a ­blueprint; and (d) construction of the measure, including administration procedures, an item set, and scoring rules and procedures. Because selection of a conceptual model is addressed in Chapters 1 and 2, the focus here is on steps b to d. EXPLICATING OBJECTIVES The first step in the design of any measure is to clarify the purposes for the ­measurement. When a conceptual model serves as a basis for the tool’s development, this step is more easily undertaken than when it does not. For example, suppose an investigator is interested in assessing a geriatric patient’s ability to perform activities of daily living (ADL) upon admission to an assisted living facility. Using self-care t­ heory, ADL are conceptually defined as the patient’s capacity to perform various physical (body care) tasks that permit the individual to provide self-care on a daily basis. Further, it is assumed that (a) body care tasks essential in everyday life are related to eating, dressing, bathing, toileting, transfer, walking, and communication; (b) the concept of self-care is not an absolute state but a continuum of ability levels that vary in the frequency with which the help of others is needed; and (c) the geriatric patient’s level of ability in performing certain ADL may differ from the same individual’s level of ability in performing other activities. On the basis of this conceptual definition of ADL self-care, the ­investigator is directed to operationalize ADL self-care in the following manner: (a) use a performance-­objective type of measure, most appropriately, observation; (b) include in the measure multiple items reflecting salient characteristics or conditions related to each of the identified body care activities essential in everyday life; and (c) provide a way for respondents to demon- strate various levels of ability to provide self-care with and without the help of others. Hence, the objective for the measure is, given a series of ADL tasks, the geriatric patient newly admitted to an assisted living facility will demonstrate the frequency with which he or she can perform the following tasks: eating, dressing, bathing, ­toileting, transfer, walking, and communicating, alone or with the help of others. It should be apparent that this objective derives from, and is consistent with, the con- ceptual definition of ADL self-care; it defines the relevant domain of content to be assessed by the measure as the geriatric patient’s performance of specific and varied ADL activities; and it specifies the type of behavior the subject will exhibit to demon- strate that the purpose of the measure has been met, that is, the frequency with which he or she is able to perform the behavior with and without the help of others. To meet this objective, the investigator looks to the conceptual framework as well as empirical findings from studies defining ADL self-care in a similar manner  4 Strategies for Designing Measurement Tools and Procedures   115 to identify and list a number of behaviors salient to the measurement of the geriat- ric patient’s ability to eat, dress, bathe, toilet, transfer, walk, and communicate. This preliminary list is then subjected to scrutiny by experts in ADL for geriatric patients living in assisted living facilities who may add and/or delete behaviors. Each of these behaviors then becomes an item on the measure. Hence, each item included in the measure should be linked to the conceptual definition; that is, items that do not relate directly to the objective for the measure are superfluous and, if included, will tend to introduce error and thus, decrease validity. To assess the level of ADL self-care, the investigator employs a 5-point ­rating scale of the frequency with which the geriatric patient requires help from others ­ranging from never (0) to always (4). A portion of the resulting measure of ADL self- care is illustrated in Figure 4.1. It should be apparent from this hypothetical example that objectives provide the link among theories, concepts, and their measurement. Additional examples of this linkage can be found in Strickland and Dilorio (2003a, 2003b) and Waltz and Jenkins (2001). It is not only important to explicate objectives, but also paramount that they are stated correctly. A poorly stated objective can be more troublesome than no stated objective. For this reason, attention now turns to approaches for writing objectives that have gained favor with use. Behavioral objectives are usually stated by using one of two approaches. The first approach is best characterized by the work of Mager (1962). In this view, an objective has essentially four components: (a) a description of the respondent, (b) a descrip- tion of the behavior the respondent will exhibit to demonstrate the accomplishment Directions: Rate the frequency with which the geriatric patient required help from others in performing ADL during the first week of being admitted to an assisted living facility using the following scale: 0 1 2 3 4 Never Almost Occasionally Almost Always Never Always Place your rating in the blank space next to the item. Rating ADL Activity* __________ Pour liquids __________ Hold cup or glass __________ Drink from cup or glass __________ Open food containers __________ Hold fork and spoon __________ Cut food __________ Raise food to mouth __________ Place food in mouth, chew, and swallow *ADL included here are limited and intended only to exemplify a few of the many activities relevant to a geriatric patient in an assisted living facility. FIGURE 4.1 A hypothetical sample measure of geriatric patients’ performance of activities of daily living (ADLs) in an assisted living facility. 116   II Understanding Measurement Design of the objective, (c) a description of the conditions under which the respondent will ­demonstrate accomplishment, and (d) a statement of the standard of performance expected to indicate accomplishment. This format for writing objectives is particularly useful when constructing measures of cognition, especially in a criterion-referenced framework, because it forces one to explicate clearly the standard of performance expected prior to the construction of items. It should be noted, however, that its use is not limited to cognitive, criterion-referenced measures and that it has utility as well when constructing measures of affect and performance in a norm-referenced frame- work. Table 4.1 illustrates Mager’s approach to the formulation of an objective for measuring the performance of a pediatric nurse practitioner student in a pediatric ambulatory care center. The second approach reflects the views of scholars like Tyler (1950) and Kibler, Barker, and Miles (1970). Although similar to Mager’s approach, a behavioral objec- tive in this case is composed of only three components: (a) a description of the respondent, (b) delineation of the kind of behavior the respondent will exhibit to demonstrate accomplishment of the objective, and (c) a statement of the kind of con- tent to which behavior relates. This approach to objective explication is quite useful within a norm-referenced measurement context because it results in an outline of content and a list of behaviors that can then be readily used in blueprinting, which is discussed in the next section of this chapter. Table 4.2 illustrates the same objective written according to the Tyler–Kibler scheme. An example of the use of Tyler’s (1950) approach to behavioral objectives can be found in Howard (2001). TABLE 4.1 Formulation of an Objective to Measure Performance of a Pediatric Nurse Practitioner (PNP) Student Using Mager’s Approach Objective: Given a well child newly admitted to the pediatric ambulatory care center, the PNP student will perform a comprehensive physical assessment as outlined in the center’s standards-of-care procedure manual. Component Example 1. Description of respondent PNP student 2. Description of behavior to be exhibited if Perform a comprehensive physical objective is accomplished assessment 3. Description of conditions under A newly admitted well child which respondent will demonstrate accomplishment 4. Statement of standard of performance Perform assessment according to pediatric ambulatory care center standards of care outlined in the procedure manual  4 Strategies for Designing Measurement Tools and Procedures   117 TABLE 4.2 Formulation of an Objective to Measure Pediatric Nurse Practitioner (PNP) Student Performance Using the Tyler–Kibler Scheme Objective: The PNP student performs a comprehensive physical assessment on a newly admitted well child. Component Example 1. Description of respondent PNP student practicing in a pediatric ambu­ latory care center 2. Description of behavior to be exhibited if Perform a comprehensive physical objective is accomplished assessment 3. Statement of the kind of content to which An outline of standards of care contained behavior relates in the center’s procedure manual was ­presented to the student in class prior to the PNP’s clinical rotation A taxonomy is a useful mechanism for defining the critical behavior to be assessed by an objective in such a manner that all who use the same taxonomy or classifying scheme will assess the same behavior in the same way, thus increasing the reliability and validity of measurement. Numerous taxonomies have been proposed for the cog- nitive, affective, and psychomotor domains. Attention here focuses briefly on those that have gained favor through empirical use: (a) Bloom’s (1956) taxonomy of the cog- nitive domain; (b) Krathwohl, Bloom, and Masia’s (1964) taxonomy for the affective domain; and (c) Fitts’s (1962, 1964) scheme for the psychomotor domain. Table 4.3 presents a simplified version of the taxonomy of the cognitive domain. In Bloom’s framework, the mental operations are grouped into a small number of simple-to-complex, hierarchically ordered categories: knowledge, comprehension, ­ application, analysis, synthesis, and evaluation. Hence, each subsequent level of mental activity involves the mental operations required at the preceding levels. For example, to be able to analyze, the respondent must first be able to know, comprehend, and apply. A simplified version of the taxonomy for the affective domain appears in Table 4.4. As with the cognitive taxonomy, levels are hierarchical in nature, and performance at higher levels allows one to assume that the respondent can perform at lower levels as well. Taxonomies for assessing the psychomotor domain are far less developed than for the other two; however, the approach by Fitts, which is summarized in Table 4.5, shows some promise in this area and is included for this reason. Fitts (1962, 1964) identifies three phases of skill development: cognitive, fixation, and autonomous. Phases overlap to some extent; that is, they are not distinct units and movement from one phase to another is a continuous process. As a subject proceeds from early to late phases, the performance of the skill becomes progressively more automatic and more accurate, and demands less concerted effort on the part of the subject, allowing ­attention to be given to other activities concurrently. 118   II Understanding Measurement Design TABLE 4.3 A Simplified Version of the Taxonomy of the Cognitive Domaina Mental Operation Examples Level Action Verbs Objective Measurement Comments Knowledge: Action verbs The newly PLC : Hospital b This is the Measures usually employed standards for lowest level subjects’ include: acute care performing of mental ability to recall cite, classify, nurse will a physical activity. or recognize complete, cite hospital assessment The PLC information in ­correct, standards for are outlined reflects the essentially the ­identify, label, performing in the information to same form as it list, mark, a physical procedure be acquired by was presented. name, read, assessment manual given the subjects The essential recall, recite, on a newly at orientation. that is part of learner behavior recognize, admitted Question: Cite the objective is remembering. show, state, critically ill hospital stated during tell, and write. adult patient. standards for the program’s performing development. a physical assessment on a newly admitted critically ill adult patient. Comprehension: Action verbs The newly PLC: During At this level, the Measures usually employed orientation subject must understanding include: acute care the instructor remember the at the most conclude, nurse will explained information rudimentary convey explain in his how a and use it to level, i.e., the meaning or her own physical solve a novel subject’s ability of decode, words how assessment problem. A key to use previously describe in to perform is performed feature is that acquired own words, a physical on a newly the item or information to explain, assessment admitted the context solve a problem. extrapolate, on a newly critically ill in which it is 1. Translation: give reasons, admitted adult patient. asked is struc­ Ability to illustrate, critically ill Question: The tured in such paraphrase, interpret adult patient. orientation a way that present in reformu- instructor ­subjects are a different late, restate, assigns made aware language, rewrite, subjects to of ­information or recognize summarize, play roles of required to paraphrases tell why, and acute care solve the symbolic translate. nurse and ­problem. Items changes. instructor in are designed a sociodrama to determine if and they are the learner can to act out solve a novel these roles as problem when explained to information to them by the be used is instructor. specified. (continued)  4 Strategies for Designing Measurement Tools and Procedures   119 TABLE 4.3 A Simplified Version of the Taxonomy of the Cognitive Domaina (continued ) Mental Operation Examples Level Action Verbs Objective Measurement Comments 2. Interpretation: Given a case PLC: The newly In Bloom’s Measures history of employed taxonomy the subjects’ a newly acute care there is a third ability to make ­admitted nurse has type of com­ an inference ­critically been taught prehension, based on the ill adult at orientation ­extrapolation, information in a patient and a to perform that is not communication ­description of a physical included to explain what the com­ assessment here as it is is meant by the prehensive on a newly so ­similar communication physical admitted to compre­ or to summa­ assessment critically ill hension and rize the infor­ performed on adult patient. interpretation. mation in the that patient, The instructor communication. the newly then presents employed the learner acute care with a case nurse history and will make description of ­conclusions the physical about the assessment patient’s performed on status on the the patient. basis of the Question: (True data. or False) On the basis of the results of the physical ­assessment, it is ­evident that the patient has altered mobility. Application: Action verbs The newly PLC: During Note: If this Requires subjects include: employed orientation question to use previously administer, acute care the instructor were testing acquired adopt a plan, nurse will taught comprehen­ information apply, carry perform a the newly sion, the in solving a out, compute, physical employed nurse would novel problem. demonstrate, assessment acute care be told to use Neither the employ, on a newly nurse to procedures question nor make use admitted perform a outlined in the the context in of, ­perform, critically ill physical standards. which it is plot, put adult patient. according asked helps in action, to hospital respondents and put standards. decide what to use. (continued) 120   II Understanding Measurement Design TABLE 4.3 A Simplified Version of the Taxonomy of the Cognitive Domaina (continued ) Mental Operation Examples Level Action Verbs Objective Measurement Comments previously Later in the Application acquired emergency requires the information must department, learner to be used to solve a critically identify the the problem. ill patient is previously Questions admitted and acquired are aimed at the instructor knowledge to determining if asks: be used. Thus, subjects are able Question: Can at this level, to select and you perform learners are correctly employ a physical required to appropriate assessment select as well knowledge in on this as use knowl­ solving a new patient? edge correctly problem. to solve a new problem. Analysis: May Action verbs: After watching PLC: The newly require subjects analyze, a video of an employed to (a) identify a arrange experienced acute care logical error in a in order, acute care nurse communication combine, nurse watches a (e.g., a compare, performing video on contradiction, contrast, a physical strategies and error in criticize, assessment techniques deduction, deduce, on a critically for minimiz­ erroneous causal ­designate, ill adult ing errors and inference) or (b) detect, patient, maximizing identify, classify, determine, the newly validity of and/or recognize ­distinguish, employed findings of relationships and acute care a physical among the formulate. nurse will list assessment elements at least five on a ­critically (i.e., facts, behaviors ill adult assumptions, of the nurse patient. hypotheses, that may Question: List conclusions, introduce all of the ideas) in a error into the actions taken communication. findings and by the acute Items at this at least five care nurse level usually behaviors perform­ assume specific likely to ing the training in a result in valid ­assessment logical process conclusions that are likely to be used. regarding to minimize the patient’s errors and status. those likely to maximize validity. (continued)  4 Strategies for Designing Measurement Tools and Procedures   121 TABLE 4.3 A Simplified Version of the Taxonomy of the Cognitive Domaina (continued ) Mental Operation Examples Level Action Verbs Objective Measurement Comments Synthesis: requires Action verbs: The newly PLC: The newly the respondent compose, employed employed to produce construct, acute care acute care or create: design, nurse will nurse (a) a unique develop, design a has read verbal or written devise, procedure extensively communication, fabricate, for the unit regarding the or (b) a plan or form, to reduce strategies and procedure for formulate the kind techniques accomplishing a hypotheses, and amount for maximiz­ particular task. integrate, of errors ing validity propose, resulting of physical reorganize, from the assessments and nurses; poor and partici­ restructure. performance pated in staff in conducting development physical programs assessments. regarding strategies and techniques for minimiz­ ing errors in this regard. Question: How can we reduce the effect of errors in procedure on the validity of the outcomes of physical assessments performed on this particular critical care unit? Evaluation: Action verbs: After observing PLC: The staff Requires the appraise, an experi­ development subjects to judge ascertain enced acute instructor the value of value, assay, care nurse demonstrates ideas, people, assess, perform a two products, meth­ ­diagnose, physical contrasting ods, etc., for a evaluate, assessment approaches specific ­purpose fix value of, on a newly to performing and state valid judge, list in admitted a physical reasons for their order of critically ill assessment. judgment (continued) 122   II Understanding Measurement Design TABLE 4.3 A Simplified Version of the Taxonomy of the Cognitive Domaina (continued ) Mental Operation Examples Level Action Verbs Objective Measurement Comments (i.e., the learners importance, adult patient, Question: must state the and rank the newly Which do criteria upon in order of employed you think is which the importance. acute care likely to result judgment is nurse will in the most based). judge the valid data comprehen­ regarding siveness and patient accuracy of status? State the results your reasons and state for choosing the reasons the approach for the you did. judgment. a Portions of the material in this table are adapted from Staropoli and Waltz (1978). b PLC refers to the prior learning condition. TABLE 4.4 Simplified Version of the Taxonomy of the Affective Domain Objective: The newly employed acute care nurse performs a physical assessment of a newly admitted critically ill adult patient. Examples Affective Levels Action Verbs Objective Measurement Receiving Action verbs include: The nurse will Over time the nurse (Attending): accept, attempt, listen carefully and consistently ­Measures ­subject’s comply, define, respectfully to all demonstrates awareness and/ identify, limit, list, opinions rendered behavior that or willingness to listen, observe, by the family of a indicates she or receive specified recognize, refrain, frail elderly patient he is listening (eye stimuli. Indicates and reject. with dementia. contact, nondirec­ the subject is capa­ tive responses) ble of directing respectfully attention toward to opinions specified materials expressed by the or behavior. family. Responding: Action verbs include: The nurse will will­ The subject seeks ­Mea­sures the ask, ­challenge, ingly comply with out agency ­subject’s tendency choose, cite, agency sugges­ suggestions to respond in a ­consult, delay, tions concerning concerning favorable ­manner doubt, hesitate, courtesy toward courtesy toward to specified inquire, offer, families of frail the family stimuli. elderly patient with and performs dementia. consistently (continued)  4 Strategies for Designing Measurement Tools and Procedures   123 TABLE 4.4 Simplified Version of the Taxonomy of the Affective Domain (continued ) Examples Affective Levels Action Verbs Objective Measurement Response behav­ query, question, within agency ior indicates that read, repeat, expectations. the subject has select, and try. become ade­ quately involved or committed to a specified stimulus. If the subject ­consents, seeks, and/or enjoys working with a specified activity, she or he is responding favorably. Valuing: Measures Action verbs The nurse will The nurse reflect that the include: consider, voluntarily volunteers to subject displays display, examine, participate participate in behaviors with express, insist, in a hospital- the benefit and acceptable join, participate, sponsored benefit supports the work consistency persist, ­practice, to raise funds she or he chooses under appropriate pursue, qualify, for research in to undertake. circumstances to seek, specify, dementia in frail indicate adop­ ­support, test, elderly. tion of a certain undertake, value or ideal. In ­volunteer, and demonstrating the weigh. value behavior, the subject can select from among differing values on specified topics and may demon­ strate a high degree of commit­ ment, conviction, or loyalty to the accepted value. Organization: Action verbs The nurse will The nurse organizes ­Mea­sures reflect include: adapt, organize a com­ the group and that the subject is analyze, munity-based schedules time able to classify a compare, support group each week to be value concept by contrast, criticize, for families of involved. (a) determining, deduce, frail elderly with dementia (continued) 124   II Understanding Measurement Design TABLE 4.4 Simplified Version of the Taxonomy of the Affective Domain (continued ) Examples Affective Levels Action Verbs Objective Measurement analyzing, demonstrate, and schedule comparing its designate, design, time weekly to characteristics, ­determine, participate in its and (b) placing all diagnose, implementation. previously classi­ gather, identify, fied values into a ­investigate, order, harmonious and organize, and orderly relation­ propose. ship, thus building a personal value system. Characterization Action verbs The nurse will The nurse is con­ by a value or include: solve problems sistent in solving value complex: construct, regarding staff problems in Measures indicate design, develop, fear or unwill­ terms of desired the subject is able ­evaluate, ingness to care outcomes rather to respond to the ­formulate, plan, for the patient than rigid princi­ complex world revise, and with dementia re ples or emotions. and environment synthesize. patient conse­ around him or her quences rather in a consistent, than rigid princi­ predictable, and ples or emotions. comprehensible manner. TABLE 4.5 Fitts’s Phases of Complex Skill Development Objective: The newly employed acute care nurse performs a physical assessment of a newly admitted critically ill adult patient. Phase of Development Measurement Comments 1. Cognitive: Measures Observation of nurse’s Phase 1 subjects dwell on the indicate that the subject performance reflects procedure and plans that tends to intellectualize preoccupation with the guide the execution of the the skill and makes exam and attention skill. frequent errors in to proceeding to the performing it. outline in the proce­ dure manual even when deviation in the sequence of events appears appropriate. (continued)  4 Strategies for Designing Measurement Tools and Procedures   125 TABLE 4.5 Fitts’s Phases of Complex Skill Development (continued ) Phase of Development Measurement Comments Interpersonal interac­ tion with the patient is limited and the nurse makes frequent eye contact with the observer each time an action is taken. 2. Fixation: Measures Observation of the nurse’s Phase 2 practice of the skill indicate a tendency to performance reflects is important and errors practice correct behavior less preoccupation decrease with practice. patterns; errors are with skills and the fewer than in Phase observer, fewer errors 1 and decrease with are noted, and a pattern practice. for ­proceeding with the assessment has emerged. 3. Autonomous: Measures Observation of the nurse’s Phase 3 skill becomes indicate increasing performance reflects ­automatic and attention speed of performance, few errors, a pattern of focuses on other aspects of errors occur infrequently, performance that is less the patient. subject resists stress rigid than during stages and interference from 1 and 2, ­interpersonal outside activities, and ­communication is able to perform other between patient and activities concurrently. provider is high, and elements of the health history are ­considered and elicited in ­conjunction with the physical assessment. From the tables it should be apparent that the use of taxonomies in explicat- ing and measuring objectives provides several advantages. A critical aspect of any behavioral objective is the word selected to indicate expected behavior. A behav- ioral term by definition is one that is observable and measurable (i.e., behavior refers to any action on the part of an individual that can be seen, felt, or heard by another person). Cognitive and affective objectives, although they are con- cerned with thinking and feeling, which are not directly observable, are inferred from psychomotor or behavioral acts. In reality, the same behavioral term can be seen, felt, or heard differently by different people. Similarly, because it is impossible to measure every action inherent in a given behavior, different people frequently define the critical behavior to be observed, using a given objective, quite differently. 126   II Understanding Measurement Design When taxonomies are employed, action verbs are used, and critical behaviors to be observed are specified, decreasing the possibility that the behaviors will be inter- preted differently and increasing the probability that the resulting measure will be reliable and valid. A measurement must match the level of respondent performance stated in the behavioral objective; that is, a performance verb at the application level of the cogni- tive taxonomy must be assessed by a cognitive item requiring the same level of perfor- mance. Any discrepancy between the stated objective and the performance required by the instrument or measurement device will result in decreased reliability and valid- ity of the measurement process. For example, if the objective for the measurement is to ascertain the ability of practicing nurses to apply gerontological content in their work with aging clients (application level of Bloom’s) and if the measure constructed to assess the objective simply requires a statement in their own words of some prin- ciples important to the care of the gerontological patient (comprehension level of the taxonomy), the outcomes of the measurement are not valid in that this tool does not measure what is intended. When taxonomies are employed, this type of discrepancy between the level of objective and level of performance measured is less apt to occur than when taxonomies are not employed. An example of the use of taxonomies can be found in Sheetz (2001), who employed Bloom’s (1956) taxonomy of the cognitive domain; Krathwohl, Bloom, and Masia’s (1964) taxonomy of the affective domain; and Harrow’s (1972) taxonomy of the psy- chomotor domain in the development of a rating scale to measure students’ clinical competence. BLUEPRINTING Given a set of objectives reflecting the process or outcomes to be assessed by the mea- sure and a content outline representative of the domain of interest, the next step is to develop a blueprint to establish the specific scope and emphasis of the measure. Table 4.6 illustrates a blueprint for a measure to assess a patient’s compliance with a discharge plan. The four major content areas to be assessed appear as column head- ings across the top of the table and critical behaviors to be measured are listed on the left-hand side of the table as row headings. Each intersection or cell thus represents a particular content-objective pairing, and values in each cell reflect the actual number of each type of item to be included in the measure. Hence, from the table it can be seen that three items will be constructed to assess the content objective pairing patient knowledge of the contents of the discharge plan/general health knowledge. The scope of the measure is defined by the cells, which are reflective of the domain of items to be measured, and the emphasis of the measure and/or relative importance of each content-behavior pairing is ascertained by examining the numbers in the cells. From the blueprint, one can readily tell the topics about which questions will be asked, the types of critical behaviors subjects will be required to demonstrate, and what is relatively important and unimportant to the constructor. Tables 4.7 and 4.8 present additional examples of blueprints that vary slightly in format. In Table 4.7, the blue- print for the knowledge subscale of the measure of compliance with the discharge  4 Strategies for Designing Measurement Tools and Procedures   127 TABLE 4.6 Blueprint for a Measure to Assess Patient’s Compliance With a Discharge Plan Content Objectives General Medications Activities Nutrition Total Health of Daily Knowledge Living Ascertain 3 5 5 5 18 patient knowledge of the con­ tents of the discharge plan Determine 2 2 2 2 8 patient attitudes toward the contents of the discharge plan Evaluate 4 10 10 10 34 patient compliance with con­ tents of the discharge plan Total 9 17 17 17 60 TABLE 4.7  lueprint for the Knowledge Subscale of a Measure of Compliance With the B Discharge Plan Objectives Content Knowledge Comprehension Application Total General health 1 1 1 3 knowledge Medications 2 3 5 (continued) 128   II Understanding Measurement Design TABLE 4.7  lueprint for the Knowledge Subscale of a Measure of Compliance With the B Discharge Plan (continued ) Objectives Content Knowledge Comprehension Application Total Activities of 2 3 5 daily living Nutrition 2 3 5 Total 1 7 10 18 TABLE 4.8 Blueprint for a Measure of Clinical Performance Objectives (Nursing Practices) I II III IV Content (King’s Model) Assessment Planning Implementation Evaluation Total (%) (%) (%) (%) (%) 1. Nurse variables 5 5 5 5 20 a. Perception b. Goals c. Values d. Needs e. Expectations 2. Patient variables 10 10 30 10 60 a. Perception b. Goals c. Values d. Needs e. Expectations f. Abilities 3. Situational variables 10 5 2 3 20 a. Structure b. Goals c. Groups d. Functions e. Physical resources f. Economic resources g. Climate Total 25 20 37 18 100 Source: King (1968).  4 Strategies for Designing Measurement Tools and Procedures   129 plan is defined by topic area and objective, but in this case, rather than listing the critical behaviors to be assessed, the performance expectations are specified using the levels of Bloom’s taxonomy. In Table 4.8, objectives are defined in terms of the steps of the nursing process, content is defined by components of the nursing conceptual model used, and numbers in the cells represent percentages of each type of item to be included rather than the actual number of items. Given the blueprint, the number (or percentage) of items prescribed in each cell would be constructed. Content validity (discussed in more detail in Chapter 6) could then be assessed by presenting content experts with the blueprint and the test and having them judge (a) the adequacy of the measure as reflected in the blueprint, that is, whether or not the domain is adequately represented to ascertain that the most appropriate elements are being assessed; (b) the fairness of the measure, whether it gives unfair advantage to some subjects over others; and (c) the fit of the method to the blueprint from which it was derived. Additional examples of blueprinting can be found in Toth (2003), Jalowiec (2003), and Jones (2003). CONSTRUCTING THE MEASURE The type of measure to be employed is a function of the conceptual model and sub- sequent operational definition of key variables to be measured. If, for example, one conceptualizes job satisfaction as a perceptual phenomenon, the measurement will require use of an affective or typical performance instrument. If, on the other hand, job satisfaction is conceptually defined as a cognitive phenomenon dependent upon one’s understanding and comprehension of factors in the work setting, a maximum performance or cognitive measure is appropriate. The essential characteristics of the types of measures are presented in Chapter 1, and instrumentation and data-­collection methods are discussed in detail in Chapters 7 to 23 and no attention is further given here. Regardless of type, every measure is composed of three components: (a) direc- tions for administration, (b) a set of items, and (c) directions for obtaining and inter- preting scores. ADMINISTRATION Clemans (1971) presents a comprehensive set of considerations to be made in prepar- ing instructions for the administration of a measure. More specific, he advocates the inclusion of the following information: 1. A description of who should administer the measure A statement of eligibility A list of essential characteristics A list of duties 2. Directions for those who administer the measure A statement of the purposes for the measure Amount of time needed for administration A statement reflecting the importance of adhering to directions Specifications for the physical environment 130   II Understanding Measurement Design A description of how material will be received and stored Specifications for maintaining security Provisions for supplementary materials needed Recommendations for response to subjects’ questions Instructions for handling defective materials Procedures to follow when distributing the measure A schedule for administration Directions for collection of completed measures Specifications for the preparation of special reports (e.g., irregularity reports) Instructions for the delivery and/or preparation of completed measures for scoring Directions for the return or disposal of materials 3. Directions for respondents A statement regarding information to be given to subjects prior to the data- collection session (e.g., materials to be brought along and procedures for how, when, and where data will be collected) Instructions regarding completion of the measure, including a request for coop- eration, directions to be followed in completing each item type, and directions for when and how to record answers 4. Directions for users of results Suggestions for use of results Instructions for dissemination of results (p. 196) The importance of providing this information as an essential component of any mea- sure cannot be overemphasized. Errors in administration are an important source of measurement error, and their probability of occurrence is greatly increased when directions for administration are not communicated clearly and explicitly in writing. Those readers who desire further specifics on the topic of administration procedures will find Clemans’s work extremely useful. Readers interested in examples of direc- tions for administration are referred to Marsh (2003), Delorio and Yeager (2003), Weinert (2003), and Emerson (2001). ITEMS Within the context of a given type of measure, there are a variety of specific item formats available, each with its own unique advantages and disadvantages in light of the specific purposes for and characteristics of the setting in which measurement is to occur. Most important, an item should be selected because it elicits the intended outcome, that is, the behavior specified by the objective(s). For example, if the objective of a measurement is to assess clinical performance, the item format should elicit performance by respon- dents in the clinical area. A written multiple-choice test or an attitude survey would not be likely to elicit clinical performance on the part of subjects, and hence would not be an appropriate item format for this measurement objective. Similarly, if a cognitive measure derived from behavioral objectives at the synthesis level of Bloom’s taxonomy  4 Strategies for Designing Measurement Tools and Procedures   131 was composed of a set of true–false items, the behavior specified by the objective and the outcome elicited would be incongruent; that is, at the synthesis level respondents are required to construct or create something new, whereas true–false items only require them to select one of two options on the basis of recall or comprehension. Conditions surrounding the measurement will also establish parameters for what is an appropriate and useful item format. For example, if a measure is to be adminis- tered to diabetics with impaired vision, item formats requiring the reading of lengthy passages would be impractical. Similarly, measures to be administered to patients in acute care settings should be short and easily administered and understood so as to avoid fatigue on the part of respondents and/or to avoid conflict with ongoing care activities. The important point is that the personal characteristics of the respondents, such as ability to read, ability to perform physical skills, computational skills, and communication skills, must be considered, and an item format must be selected that enhances their ability to respond rather than one that hinders some or all of the subjects. Other factors in the measurement setting are also important in selecting the for- mat. If an instrument is to be administered by individuals without training or expe- rience in measurement, or if it is to be employed by a variety of different people, the format should be selected with an eye to easy administration. For example, an instru- ment employing only one item format would be likely to require less time and less complex directions with less probability of being misinterpreted than one employing a number of different item formats. If tables or other illustrations are to be used in the tool, resources should be available for preparing them correctly and to scale; that is, incorrectly or inadequately prepared illustrative materials will increase measurement error, reduce reliability and validity, and decrease the subjects’ motivation to respond. If space is limited or reproduction of the tool is apt to be problematic, items requiring lengthy explanations or repetitions of rating scales or other content on each page of the tool are impractical. If scoring is to be undertaken by a few individuals without the advantage of computers, an item format that is automatic and requires little or no judgment on the part of the scorer is indicated, for example, multiple choice or short answers. When computer scoring is employed, attention should be paid to require- ments imposed by available computer software programs. There are as many different sets of conditions to be considered as there are varied measurement settings. For this reason, only a few of the more frequently overlooked have been included here. In all cases, it is essential that careful analysis of the mea- surement situation be made and an item format should be selected that capitalizes on the factors inherent in a given situation. Examples of such accommodations can be found in Jansen and Keller (2003) and in Jalowiec (2003), who measured attitudinal demands in community dwelling elders. All items can be thought of as on a continuum ranging from objective to ­subjective. Objective items, those allowing little or no latitude in response and hence requiring no judgment in scoring, are often referred to as selection-type items. Examples of ­selection-type formats are true–false, matching, multiple choice, and scaled response. These items are so named because they require the subjects to choose their responses 132   II Understanding Measurement Design from a set of options presented to them. Table 4.9 illustrates some of the more ­common varieties of selection-type items. The multiple-choice format is one of the most common available and is employed widely, especially for cognitive tools. For this reason, the construction of multiple-choice items is discussed in more detail in Chapter 18. Subjective items allow more latitude on the part of respondents in constructing their answer and therefore require more judgment on the part of the scorers. Supply- type items, so named because they require the subject to respond by supplying words, statements, numbers, or symbols to best characterize subjective items. The more fre- quently encountered supply-type item formats are exemplified in Table 4.10. When a norm-referenced measure is employed, one designs items that are likely to make fine distinctions between respondents with differing levels of the attribute TABLE 4.9 Selection-Type Item Formats Format Examples Alternate choice: Consists of a statement to be responded to by selecting from one of two options. Format provides for presentation of a large number of items in relatively short periods of time, therefore allowing broader, more representative sampling from the domain. There is a tendency, however, to select materials out of context, resulting in ambiguity and/or measurement of trivia. 1. True–False Consists of a declarative state that is true (T) or false (F). I usually feel in control of my life. T or F Moderate excesses of digitalis cause premature heartbeats and vomiting. T or F 2. Right–Wrong Consists of a statement, question, equation, or the like that is identified as right (R) or wrong (W) by the respondent. The formula for determining the odds of an event is: Probability of occurrence Odds ratio = Probability of nonoccurrence R or W The turnover among nurses in this hospital is largely the result of their feeling undervalued by the administration. R or W 3. Yes–No Consists of a direct question to be answered by a yes (Y) or no (N). Is it more appropriate to serve shredded wheat than Wheatena for breakfast to a patient on a 250 mg sodium diet? Y or N Is it more desirable for a faculty member in this institution to spend time conducting research than consulting in that person’s specialty area? Y or N (continued)  4 Strategies for Designing Measurement Tools and Procedures   133 TABLE 4.9 Selection-Type Item Formats (continued ) Format Examples 4. Cluster Consists of an incomplete statement with suggested completions, each of which is to be judged true (T) or false (F). Pulse pressure is the: 1. Difference between venous and systolic pressure T or F 2. Difference between arterial and venous pressure T or F 3. Difference between diastolic and systolic pressure T or F 4. Pressure and expansion of the artery as blood flows toward the capillaries T or F 5. All of the above T or F 5. Correction Combines selection and supply by presenting a statement and directing the respondent to correct false statements by substituting appropri­ ate word(s). Lightening usually occurs approximately 4 weeks before delivery. I am satisfied with policies in this college of health sciences. Matching: Consists of a list of words or statements, a list of responses and ­directions for matching responses to words or ­statements. Imperfect and association type are preferred because they allow assessments of higher level behaviors. Because ­homogeneity is paramount in writing such items, the desire for homogeneity when heterogeneity is more ­appropriate may result in a shift in emphasis away from that desired. 1. Perfect Consists of a list of words or statements, a list of responses, matching and ­directions for matching responses to words or state­ ments. Imperfect and association type are preferred because they allow assessments of higher level behaviors. Because ­homogeneity is ­paramount in writing such items, the desire for homogeneity when heterogeneity is more appropriate may result in a shift in emphasis away from that desired. Each response matches one and only one word or statement. On the line before each poison, place the number of the symptoms that usually characterize it. Poisons Symptoms __ Acids 1. Diarrhea, garlic breath __ Rat poisons 2. Restlessness, rapid pulse __ Cocaine 3. Drowsiness, flushed cheeks __ Carbon monoxide 4. Dyspnea, cyanosis (continued) 134   II Understanding Measurement Design TABLE 4.9 Selection-Type Item Formats (continued ) Format Examples 2. Imperfect Some responses do not match any of the words or statements. matching Match each term with its definition. __ Mode 1. Score that separates upper 50% of scores in the __ Mean ­distribution from the lower 50% of the scores __ Median 2. Score obtained by the largest number of respondents 3. Largest number of respondents selecting a given score 4. Sum of scores in a distribution divided by the total number of scores 5. The average 3. Statement Requires respondent to use higher level mental operations such as classification analysis, evaluation. Judge the effects of the nurse’s intervention on the patient’s physical comfort in each of the numbered situations using the following. a. Nursing intervention would tend to reduce the patient’s physical comfort. b. Nursing intervention would tend to increase the patient’s physical comfort. c. Nursing intervention would tend to have little or no effect on the patient’s physical comfort. 1. A primigravida in the first stages of labor is encouraged by the nurse to walk around the corridors.___ 2. As labor progresses, the nurse remains with the primigravida, ­directing her attention to the fact that labor is progressing as expected.___ 3. As labor progresses, the nurse teaches the patient various breath­ ing and relaxing techniques.___ 4. The nurse assesses and records the frequency, duration, and ­intensity of the contractions at regular intervals. ___ 5. The nurse encourages the patient to void at regular intervals. ___ Multiple choice: Components of a multiple-choice item are: (a) stem—which is an introductory statement or question; and (b) responses or suggested answers. Examples of the varied types of multiple-choice items follow. 1. Correct Items that require or permit a correct answer are those that eliminate answer the need for the respondent to make a judgment regarding the ­correctness of his or her response, that is, matters of fact provide a ­suitable basis for such items. The basic service unit in the administration of public health is: a. The federal government b. The state health department c. The local health department d. The public health nurse (continued)  4 Strategies for Designing Measurement Tools and Procedures   135 TABLE 4.9 Selection-Type Item Formats (continued ) Format Examples 2. Best answer For many of the important questions that need to be asked it is impossible to state an absolutely correct answer within the ­reasonable limits of a multiple-choice item. Even if space ­limitations were not a factor, two experts would probably not agree on the precise wording of the best answer. The use of this type of item, which has one best answer, permits the item writer to ask more significant questions and frees the writer from the ­responsibility of stating a correct answer so precisely that all authorities would agree that the particular wording used was the best possible wording. The primary responsibility of the instructor in health services is to: a. Provide an emotional and social environment that adds a wholesome and healthful tone to the child’s school day b. Provide emergency or first aid care when a child becomes ill or injured in school c. Provide up-to-date material about health as part of the curriculum d. Screen for abnormalities and sickness and record findings 3. Based on The responses to this type of question represent generalizations on opinion the basis of literature written by the advocates/experts relative to a preferred approach to the topic. No authoritative sanction for one particular generalization is likely to be available, yet respondents familiar with this literature would probably agree on a best answer to this item. Advocates of the specialized approach to school nursing point out that: a. The health of a child cannot be separated from that of the family and community as a whole b. Specialized nursing care for the child cannot be separated from that for the family and community as a whole c. A specialized program offers greater diversity and challenge to a well-prepared community health nurse d. Specialized nursing in the school allows the nurse to function without the disadvantages of a dual channel of administrative responsibility 4. Novel Requiring the respondent to predict what would happen under certain question circumstances is a good way of measuring understanding of the principle involved. The problem of air pollution is most likely to be reduced in the future by which of the following? a. Urban population will wear air purifying equipment b. Cities will be enclosed to facilitate air purification c. Development of hybrid cars that will not pollute the air d. Use of nonpollutant fuels (continued) 136   II Understanding Measurement Design TABLE 4.9 Selection-Type Item Formats (continued ) Format Examples 5. Selective Given an item that had not been the specific object of instruction, it will recall function to assess learners’ ability to recall a variety of information about the content area, to select that which is relevant, and to base a generalization upon it. The U.S. Public Health Service was reorganized in 1966 to: a. Include the United Nations Children’s Emergency Fund program b. Include the Agency of International Development c. Combine the voluntary agency services with the official ones d. Provide leadership in control of disease and environmental hazards and manpower development 6. Descriptive Inexperienced items writers tend to seek items having very short responses. response This can seriously limit the significance and scope of the achievements measured. In an item measuring the ability to define an important term, it is usually better to place the term to be defined in the item stem and to use definitions or identifications as the responses. The same principle should be applied to other items, that is, one-word responses need not be avoided altogether, but they should seldom be prominent in any measure. The child’s socialization may be defined as: a. A behavioral process in which behavior conforms to the social ­practices of family and extra family groups b. A process of developing an effective set of performances ­characteristic of self-control c. The genetically determined or hereditary mechanisms that determine the individual’s physical trait d. All of the above 7. Combined A difficulty with four-option multiple-choice items is securing four response good alternatives. One solution is to combine questions with two ­alternatives each to give the necessary four alternatives. A school community safety program should be concerned with: a. Teaching children how to avoid accidents b. Teaching adults and children to eliminate physical hazards that endanger them c. Eliminating physical hazards that endanger children and teaching them to avoid accidents d. Teaching children and adults to avoid accidents and eliminating ­physical hazards that endanger them 8. Combined This is a variation of the item type in which essentially two or more question and alternatives each are combined to give four alternatives explanation When was the World Health Organization created and why? a. 1954 to prevent the spread of disease from one continent to another b. 1948 to achieve international cooperation for better health ­throughout the world c. 1945 to structure health privileges of all nations d. 1948 to provide for a liberated population and aid in the relief of suffering (continued)  4 Strategies for Designing Measurement Tools and Procedures   137 TABLE 4.9 Selection-Type Item Formats (continued ) Format Examples 9. Introductory The use of a separate sentence frequently adds to the clarity of the sentence item stem if it is necessary to present background information as well as to ask the question itself. Combining these two ­elements into a single question-sentence probably would make it too complex. When a group is working as a health care team, overlapping of ­activities may occur. What is essential if this is to be prevented? a. Auxiliaries will work within a fairly circumscribed field b. The public health nurse will assume responsibility for all phases of the nursing team’s activities c. The functions of all personnel will be defined d. Maximum skills of each member will be used 10. Necessary If the following question were asked only about environmental health qualification program personnel in general without qualifying urban personnel, it would be difficult for the respondent to give a firm answer to the question, given the existing differences between the ­environmental health programs in other than urban areas, that is, county, state, federal. Generally speaking, the environmental health program personnel in urban centers are: a. Persons with professional training in civil or sanitary engineering b. Sanitary inspectors or non engineering personnel with indoctrination and orientation by the health department c. Public health engineers whose training embraces the public health aspect of both sanitary engineering and sanitary inspection d. All of the above 11. N  one of the Whenever each of the responses can be judged unequivocally above and/ as ­correct or incorrect in response to the question posed in or all of the the item stem, it is appropriate to use none of the above as a above as response. It would be appropriate to use all of the above in a options similar situation in which more than one perfectly correct answer is possible. A necessary requirement for receiving funds under the Comprehensive Health Planning Act of 1966 is that: a. State boards of health must have at least 10% lay representation b. The council responsible for developing a comprehensive health plan for the state must have 50% consumer participation c. Health and welfare councils, whether or not actively involved in health planning on the state level, must have at least 25% lay representation d. None of the above (continued) 138   II Understanding Measurement Design TABLE 4.9 Selection-Type Item Formats (continued ) Format Examples 12. True It is not necessary for the incorrect options to a test item to be ­statements themselves incorrect statements. They simply need to be incorrect as distractors answers to the stem question. Judgments concerning the relevance of knowledge may be as important as judgments concerning its truth. This is particularly useful as a technique for testing achievement that is sometimes thought to be testable only by using essay measures. The general purpose of parent education programs is to: a. Teach parents information they need to know throughout their ­children’s changing developmental stages b. Help parents reinforce their understanding and strengths in regard to themselves and their children c. Develop attitudes of healthy family life as parents of young children d. Cover a wider range of subject matter, format, and method than is possible on an individual basis 13. Stereotypes Phrases, such as operant behavior and homeostasis, which a in distractors respondent may have heard without understanding, provide excellent distractors at an elementary level of discrimination. The particular process of interaction between the organism and its ­environment that results in a specifiable change in both is referred to as: a. Homeostasis b. Human behavior c. Operant behavior d. Developmental dynamism 14. Heterogeneous When responses to an item vary widely because of their wide responses differences, only an introductory knowledge of the content is required for a successful response. The index of economic welfare is: a. Square feet of housing space b. Per capita national income c. Rate of growth of industrialization d. Morbidity and mortality rates 15. Homogeneous Homogeneity of the responses to an item requires a considerably responses high level of knowledge of the content and thus, makes the (harder item) item difficult in comparison to an item using heterogeneous options. Funds for occupational health programs were allocated to state and local health departments as a result of: a. Social Security Act b. Clean Air Act c. Community Health Centers Act d. Occupational Health Act (continued)  4 Strategies for Designing Measurement Tools and Procedures   139 TABLE 4.9 Selection-Type Item Formats (continued ) Format Examples 16. Multiple When the values of two variables fitting the specification in the item clues (easier stem are used, the result is a fairly easy question. That is, the item) respondent need only know one of the values or know one in each of the distractors to respond successfully. The amount estimated to eliminate poverty in the United States is said to be which of the following? a. 2% of the gross national product and 1/5 of the cost of national defense b. 3% of the gross national product and 1/6 of the cost of national defense c. 4% of the gross national product and 1/4 of the cost of national defense d. 5% of the gross national product and 1/3 of the cost of national defense Scaled A statement or question is presented and the respondents answer by response: marking the area on the scale that represents their answer. 1. Number The use of numerical anchors facilitates analysis of data by computer as anchors well as serves to remind respondents of the meaning of the scale steps. Rate your level of satisfaction with your present job using the following scale. 1 2 3 4 5 Not Very satisfied satisfied 2. Percentage Using the scale: anchors A. None of the time B. 1%–25% of the time C. 26%–50% of the time D. 51%–75% of the time E. 76%–99% of the time F. All of the time Rate each of the following actions performed by the nurse you observe administering medications on Unit B. Rating 1. Washes hands prior to administering medications        2. Selects correct medication to be given at designated time    3. Checks patient’s ID prior to administering medication      4. Stays with patient until medication taken            5. Records administration of medication correctly         3. Degrees of Indicate your degree of agreement with the following statement: agreement/ Antagonistic behavior on the part of a patient indicates a need on the disagreement patient’s part for additional attention and time from the nurse. ___/___/___/___/___/___/ 1 2 3 4 5 6 Completely Completely agree disagree (continued) 140   II Understanding Measurement Design TABLE 4.9 Selection-Type Item Formats (continued ) Format Examples 4. Adjectives Using the A–E scale, rate the importance you place on each of he following faculty activities. A. No importance B. Little importance C. Some importance D. Important E. A great deal of importance _____ Conducting clinical research _____ Consulting with nursing staff _____ Teaching students _____ Practicing clinically _____ Pursuing professional development activities 5. Actual Observe the students intervening with the patients on Unit C and then behavior check the behavior that best describes what you observed. _____ Did not adapt planned intervention to meet changes in patient situation. _____ Adapted implementation with guidance to accommodate changes in patient situation. _____ Adapted implementation independently to accommodate changes in patient situation. _____ Implemented independently a preplanned alternate intervention to accommodate changes in patient situation. 6. Products Here are four care plans for confused frail elderly patients living in a long-term care facility. Indicate the one you believe is most ­reflective of accepted indicators of quality care for this patient population. 1. Care plan X 2. Care plan Y 3. Care plan Z 4. Care plan O Context In this case, the item has meaning to the subject only in relation dependent to other material presented, for example, scenario, picture graph, table. Visual presentation of material, especially pictures, permits the presentation of a problem or situation in a very clear and simple manner, thus eliminating confusion due to reading or inability to adequately describe a phenomenon. Parents are given a series of pictures portraying behaviors usually observed in children of 2–3 years of age and asked to identify those observed in their own children. (continued)  4 Strategies for Designing Measurement Tools and Procedures   141 TABLE 4.9 Selection-Type Item Formats (continued ) Format Examples Interpretation Consists of introductory material followed by a series of questions calling for various interpretations, thus provides for measuring ability to interpret and evaluate materials. This type of item allows one to ask meaningful questions on relatively complex topics. Disadvantages stem from the fact that such items are time consuming to administer and difficult to construct. ANOVA Table Source df SS MS F Between 9 72 24 1.5 groups Within groups 84 448 16 Total 93 520 1. How many treatments were studied in the preceding table? a. 2 b. 3 c. 4 d. Cannot answer from the table. 2. How many subjects participated in the study? a. 84 b. 92 c. 93 d. 94 More in-depth discussion of the item types discussed here can be found in Nunnally (1967), Coffman (1971), Wesman (1971), Ebel (1975), Gronlund (1976), Martuza (1977), Staropoli and Waltz (1978), E. C. King (1979), Nunnally and Bernstein (1994), and in Part IV, Instrumentation and Data-Collection Methods. being measured, so that the distribution of responses to the measure will resemble a normal curve with a few high and low scores and with most scores falling in the mid- dle range. Hence, one wants to avoid items that are too easy or too difficult for most respondents. The difficulty of an item is defined as the percentage of respondents a­ nswering that item correctly or appropriately. In other words, if a cognitive test item is answered correctly by 40 of 100 respondents, the item has a difficulty level of 40% or 0.40. Hence, difficulty may vary from 0, in the case in which no one responds ­correctly, to 1.00, in the case in which all respondents respond correctly. When affective or performance measures are employed, the term appropriately or as expected is more meaningful than the term correct; that is, if subjects’ responses are to be compared 142   II Understanding Measurement Design TABLE 4.10 Supply-Type Item Formats Format Example Comments Short answer 1. Question What term is used to For cognitive tests designate that portion of this format tends to the infant’s body that lies measure only facts. nearest the internal os? The respondents are (presentation) presented with a question or incomplete statement 2. Completion Normal labor usually is written in such a way that divided into three stages: it is clear to respondents (1) ______, (2) ______, what is expected of them, (3) ______. (dilating, followed by a blank expulsive, placental) space in which they write what is called for by the directions. Preferably 3. Identification/Association After each event, indicate short-answer questions the stages of labor should require an answer during which it usually that is a single word, occurs.