Test Conceptualization and Development

• Test development occurs in five stages including test conceptualization, test construction, test tryout, item analysis, and test revision.

Test Conceptualization

• Includes indulging in self-talk, reviewing of available literature on existing tests, and looking at emerging social phenomenon or pattern of behavior.
• Emerging social phenomenon or pattern of behavior might serve as the stimulus for the development of a new test.

Preliminary Questions

• What the test is designed to measure, what the objective of the test is and if there is a need for the test.
• Who will use the test, who will take the test, and what content the test will cover.
• How the test will be administered and what is the ideal format.
• What special training is required for test users, what types of responses are required of test takers, and who benefits from an administration of this test.
• Is there any potential for harm as a result of administering this test and how meaning will be attributed to it.

Pilot Work

• Defined as the preliminary research surrounding the creation of a test.
• The test developer runs a trial.
• Test items can be studied to evaluate if they should be included in the final instrument.
• Once pilot work has been completed, the process of test construction begins.

Test Construction: Scaling

• Scaling is the process of setting rules for assigning numbers in measurement.
• L. L. Thurstone is credited for being at the forefront of efforts to develop methodologically sound scaling methods.

Types of Scales

• If age of test taker is important for performance it is an age-based scale, if grade is important, it is a grade-based scale.
• Scales can be unidimensional as opposed to multidimensional.
• Unidimensional scales measure only one aspect trait, while multidimensional scales measure several different aspects.
• Scales can be comparative as opposed to categorical, using comparisons like average, low, or extrovert/introvert.

Scaling Methods

• Rating scales are groupings of words, statements or symbols to make judgments of the strength of a particular trait, attitude, or emotion.
• The final test score on a summative scale is obtained by summing the ratings across all the items.
• The Likert scale is a type of summative scaling where each item presents the test taker with five or seven alternative responses, usually on an agree-disagree continuum.

Paired Comparisons

• Present test takers with pairs of stimuli like photos, objects or statements to compare.
• Test takers must select one of the stimuli according to a rule, such as which statement they agree more with.

Comparative Scaling

• Entails judgments of a stimulus in comparison with every other stimulus on the scale.
• Tasks might include sorting cards from most justifiable to least, or ranking items from 1 to 30.

Categorical Scaling

• Places stimuli into one of two or more alternative categories, that differ quantitatively based on a continuum.
• Test takers could be asked to sort cards into categories like never, sometimes, or always justified.

Guttman Scale

• Items arranged so agreeing with a particular item implies agreement with lower-ranked items.

Item Writing

• Considers the range of content items should cover, type of formats to use, and total number of items to write for each content area.
• An item pool is the reservoir or well from which items will/will not be drawn.
• For a 30-item test, the item pool should have 60 items, with approximately half eliminated.

Item Format

• Item format refers to variables like the item's form, plan, structure, arrangement, and layout.
• Selected-response format items require selecting a response from a set of alternative responses.
• Constructed-response format items require supplying or creating the correct answer, not just selecting it.

Types of Selected-Response Item Formats

• A multiple-choice format includes a stem, a correct alternative or option, and several incorrect alternatives called distractors or foils.
• A matching format presents two columns, premises on the left and responses on the right.
• A binary choice item is a multiple-choice item with only two possible responses.
• A completion item requires providing a word or phrase to complete a sentence, which might include fill in the blanks, or short answer or essay type questions..

Writing Items for Computer Administration

• Two advantages of digital media are storing items in an item bank and individualizing testing through item branching.
• An item bank is a large and easily accessible collection of test questions.
• Computerized adaptive testing (CAT) is interactive wherein items presented are based in part on the test-taker's performance on previous items.
• More difficult items should not be given until the person gets two consecutive items right.
• The test should stop if the person gets five consecutive numbers wrong.

Floor Effect

• This occurs when test takers score very low such as a group of students failing all items.
• Occurs if you administer a test containing items appropriate to the 9th standard average student to students who are younger.
• In computer adaptive testing, a pool of easier items can be given to distinguish between them.

Ceiling Effect

• Occurs when a set of students passes all items, and is addressed by providing more difficult questions if available in the item pool.

Measurement of Personality

• Computerized adaptive testing can be used, starting with items measuring the test-takers responses to issues like depression, then providing in-depth tests based on those responses.

Scoring Items: Cumulative Model

• The rule in a cumulatively scored test is that the higher the score on the test, the higher the test taker is on the ability, trait, or other characteristic the test measures such as high, low or moderate intelligence.

Category Scoring

• Class or category scoring earns credit toward placement in a category with similar responders.
• A diagnostic test that uses category scoring may require a certain number of symptoms to qualify for a specific diagnosis such as depressed or non depressed.

Ipsative Scoring

• Compares a test taker's score on a scale within a test to another scale within that same test.

Test Tryout

• The test developer tries administering the test to people for whom the test is designed.
• For selecting higher corporates, administer the test to them.
• At minimum, administer to 5-10 people to reduce the change factor.
• The time, instructions and atmosphere should be similar to the final test.

Item Analysis

• Includes item difficulty index, item reliability index, item validity index, item-discrimination index, and ICC.

The Item-Difficulty Index

• Obtained by calculating the proportion of test-takers who answered the item correctly.
• Ranges between 0.0 and 1.0, with a higher value indicating a greater proportion of examinees responded correctly, indicating an easier item.
• An item having a difficulty index of .75 is easy, while .30 is difficult.
• A "giveaway" item may be included near the beginning of an achievement test to spur motivation, a positive test-taking attitude, and to lessen test-related anxiety.

The Item-Discrimination Index

• Measures how well an item distinguishes between those who are knowledgeable and those who are not.
• Examines the relationship between performance on the given item and the overall test score, so that examiners who responded correctly also did well on the test.

Item Reliability Index

• Provides an indication of the internal consistency of a test.
• The ideal item measures aspects which the test is designed to measure, and items that do not are eliminated.

The Item-Validity Index

• A statistic that indicates to what degree a test is measuring what it purports to measure.
• The higher the item-validity index, the greater the test's criterion-related validity.

Item Characteristic Curve

• Provides a graphic representation of item discrimination and difficulty.

Other Considerations in Item Analysis: Guessing

• Ability to rule out one or more of the distractor alternatives represents informed guessing.
• Test takers should be instructed to answer only when certain, or to complete all items and guess when in doubt.
• There is no single solution to date for the problem of guessing and its effect on test scores.
• Instructions to test administrators may state that guessing is not allowed.
• Provide specific instructions for scoring and interpreting omitted items.

Item Fairness

• A biased item favors one group over another when group ability differences are are held constant.
• These items are identified by ICC Curves.
• Items are not fair if total test scores between groups are the same, but the ICC Curve shapes are different.

Qualitative Item Analysis

• Uses a variety of non-statistical verbal procedures to gauge the viewpoint of test takers so improvements can be done based on feedback.
• Interviews and group discussions are used to gather information.

Qualitative Item Analysis Questions

• Test length, gauging feelings about length of time to complete and number of items.
• Test language, and determining if any aspects of the test were difficult to understand.
• Giving an opportunity to give feedback might help test takers to lash out at instructor/ institution if did not have good experience.

Think Aloud Test Administration

• "Think aloud” test administration is a qualitative research tool designed to shed light on the test-taker's thought processes during the test.
• Individuals verbalize their thoughts as they occur during one-to-one test administration.
• Yields valuable insights for achievement or personality tests during item misinterpretation.

Expert Panels and Sensitivity Reviews

• A sensitivity review is a study of test items, typically conducted during the test development process.
• Evaluates items for fairness to all prospective test-takers and for offensive language, stereotypes, or situations.

Test Revision

• It is a stage in New Test Development because some items have many weaknesses, therefore becoming candidates for deletion or revision and item pool items may be picked.
• Test revision can occur if stimulus materials look dated and test-takers cannot relate to them.
• Verbal content of items and instructions can contain dated vocabulary to be changed.
• Cultural shifts can cause new word meaning to offend.
• Revision could improve reliability and validity, while new theory can change test items.

Cross-Validation and Co-Validation

• Cross-validation refers to the revalidation of a test on a sample of test-takers other than those on whom test performance was originally found to be a valid predictor of some criterion.
• Co-validation is the test validation process conducted on two or more tests measuring the same aspect using the same sample of test-takers.