Qualities of Language Tests

Test Usefulness: Qualities of Language Tests

• The most important quality of a language test is its usefulness.
• Test usefulness has not been defined precisely enough to provide a basis for designing and developing a test to determine its usefulness.
• Test usefulness provides a metric to evaluate tests and all aspects of test development and use.
• All test development and use should be informed by a model of test usefulness.
• A model includes six test qualities: reliability, construct validity, authenticity, interactiveness, impact, and practicality.
• Three principles operationalize the model of usefulness in language tests.
• The model and principles provide a basis for answering the question: How useful is this particular test for its intended purpose(s)?
• The traditional approach to describing test qualities has been to discuss maximizing them all.
• Some language testers believe that maximizing one quality leads to the virtual loss of others.
• Some believe reliability and validity conflict, or that authentic and reliable test tasks are not simultaneously possible.
• A more reasonable position is that there is tension among test qualities, but this need not lead to abandonment of any.
• Test developers need to recognize the complementarity among the qualities.
• An appropriate balance among the qualities is needed, which varies from one testing situation to another.
• An appropriate balance can be determined only by considering the different qualities in combination as they affect overall usefulness.

Usefulness Equation

• Usefulness = Reliability + Construct validity + Authenticity + Interactiveness + Impact + Practicality
• Test usefulness can be described as a function of several different qualities, all of which contribute in unique but interrelated ways to the overall usefulness of a given test.
• A basis for operationalizing this view of usefulness in the development and use of language tests is provided by three principles.
• The overall usefulness of the test is to be maximized, rather than the individual qualities that affect usefulness.
• The individual test qualities cannot be evaluated independently, but must be evaluated in terms of their combined effect on the overall usefulness of the test.
• Test usefulness and the appropriate balance among the different qualities cannot be prescribed in general, but must be determined for each specific testing situation.
• Any given language test must be developed with a specific purpose, a particular group of test takers, and a specific target language use domain in mind to be useful.
• Usefulness thus cannot be evaluated in the abstract, for all tests.
• Descriptions about test usefulness can be made in terms of the six test qualities, outline general considerations, and procedures for assessing these.
• There are no general prescriptions about either what the appropriate balance among the different qualities should be or what are minimum acceptable levels.

Overall Usefulness

• Evaluating the overall usefulness of a given test is essentially subjective.
• In a large-scale test, the test developer may want to design the test to achieve the highest possible levels of reliability and validity.
• In a classroom test, the teacher may want to utilize test tasks that will provide higher degrees of authenticity, interactiveness, and impact.
• It is essential to take a systemic view, considering tests as part of a larger societal or educational context.
• An important distinction between tests and other components of an instructional program is in their purpose.
• While the primary purpose of other components is to promote learning, the primary purpose of tests is to measure.
• Tests can serve pedagogical purposes, but this is not their primary function.
• Tests provide the major justification for using test scores-numbers as a basis for making inferences or decisions.

Reliability

• Reliability is often defined as consistency of measurement.
• A reliable test score will be consistent across different characteristics of the testing situation.
• Reliability can be considered to be a function of the consistency of scores from one set of tests and test tasks to another.

Reliability Equation

• Scores on test tasks with characteristics A <-> Scores on test tasks with characteristics A'
• The double-headed arrow is used to indicate a correspondence between two sets of task characteristics which differ only in incidental ways.
• It should not make any difference to a particular test taker whether she takes the test on one occasion and setting.
• In a test designed to rank order individuals, scores should rank individuals in the same order across different forms.
• In a test designed to distinguish masters from non-masters, the test should identify the same individuals.
• A given composition should receive the same score irrespective of which particular rater scored it.
• Reliability is clearly an essential quality of test scores, for unless test scores are relatively consistent, they cannot provide information about the ability.
• They need to be recognized that it is not possible to eliminate inconsistencies entirely.
• Variations across test tasks should be minimized, which do not correspond to variations in TLU tasks.

Construct Validity

• Construct validity pertains to the meaningfulness and appropriateness of the interpretations that are made on the basis of test scores.
• It is crucial to be able to justify these interpretations when scores are interpreted as indicators of language ability.
• Test developers and users must be able to provide adequate justification for any interpretation of a given test score.
• Construct validity is the extent to which a given test score can interpret as an indicator of the measure.
• It also relates to the domain of generalization to which score interpretations generalize.
• Construct validation is the ongoing process of demonstrating that a particular interpretation of test scores is justified.
• Several types of evidence (for example, content relevance and coverage, concurrent criterion relatedness, predictive utility) can be provided as part of the validation process.

Construct Validity Equation

• TEST SCORE with construct definition and characteristics of the test task <-> LANGUAGE ABILITY in Domain of Generalization
• Test scores are to be interpreted appropriately as indicators of the ability intended to measure with respect to a specific domain of generalization.
• We need to consider both the construct definition and the characteristics of the test task.

Reliability Summary

• The primary purpose of a language test is to provide a measure that can be interpreted as an indicator of an individual's language ability.
• Reliability and construct validity are essential to the usefulness of any language test.
• Reliability is a necessary condition for construct validity, and hence for usefulness.
• Reliability is not a sufficient condition for either construct validity or usefulness.

Authenticity

• Performance on language tests corresponds to language use in specific domains other than the language test itself.
• A test task whose characteristics correspond to those of TLU tasks as relatively authentic.
• Authenticity is defined as the degree of correspondence of the characteristics of a given language test task to the features of a TLU task.
• Arrow 'B' in Figure 1.1 in Chapter 1 is an example of this relationship.
• Authenticity provides a means for investigating the extent to which score interpretations generalize beyond performance on the test to language use in the TLU domain, or to other similar nontest language use domains.
• This links authenticity to construct validity, since investigating the generalizability of score interpretations is an important part of construct validation.
• Authenticity is important because of its potential effect on test takers' perceptions of the test and, hence, on their performance.
• One way in which test takers and test users tend to react to a language test is in terms of the perceived relevance, to a TLU domain, of the test's topical content and the types of tasks required.
• Defining authenticity in terms of task characteristics promotes authentic tests.

Authenticity Equation

• Characteristics of the test task <-> Authenticity <-> Characteristics of the TLU task
• Identified critical features must define tasks in the TLU domain, using as a starting point a framework of task characteristics such as that described in the next chapter.
• Definition of authenticity can apply to a wide variety of domains, including language classrooms in which the teaching is communicative, or task-based.

Interactiveness

• Interactiveness is the extent and type of involvement of the test taker's individual characteristics in accomplishing a test task.
• Most relevant characteristics include language ability (language knowledge and strategic competence, or metacognitive strategies), topical knowledge, and affective schemata.
• The interactiveness of a given language test task can thus be characterized in terms of the ways in which the test taker's areas of language knowledge, metacognitive strategies, topical knowledge, and affective schemata are engaged by the test task.
• A test task that requires a test taker to relate the topical content of the test input to her own topical knowledge is likely to be relatively more interactive than one that does not.

Interactiveness Equation

• Language Use (Language knowledge, Metacognitive strategies, Topical Knowledge, Affective schemata) <-> INTERACTIVENESS <-> Characteristics of language test task
• Authenticity pertains to the correspondence between test tasks and TLU tasks, and must thus consider the characteristics of both kinds of tasks.
• Interactiveness resides in the interaction between the individual (test taker or language user) and the task (test or TLU).
• Tasks must also involve the areas of language knowledge and competence of the test taker.

Impact

• Impact refers to a test's influence on society, education systems, and individuals.
• Test use impacts at the micro level, in terms of individuals affected by the particular test use, and macro level, in terms of the educational system or society.
• The very acts of administering and taking a test imply certain values and goals, and have consequences.
• Whenever test is used, values and goals are in specific context and the test choice will have specific consequences for, or impact on, both the individuals and the system involved.

Washback

• Impact includes what is referred to as 'washback': the effects of testing on teaching and learning.
• Washback has the potential for affecting not only individuals, but the educational system as well.
• It can be more complex and thorny than simply the effect of testing on teaching.
• It is a good idea to look into what test developers feel is the washback of tests.
• Impact stakeholders are: the test takers, test decision makers, those indirectly effective (future classmates, employers), and those of the system effect.

Taking the Test

• Individuals can be affected by three aspects of the testing procedure.
• There is the experience of taking the test itself and preparing for it, the feedback one receives during performance, and the decisions made about those on the basis of test scores.
• Topical knowledge, for example, can be affected if the test provides topical or cultural information that is new.
• Test takers' areas of language knowledge may also be affected by the test.
• The test taker may improve her language knowledge either while taking the test or from feedback received.
• If test takers are involved in the test design and development, test tasks are likely to be perceived as more authentic and interactive.
• Feedback needs to be relevant, complete, and meaningful to the test taker as possible, specifically in the form of scores and verbal descriptions.
• Finally acceptance of high-stakes test may have serious consequences if there are inaccurate and unfair decision.
• Fair decisions are those that are equally appropriate, regardless of individual test takers' group membership.
• What result might occur of the test taken and their overall impact?
• This can be measured in 4 points: the test intended uses, potential consequences as a result, ranking outcomes according to desire, and collect information regarding the outcomes.
• If the effects of test used are negative that this needed. We can always look for alternatives.

Impact on Teachers

• Teachers and instructional programs are also impacted by the test and need to be assessed as well given the goals of the program.
• As most teachers are concerned: do they feel that what they teach doesn’t correlate with the curriculum taught in class?

###Impacts On Society

• There are also impacts on society and education systems as it leads to change in test use.
• Should the cultural aspects and social goals be measured to avoid value discrimination and to align to societal norms?

Practicality

• Practicality is different in nature from the other five qualities.
• Practicality pertains primarily to the ways in which the test will be implemented, and, to a large degree, whether it will be developed and used at all.
• Practicality determines the degree to which test specifications can be met within the limits of existing resources.
• Threshold: If the resource demands of the test specification do not exceed the available resources at any stage in test development, then the test is practical and development and use can proceed.
• Practicality: as is shown in the formula below requires that the available resources and the required resources must at minimum be equal, and thus a practical test is one whose design, development, and use do not require more resources than are available.

Practicality Equation

• Practicality = Available resources / Required resources
• Three resources need to be considered: human, material, or financial.
• Practicality can change from one situation to another, so there must be some sense and reason why particular areas should be used.
• Resource allocations should be made based the demands of the resources needed to make the test appropriate.

Conclusion

• Tests should be useful and aligned to the testing situation.
• Developers must specifically consider those qualities and not merely in terms of theories and stats.
• Finally, from these, planning that should be based ex ant rather than ex post facto.