The Politics of Policy Evaluation PDF

The Oxford Handbook of Public Policy Robert Goodin (ed.) et al. https://doi.org/10.1093/oxfordhb/9780199548453.001.0001 Published: 2008 Online ISBN: 9780191577413 Print ISBN: 9780199548453...

The Oxford Handbook of Public Policy Robert Goodin (ed.) et al. https://doi.org/10.1093/oxfordhb/9780199548453.001.0001 Published: 2008 Online ISBN: 9780191577413 Print ISBN: 9780199548453 Downloaded from https://academic.oup.com/edited-volume/28180/chapter/213055044 by Universiteit van Amsterdam user on 17 December 2024 Search in this book CHAPTER 15 The Politics of Policy Evaluation  Mark Bovens, Paul 't Hart, Sanneke Kuipers https://doi.org/10.1093/oxfordhb/9780199548453.003.0015 Pages 319–335 Published: 02 September 2009 Abstract This article discusses the politics of policy evaluation and approaches this in two ways, each with its own shortcomings and crucial strengths. The rst approach looks at the roles and functions of policy evaluation and puts them in the wider politics of public policy making. The second looks at how the key schools of policy analysis propose to deal with the contested and inherently political nature of evaluation. The article ends with an original view of how policy analysis may cope with the challenge of ex post evaluation. Keywords: policy evaluation, roles and functions, public policy making, policy analysis, evaluation, ex post evaluation Subject: Comparative Politics, Public Policy, Politics Series: Oxford Handbooks Collection: Oxford Handbooks Online  1. Evaluation between “Learning” and “Politicking” In this chapter policy evaluation refers to the ex post assessment of the strengths and weaknesses of public programs and projects. This implies we shall not address the voluminous literature on ex ante policy analysis, where methods to evaluate policy alternatives are developed and o ered to policy makers and other stakeholders as decision‐making aids (see, e.g., Nagel 2002; Dunn 2004). We shall argue that policy evaluation is an inherently normative act, a matter of political judgement. It can at best be informed but Downloaded from https://academic.oup.com/edited-volume/28180/chapter/213055044 by Universiteit van Amsterdam user on 17 December 2024 never fully dominated by scholarly e orts to bring the logic of reason, calculation, and dispassionate truth seeking to the world of policy making. Policy analysis's mission to “speak truth to power” (Wildavsky 1987) is laudable, and should be continued forcefully, but scholars should not be naive about the nature of the p. 320 evaluation game they participate in (Heineman et al. 1990, 1). In the ideal world of policy analysis, policy evaluation is an indispensable tool for feedback, learning, and thus improvement. In the real world of politics, it is always at risk of degrading into a hollow ritual or a blame game that obstructs rather than enhances the search for better governance. When public policies are adopted and programs implemented, the politics of policy making do not come to an end. The political and bureaucratic controversies over the nature of the problems to be addressed and the best means by which to do so that characterize the policy formulation and policy selection stages of the policy cycle do not suddenly abate when “binding” political decisions are made in favour of option X or Y. Nor do the ambiguities, uncertainties, and risks surrounding the policy issue at stake evaporate. They merely move from the main stage, where political choices about policies are made, to the less visible arenas of policy implementation, populated by (networks of) bureaucratic and non‐governmental actors who are involved in transforming the words of policy documents into purposeful actions. At one time or another, the moment arrives to evaluate what has been achieved. This moment may be prescribed by law or guided by the rhythm of budget or planning and control cycles. It may, however, also be determined by more political processes: the replacement of key o cials, elections that produce government turnovers, incidents or gures that receive publicity and trigger political calls for an investigation, and so on. Whatever its origins, the ideal‐typical structure of a formal evaluation e ort is always the same: an evaluating body initiates an investigation with a certain scope (what to evaluate: which programs/projects, policy outcomes, and/or policy‐making processes, over which time period?); it employs some—explicit or implicit —evaluation criteria; it gathers and analyzes pertinent information; it draws conclusions about the past and recommendations for the future; and it presents its ndings. Beneath this basic structure, tremendous variations exist in evaluation practices (Fischer 1995; Vedung 1997; Weiss 1998; Weimer and Vining 1999; Nagel 2002; Dunn 2004). They di er in their analytical rigor, political relevance, and likelihood to produce meaningful learning processes (cf. Rose 1993). Bodies that conduct evaluations range from scienti c researchers acting on their own accord to consulting rms to public think tanks, and from institutionalized watch dogs such as ombudsmen or courts of audit, to political bodies such as parliamentary commissions. Some of these evaluations are discreet and for direct use by policy makers; others occur in a blaze of publicity and are for public consumption and political use. One and the same policy program or episode may be evaluated by several of these bodies simultaneously or over time. It frequently happens that one type of evaluation exercise triggers others. For instance, the crash of a Dutch military cargo plane at Eindhoven airport in 1996 and the subsequent disaster response by the military and local authorities led to no less than fteen separate investigation e orts by various government bodies, courts, and think tanks. This cascading e ect was partly caused by the fact that both the p. 321 cause of the accident and the adequacy of the response were subject to speculation and controversy, including the taking of provisional disciplinary sanctions against military airport o cials. Moreover, di erent evaluation bodies may even compete overtly: government‐initiated versus parliamentary evaluations, di erent chambers of parliament with di erent political majorities each conducting their own  investigations into some presumed policy asco, governmental versus stakeholder evaluations, national versus IGO evaluations, and so on. The Reagan government's so‐called Iran‐Contra a air (which included the selling of arms to Iran in the hope of securing the release of American hostages held by Shi'ites in Lebanon) set in motion three evaluation e orts: one by a blue‐ribbon presidential commission, one by the Senate, and one by the House of Representatives. Not surprisingly, the three reports were all critical of the course and outcomes of the policy, but di ered markedly in the attribution of responsibility for what happened (see Draper 1991). Downloaded from https://academic.oup.com/edited-volume/28180/chapter/213055044 by Universiteit van Amsterdam user on 17 December 2024 In the ideal world of the positivist social scientist, we stand to gain from this multiplicity: presumably it results in more facts getting on the table, and thus a more solid grasp of what happened and why. In the real world, multiple evaluations of the same policy tend to be non‐cumulative and non‐complementary. Their methods and ndings diverge widely, making it hard to reach a single authoritative or at least consensual judgement about the past and to draw clear‐cut lessons from it. In this chapter we shall approach the politics of policy evaluation in two ways. First we shall elaborate on the roles and functions of policy evaluation in the broader politics of public policy making. Then we shall look at how key schools of policy analysis propose to deal with the essentially contested, inherently political nature of evaluation. Each, we argue, has crucial strengths and shortcomings. In the nal section, we o er our own view of how policy analysis may cope with the conundrum of ex post evaluation. 2. The Politics of Policy Evaluation It is only a slight exaggeration to say, paraphrasing Clausewitz, that policy evaluation is nothing but the continuation of politics by other means. This is most conspicuous in the assessment of policies and programs that have become highly controversial: because they do not produce the expected results, because they were highly contested to begin with, because they are highly costly and/or ine cient, because of p. 322 alleged wrongdoings in their implementation, and so on. The analysis of such policy episodes is not a politically neutral activity, which can be done by fully detached, unencumbered individuals (Bovens and 't Hart 1996). The ominous label of “failure” or “ asco” that hovers over these policies entails a political statement. Moreover, once policies become widely viewed as failures, questions about responsibility and sometimes even liability force themselves on to the public agenda. Who can be held responsible for the damage that has been done to the social fabric? Who should bear the blame? What sanctions, if any, are appropriate? Who should compensate the victims? In view of this threat to their reputations and positions, many of the o cials and agencies involved in an alleged asco will engage in tactics of impression management, blame shifting, and damage control. The policy's critics, victims, and other political stakeholders will do the opposite: dramatize the negative consequences and portray them as failures that should, and could, have been prevented (cf. Weaver 1986; Gray and 't Hart 1998; Anheier 1999; Hood 2002). The pivotal importance of blaming entails the key to understanding why the evaluation of controversial policy episodes itself tends to be a highly adversarial process. The politics of blaming start at the very instigation of evaluation e orts: which evaluation bodies take on the case, how are they composed and briefed (Lipsky and Olson 1977)? It is highlighted especially by the behaviour of many stakeholders during the evaluation process. To start with, the very decision to have an incident or program evaluated may be part of a political strategy. Penal policy constitutes an interesting example of this. In most countries, prison escapes take place from time to time, and in some periods their incidence increases. But there appears to be no logical connection between objecti able indicators of the severity of the problem such as their frequency, their success rate, the number of escapees per annum, and the likelihood of major evaluation and learning e orts being undertaken at the political level. In the Netherlands, for example, political commotion about prison escapes rose to peak levels at a time when all penal system performance indicators were  exceptionally good after an earlier period of problems and unrest. Rather, the scale, scope, and aims of a post‐escape investigation seem to be a function of purely coincidental factors such as the method of escape and the level of violence, as well as the nature of the political climate regarding criminal justice and penal policy at any given time (Boin 1995; Resodihardjo forthcoming). Even seemingly routine, institutionalized evaluations of unobtrusive policy programs tend to have political edges to them, if only in the more subterraneous world of sectoral, highly specialized policy networks. Even in those less controversial instances, policy evaluations are entwined with processes of accountability and Downloaded from https://academic.oup.com/edited-volume/28180/chapter/213055044 by Universiteit van Amsterdam user on 17 December 2024 lesson drawing that may have winners and losers. However technocratic and seemingly innocuous, every policy program has multiple stakeholders who have an interest in the outcome of the evaluation: decision makers, executive agencies, clients, pressure groups. All of them know that apart from (post‐election) political turnovers or crucial court cases, evaluations are virtually the only moments when existing policy trajectories can be reassessed and historical path dependencies may be broken (cf. Rose and Davies 1994). Evaluations hold the promise of a reframing of a program's rationale and objectives, a recalibration of p. 323 the mix of policy instruments it relies on, a reorganization of its service delivery mechanisms, and, yes, a redistribution of money and other pivotal resources among the various actors involved in its implementation. Hence in the bulk of seemingly “low‐politics program” evaluations, the stakes for the circle of interested parties may be high (Vedung 1997, 101–14; Pawson and Tilly 1997; Radin 2000; Hall and Hall 2004, 34–41). Astute players of the evaluation game will therefore attempt to produce facts and images that suit their aims. They will produce—or engage others to produce— accounts of policy episodes that are, however subtly, framed and timed to convey certain ideas about what happened, why, and how to judge this, and to obscure or downplay others. They will try to in uence the terms of the evaluation, in particular also the choice and weighting of the criteria by which the evaluators arrive at their assessments. Evaluating bodies and professional policy analysts will inevitably feel pressures of this kind building up during the evaluation process. The list of tactics used by parties to in uence the course and outcomes of evaluation e orts is long, and somewhat resembles the stratagems of bureaucratic and budgetary politics: evaluators' briefs and modus operandi may be subject to continuous discussion; key documents or informants may prove to be remarkably hard, or sometimes remarkably easy, to encounter; the drafting and phrasing of key conclusions and recommendations may be a bone of contention with stakeholder liaisons or in advisory committees; there may be informal solicitations and démarches by stakeholders; reports may be prematurely leaked, deeply buried, or publicly lambasted by policy makers. In short, even the most neutral, professional evaluators with no political agenda of their own are likely to become both an object and, unwittingly or not, an agent of political tactics of framing, blaming, and credit claiming (see Bovens et al. 1999; Brändström and Kuipers 2003; Pawson and Tilley 1997; Stone 1997).  3. Dealing with the Political in Policy Evaluation Policy scientists have long recognized these political rami cations of policy evaluation, but have found it impossible to agree on how to cope with them. The cybernetic notion of evaluation as a crucial, authoritative “feedback stream” that enhances re ection, learning, and thus induces well‐considered policy continuation, change, or termination, has ceased to be a self‐evident rationale for elaborating evaluation p. 324 theory and methodology. The political realities have simply been too harsh. “The eld of evaluation is Downloaded from https://academic.oup.com/edited-volume/28180/chapter/213055044 by Universiteit van Amsterdam user on 17 December 2024 currently undergoing an identity crisis,” lamented two advocates of the positivist approach to policy analysis twenty years ago (Palumbo and Nachimas 1983, 1). At that time, a multitude of alternative approaches had taken the place of the single methodology and assumption set of the classical, rst‐ generation policy analyst of the science‐for‐policy kind. The mood of optimism and its belief in planned government intervention that had characterized for instance Johnson's “Great Society Program” in the United States was replaced by a mood of scarcity and skepticism (Radin 2000; see also Rossi and Freeman 1993, 23). The focus in policy analysis shifted from ex ante evaluation to ex post evaluation, because the creation of large public policies became less fashionable than the scrutiny of existing programs (Radin 2000, 34). As Dye (1987, 372) put it, it became “exceedingly costly for society to commit itself to large‐scale programs and policies in education and welfare, housing, health and so on, without any real idea about what works.” Instrumental policy evaluation continued to be a stronghold in the eld of policy analysis, although it was now increasingly exploited as a tool to measure ex post cost‐bene t ratios to support retrenchment e orts by New Right governments (Radin 2000; Fischer 1995). At the same time, the value trade‐o s and political controversies involved in the scrutiny of existing public policies raised questions about the neutrality assumptions of policy analysis. The apolitical, quantitative assessments of policy outcomes that were supposed to support optimal decision making in the 1950s and 1960s became the subject of increasing criticism. The judgemental character of policy evaluation provoked discussion about its inherently normative, political nature, and about the initial stubbornness among policy analysts steeped in the rationalistic tradition to deny that evaluating policy impact is “an activity which is knee‐deep in values, beliefs, party politics and ideology, and makes ‘proving’ that this policy had this or that impact a notion which is deeply suspect” (Parsons 1995, 550). A new generation of policy analysts came up, and rejected the fundamental assumption that it is possible to measure policy performance in an objective fashion. Like Hugh Heclo, they argued that “a mood is created in which the analysis of rational program choice is taken as the one legitimate arbiter of policy analysis. In this mood, policy studies are politically deodorized—politics is taken out of policy‐making” (Heclo 1972, 131). Several approaches to policy evaluation were developed to “bring politics back in” (Nelson 1977; Fischer 1980; Majone 1989). The diversity of evaluation approaches that has developed since will be discussed here in terms of two traditions. The dividing line between those traditions will be based on the way norms, values, interests, and power are accommodated in evaluation. The rationalistic tradition with its strong emphasis on value neutrality and objective assessments of policy performance tries to save evaluation from the pressures of politics, by ignoring these pressures or somehow superseding them. In contrast, the argumentative tradition sees policy evaluation as a contribution to the informed debate among competing interests and therefore explicitly incorporates politics in the ex post analysis of policy performance. p. 325 3.1 Rationalistic Policy Evaluation The rationalists advocate a rigorous separation of facts and values and explicitly strive to produce apolitical knowledge (Hawkesworth 1988; Lynn 1999; Mabry 2002). Policy analysis is rooted in positivism and strives to produce factual data about societal structures and processes by employing concepts and methods borrowed from the natural and physical sciences. Policy analysis serves to bring about rational decision making in the policy process. Judgements about a program's or project's e ectiveness and e ciency have to be based on reliable empirical data. It is the task of the policy analyst to produce information that is free Downloaded from https://academic.oup.com/edited-volume/28180/chapter/213055044 by Universiteit van Amsterdam user on 17 December 2024 from its psychological, cultural, and linguistic context. Because such information transcends historical and cultural experiences, it is assumed to have political and moral neutrality. Rational methods can be used to construct theoretical policy optimums (in terms of both e ciency and e cacy); in evaluation one can then measure the distance of actual policy outcomes from this optimum. Evaluation thus yields policy‐relevant information about the discrepancies between the expected and factual policy performance (Dunn 2004). According to Berk and Rossi (1999, 3) evaluation research is “essentially about providing the most accurate information practically possible in an even‐handed manner.” Political decisions and judgements require testimonies based on generally applicable and scienti cally valid knowledge for “it is rarely prudent to enter a burning political debate armed with only one case study” (Chelimsky 1987, 27). The e ort to “remedy the de ciencies in the quality of human life” requires continuous evaluation directed at the improvement of policy programs, based on valid, reliable empirical information (Rossi, Freeman, and Lipsey 1999, 6). This form of policy evaluation assumes the existence of an exogenously produced, i.e. given, set of clear and consistent policy goals and/or other evaluation standards. It also assumes intersubjective agreement on which indicators can be identi ed to measure the achievement of these goals. Some rationalistic evaluators might acknowledge that evaluation is in essence a judgement on the value of a policy or program and therefore goes beyond the realms of empirical science (Dunn 2004), or that policy evaluation takes place in a political context with a multitude of actors and preferences involved. For example, Nagel's (2002) approach to ex ante policy evaluation includes political considerations to the extent that it proposes a “win‐ win analysis” to be made: a survey and assessment of the preferred alternatives of political actors involved to nd among them an alternative that exceeds the best initial expectations of representatives of the major viewpoints in the political dispute. But their bottom line is clear: Dunn (2004), for instance, asserts that the outcome of policy evaluation is a value judgement, but that the process of evaluation nevertheless has to provide unbiased information. Likewise, the Rossi et al. (1999) handbook self‐consciously advocates the systematic application of social research procedures, emphasizing the analysis of costs and bene ts, targets, and e ects. Earlier, they did not only argue that evaluation should provide value‐neutral information to political decision makers, but also that context‐sensitive, biased, and argumentative p. 326 evaluators are “engaged in something other than evaluation research” (Rossi and Freeman 1993, 33). A remarkably in uential institutionalized manifestation of the rationalistic approach to policy evaluation is the Organization for Economic Co‐operation and Development (OECD). The OECD aims to foster good governance by monitoring and comparing economic development, deciphering emerging issues, and identifying “policies that work” (according to its own website at www.oecd.org). Its country reports have gained considerable authority over the years and its standardized comparisons are used as verdicts on national policy performance.  3.2 Argumentative Policy Evaluation This brings us to the other camp. The argumentative critics of the rationalist approach complain that the positivist world view is fundamentally distorted by the separation of facts from values. Policy intervention with respect to social and political phenomena is an inherently value‐laden, normative activity which allows but for a biased evaluation (Fischer and Forester 1993; Guba and Lincoln 1989). The so‐called “post‐ positivists” or social constructivists understand society as an organized universe of meanings, instead of a mere set of physical objects to be measured. It is not the objects per se that are measured, but the Downloaded from https://academic.oup.com/edited-volume/28180/chapter/213055044 by Universiteit van Amsterdam user on 17 December 2024 interpretation of the objects by the scientist. The system of meanings shapes “the very questions that social scientists choose to ask about society, not to mention the instruments they select to pursue their questions” (Fischer 1995, 15). Facts depend on a set of underlying assumptions that give meaning to the reality we live in. These assumptions are in uenced by politics and power, and empirical ndings based on these underlying assumptions “tend to reify a particular reality” (Fischer 1998, 135). The rst evaluation of the “Great Society's” Head Start program for socially deprived children was a measurement of the participating children's cognitive development shortly after the program's implementation. This measurement was a relatively simple quantitative assessment of only one of the program's possible positive e ects. It showed a lack of improvement in the children's cognitive capacities and that, compared to the total costs of the government intervention, the program had been an expensive failure. If only the evaluators had accepted the program's underlying assumptions that children would bene t from their participation by gaining social experience that would teach them how to function successfully in middle‐class‐oriented educational institutions, they would have awaited the results of long‐term monitoring. The short‐term evaluation outcomes were very welcome to the new Nixon administration as an argument to cut down on Head Start considerably (Fischer 1995). The short‐term cost‐bene t analysis that be tted Nixon's attack on large‐scale government planning e orts served to prove him right. p. 327 Likewise, the standardized comparison of budgetary and performance gures employed by think tanks such as the OECD leaves open much interpretative and therefore contested ground. One ground for dispute concerns the construction of the categories. In the OECD's report, the Belgian unemployment rate was put just above 8 per cent of the total labor force; in contrast, the Belgian unemployment agency's (www.rva.be) own reports state that it pays unemployment bene ts to more than a million people monthly, i.e. 23.5 per cent of the labor force (Arents et al. 2000). The disparity can only be explained by examining closely the de nitions of “unemployment” used in studies such as these. To post‐positivists this is just one example among many. They claim it is an illusion to think that separation between values and facts is possible. Moreover, it is impossible to create a division of labor between politics and science where politicians authoritatively establish policy values and scientists can neutrally assess whether the policy outcomes meet the prior established norms (Majone 1989). Policy analysts should actively engage in and facilitate the debate on values in policy making and function as a go‐between for citizens and politicians. By attempting to provide “the one best solution” in ex ante policy analysis and the “ultimate judgement” in ex post evaluation, the ambition of most (rationalist) policy scientists has long been to settle rather than stimulate debates (Fischer 1998). The advocates of the argumentative approach see yet another mission for policy analysis, including evaluation. Knowledge of a social object or phenomenon emerges from a discussion between competing frameworks (Yanow 2000). This discussion—or discursive interaction—concerning policy outcomes can uncover the presuppositions of each framework that give meaning to its results from empirical research. Policy analysts can intervene in these discussions to help actors with di erent belief systems understand where their disagreements have epistemological and ethical roots rather than simply boiling down to di erent interests and priorities (Van Eeten 1999; Yanow 2000). If evaluations can best be understood as forms of knowledge based on consensually accepted beliefs instead of on hard‐boiled proof and demonstration (Danziger 1995; Fischer 1998), it becomes quite important to ascertain whose beliefs and  whose consensus dominates the retrospective sense‐making process. Here, the argumentative approach turns quite explicitly to the politics of policy evaluation, when it argues that the deck with which the policy game is played at the evaluation can be stacked as a result of institutionalized “mobilization of bias.” In that sense evaluation simply mirrors the front end of the policy process (agenda setting and problem de nition): some groups' interests and voices are organized “in” the design and management of evaluation proceedings, whereas other stakeholders are organized “out.” Some proponents of argumentative policy evaluation therefore argue that the policy analyst should not just help expose the meaning systems by which these facts are being interpreted; she should also ensure that under‐represented groups can make Downloaded from https://academic.oup.com/edited-volume/28180/chapter/213055044 by Universiteit van Amsterdam user on 17 December 2024 their experiences and assessments of a policy heard (Fischer and Forester 1993; Dryzek 2000). DeLeon (1998) quali es the argumentative approach's enthusiasm about “consensus through p. 328 deliberation.” He cautions that the democratic ambitions of the post‐positivists bear the risk of the tyranny of the majority as much as the shortcomings of positivism. The in nite relativism of the social constructivists makes it di cult to decide just whose voice is most relevant or whose argument is the strongest in a particular policy debate. The evaluation by social constructivists may well recognize the political dimension of analytic assessments of policy outcomes, but it does not by de nition lead us to more carefully crafted political judgements. 4. Doing Evaluation in the Political World How then, should we cope with the normative, methodological, and political challenges of policy evaluation? In our view, the key challenge for professional policy evaluators should not be how to save objectivity, validity, and reliability from the twin threats of epistemological relativism and political contestation. This project can only lead to a kind of analytical self‐deception: evaluators' perfunctory neglecting or “willing away” pivotal philosophical queries and political biases and forces (Portis and Levy 1988). It may be more productive to ask two alternative questions. How can policy analysts maximize academic rigor without becoming politically irrelevant? And how can policy evaluations be policy relevant without being used politically? The rst question requires evaluators to navigate between the Scylla of seemingly robust but irrelevant positivism and the Charybdis of politically astute but philosophically problematic relativism. The second question deals with the applied dimension. It alerts evaluators to the politics of evaluation that are such a prominent feature of contemporary policy struggles and of political attempts to “learn” from evaluations. The approach to evaluation advocated here should be viewed within the context of a broader repositioning of policy science that we feel is going on, and which entails an increased acceptance of the once rather sectarian claim of the argumentative approach that all knowledge about social a airs—including public policy making—is based on limited information and social constructions. If one does so, the hitherto predominantly positivist and social engineering‐oriented aims and scope of policy evaluation need to be revised or at least broadened. Be tting such a “revisionist” approach to policy analysis is the essentially incrementalist view that public policy makers' best bet is to devote the bulk of their e orts to enabling p. 329 society to avoid, move away from, and e ectively respond to what, through pluralistic debate, it has come to recognize as important present and future ills (Lindblom 1990). Policy analysis is supposed to be an integral part of this project, but not in the straightforward manner of classic “science for policy.” Instead, the key to its unique contribution lies in its re ective potential. We agree with Majone (1989, 182) that: It is not the task of analysts to resolve fundamental disagreements about evaluative criteria and standards of accountability; only the political process can do that. However, analysts can contribute to societal learning by re ning the standards of appraisal and by encouraging a more sophisticated understanding of public policies than is possible from a single perspective.  This also goes for evaluating public policies and programs. Again we cite Majone (1989, 183): “The need today is less to develop ‘objective’ measures of outcomes—the traditional aim of evaluation research—than to facilitate a wide‐ranging dialogue among advocates of di erent criteria.” In a recent cross‐national and cross‐sectoral comparative evaluation study, an approach to evaluation was developed that embodies the main thrust of the “revisionist” approach (Bovens, 't Hart, and Peters 2001). The main question of that project, which involved a comparative assessment of critical policy episodes and programs in four policy sectors in six European states, was how the responses of di erent governments to Downloaded from https://academic.oup.com/edited-volume/28180/chapter/213055044 by Universiteit van Amsterdam user on 17 December 2024 highly similar major, non‐incremental policy challenges can be evaluated, and how similarities and di erences in their performance can be explained. A crucial distinction was made between the programmatic and the political dimension of success and failure in public governance. In a programmatic mode of assessment, the focus is on the e ectiveness, e ciency, and resilience of the speci c policies being evaluated. The key concerns of programmatic evaluation pertain to the classical, Lasswellian–Lindblomian view of policy making as social problem solving most rmly embedded in the rationalistic approach to policy evaluation: does government tackle social issues, does it deliver solutions to social problems that work, and does it do so in a sensible, defensible way (Lasswell 1971; Lindblom 1990)? Of course these questions involve normative and therefore inherently political judgements too, yet the focus is essentially instrumental, i.e. on assessing the impact of policies that are designed and presented as purposeful interventions in social a airs. The simplest form of programmatic evaluation—popular to this day because of its straightforwardness and the intuitive appeal of the idea that governments should be held to account on their capacity to deliver on their own promises (Glazer and Rothenberg 2001)—is to rate policies by the degree to which they achieve the stated goals of policy makers. Decades of evaluation research have taught all but the most hard‐headed analysts that despite its elegance, this method has big problems. Goals may be untraceable in policy documents, symbolic rather than substantial, deliberately vaguely worded for political reasons, and contain p. 330 mutually contradictory components. Goals also often shift during the course of the policy‐making process to such an extent that the original goals bear little relevance for assessing the substance and the rationale of the policy that has actually been adopted and implemented in the subsequent years. Clearly, something better was needed. In our view, a sensible form of programmatic policy evaluation does not fully omit any references to politically sanctioned goals—as once advocated by the proponents of so‐ called “goal‐free” evaluation—but “embeds” and thus quali es the e ectiveness criterion by complementing and comparing it with other logics of programmatic evaluation. In the study design, case evaluators had to examine not only whether governments had proven capable of delivering on their promises and e ectuating purposeful interventions. They were also required to ascertain: (a) the ability of the policy‐making entity to adapt its program(s) and policy instruments to changing circumstances over time (i.e. an adaptability/learning capacity criterion); (b) its ability to control the costs of the program(s) involved (i.e. an e ciency criterion). In keeping with Majone's call, these three general programmatic evaluation logics were then subject to intensive debate between the researchers involved in the study: how should these criteria be understood in concrete cases, what data would be called for to assess a case, and what about the relative weight of these three criteria in the overall programmatic assessment? Sectoral expert subgroups gathered subsequently to specify and operationalize these programmatic criteria in view of the speci c nature and circumstances of the four policy areas to be studied. The outcomes of these deliberations about criteria (and methodology) are depicted in Fig. 15.1. The political dimension of policy evaluation refers to how policies and policy makers become represented and evaluated in the political arena (Stone 1997). This is the discursive world of symbols, emotions, political ideology, and power relationships. Here it is not the social consequences of policies that count, but the political construction of these consequences, which might be driven by institutional logics and political  considerations of wholly di erent kinds. In the study described above, the participants struggled a lot with how to operationalize this dimension in a way that allowed for non‐idiosyncratic, comparative modes of assessment and analysis. In the process it became clear that herein lies an important weakness of the argumentative approach: it rightly points at the relevance of the socially and politically constructed nature of assessments about policy success and failure, but it does not o er clear, cogent, and widely accepted evaluation principles and tools for capturing this dimension of policy evaluation. In the end, the evaluators in the study opted for a relatively “thin” but readily applicable set of political evaluation measures: the incidence and degree of political upheaval (traceable by content analysis of press coverage and Downloaded from https://academic.oup.com/edited-volume/28180/chapter/213055044 by Universiteit van Amsterdam user on 17 December 2024 parliamentary investigations, political fatalities, litigation), or lack of it; and changes in generic patterns of political legitimacy (public satisfaction of policy or con dence in authorities and public institutions). An essential bene t of discerning and contrasting programmatic and political evaluation modes is that it p. 331 highlights the development of disparities between a policy‐making entity's programmatic and political performance. This should not surprise the politically astute evaluator: political processes determine whether programmatic success, or lack of it, is acknowledged by relevant stakeholders and audiences. The dominant assessment of many conspicuous “planning disasters”—the Sydney Opera House for example— has evolved over time, as certain issues, con icts, and consequences that were important at the time have evaporated or changed shape, and as new actors and power constellations have emerged (compare Hall 1982 p. 332 to Bovens and 't Hart 1996). In the Bovens et al. study, some remarkable asymmetries between programmatic and political evaluations were identi ed. In the banking sector, for example, (de‐)regulatory policies and/or existing instruments for oversight in Spain, the UK, France, and Sweden did not prevent banking ascos of catastrophic proportions (i.e. major programmatic failures); at the same time, the political evaluation of these policies in terms of the evaluation criteria outlined above was not particularly negative. Likewise, in programmatic terms German responses to the HIV problem in the blood supply were at least as bad as those in France; in France this became the stu of major political scandal and legal proceedings, whereas in Germany the evaluation was depoliticized and no political consequences resulted. These types of evaluation asymmetries defy the commonsense, “just world” hypothesis that good performance should lead to political success, and vice versa. Detecting asymmetries then challenges the analyst to explain these discrepancies in terms of structural and cultural features of the political system or policy sector and the dynamics of the evaluation process in the cases concerned (see Bovens, 't Hart, and Peters 2001, 593.).  Downloaded from https://academic.oup.com/edited-volume/28180/chapter/213055044 by Universiteit van Amsterdam user on 17 December 2024 Fig. 15.1. Programmatic policy evaluation: an example (taken from Bovens et al. 2001, 20–2) Talking not so much about policy analysts but about policy practitioners, Schön and Rein (1994) have captured the approach to policy evaluation advocated here under the heading of “frame‐re ection.” This implies willingness on the part of analysts to re ect continuously upon and reassess their own lenses for looking at the world. In addition, they need to make e orts to communicate with analysts using a di erent set of assumptions. In the absence of such a re ective orientation, policy analysts may nd that they, and their conclusions, are deemed irrelevant by key players in the political arena. Or they may nd themselves set up unwittingly to be hired guns in the politics of blaming. They ought to be neither. Re ective policy analysts may strive for a position as a systematic, well‐informed, thoughtful, and fair‐ minded provider of inputs to the political process of argumentation, debate, maneuvering, and blaming that characterizes controversial policy episodes. In our view, their e ectiveness could be enhanced signi cantly if they adopt a role conception that be ts such a position: explicit about their own assumptions; meticulous in developing their arguments; sensitive to context; and striving to create institutional procedures for open and pluralistic debate. At the same time, since the political world of policy ascos in particular is unlikely to be supportive of such frame re ection, policy analysts need a considerable amount of political astuteness in assessing their own position in the eld of forces and in making sure that their arguments are heard at what they think is the right time, by the right people, and in the right way. Finding ways to deal creatively with the twin requirements of scholarly detachment and political realism is what the art and craft of policy evaluation are all about. p. 333 References Anheier, H. K., ed. 1999. When Things Go Wrong. London: Sage. Google Scholar Google Preview WorldCat COPAC Arents, M., Cluitmans, M. M., and Van der Ende, M. A. 2000. Benefit Dependency Ratios: An Analysis of Nine European Countries, Japan and the US. The Hague: Elsevier. Google Scholar Google Preview WorldCat COPAC Downloaded from https://academic.oup.com/edited-volume/28180/chapter/213055044 by Universiteit van Amsterdam user on 17 December 2024 Berk, R. A., and Rossi, P. H. 1999. Thinking about Program Evaluation. Thousand Oaks, Calif.: Sage. Google Scholar Google Preview WorldCat COPAC Boin, R. A. 1995. The Dutch prison system in the 1990s: organizational autonomy, institutional adversity, and a shi in policy. American Jails, 9(4): 88–91 (part I), and American Jails, 9(5): 87–95 (part II). Google Scholar WorldCat Bovens, M., and 't Hart, P. 1996. Understanding Policy Fiascoes. New Brunswick, NJ: Transaction. Google Scholar Google Preview WorldCat COPAC —— —— et al. 1999. The politics of blame avoidance. Pp. 123–48 in When Things Go Wrong, ed. H. K. Anheier. London: Sage. Google Scholar Google Preview WorldCat COPAC —— , —— , and Peters, B. G. (eds.) 2001. Success and Failure in Public Governance: A Comparative Study. Cheltenham: Edward Elgar. Google Scholar Google Preview WorldCat COPAC Brändström, A., and Kuipers, S. L. 2003. From “normal incidents” to political crises: understanding the selective politicization of policy failures. Government and Opposition, 38: 279–305. 10.1111/1477-7053.t01-1-00016 Google Scholar WorldCat Crossref Chelimsky, E. 1987. The politics of program evaluation. Society, 25: 24–32. 10.1007/BF02695393 Google Scholar WorldCat Crossref Danziger, M. 1995. Policy analysis postmodernized: some political and pedagogical ramifications. Policy Studies Journal, 23: 435–50. 10.1111/j.1541-0072.1995.tb00522.x Google Scholar WorldCat Crossref deLeon, P. 1998. Introduction: the evidentiary base for policy analysis: empiricist versus postpositivist positions. Policy Studies Journal, 26: 109–13. 10.1111/j.1541-0072.1998.tb01927.x Google Scholar WorldCat Crossref Draper, T. 1991. A Very Thin Line: The Iran‐Contra A airs. New York: Hill and Wang. Google Scholar Google Preview WorldCat COPAC Dryzek, J. S. 2000. Deliberative Democracies and Beyond: Liberals, Critics, Contestations. Oxford: Oxford University Press. Google Scholar Google Preview WorldCat COPAC Dunn, W. N. 2004. Public Policy Analysis: An Introduction, 3rd edn. Upper Saddle River, NJ: Prentice Hall. Google Scholar Google Preview WorldCat COPAC Dye, T. R. 1987. Understanding Public Policy, 6th edn. Englewood Cli s, NJ: Prentice Hall. Google Scholar Google Preview WorldCat COPAC Edelman, M. 1988. Constructing the Political Spectacle. Chicago: University of Chicago Press.  Google Scholar Google Preview WorldCat COPAC Fischer, F. 1980. Politics, Values, and Public Policy: The Problem of Methodology. Boulder, Colo.: Westview. Google Scholar Google Preview WorldCat COPAC —— 1995. Evaluating Public Policy. Chicago: Nelson Hall. Google Scholar Google Preview WorldCat COPAC Downloaded from https://academic.oup.com/edited-volume/28180/chapter/213055044 by Universiteit van Amsterdam user on 17 December 2024 —— 1998. Beyond empiricism: policy inquiry in postpositivist perspective. Policy Studies Journal, 26: 129–46. 10.1111/j.1541- 0072.1998.tb01929.x WorldCat Crossref —— , and Forester, J. 1993. The Argumentative Turn in Policy Analysis and Planning. Durham, NC: Duke University Press. Google Scholar Google Preview WorldCat COPAC Glazer, A., and Rothenberg, L. S. 2001. Why Government Succeeds and Why it Fails. Cambridge, Mass.: Harvard University Press. Google Scholar Google Preview WorldCat COPAC Gray, P., and Hart, P. 't. (eds.) 1998. Public Policy Disasters in Western Europe. London: Routledge. Google Scholar Google Preview WorldCat COPAC Guba, E. G., and Lincoln, V. S. 1989. Fourth Generation Evaluation. Newbury Park, Calif.: Sage. Google Scholar Google Preview WorldCat COPAC Hajer, M., and Wagenaar, H. (eds.) 2003. Deliberative Policy Analysis: Understanding Governance in the Network Society. Cambridge: Cambridge University Press. Google Scholar Google Preview WorldCat COPAC Hall, I., and Hall, D. 2004. Evaluation and Social Research: Introducing a Small‐Scale Practice. Basingstoke: Palgrave. Google Scholar Google Preview WorldCat COPAC p. 334 Hawkesworth, M. E. 1988. Theoretical Issues in Policy Analysis. Albany: State University of New York Press. Google Scholar Google Preview WorldCat COPAC Heclo, H. H. 1972. Review article: policy analysis. British Journal of Political Science, 2(1): 83–108. 10.1017/S0007123400008449 Google Scholar WorldCat Crossref Heineman, R. A., Bluhm, W. T. et al. 1990. The World of the Policy Analyst. Chatham, NJ: Chatham House. Google Scholar Google Preview WorldCat COPAC Hood, C. 2002. The risk game and the blame game. Government and Opposition, 37: 15–37. 10.1111/1477-7053.00085 Google Scholar WorldCat Crossref Lasswell, H. 1971. A Pre‐View of Policy Science. New York: Elsevier. Google Scholar Google Preview WorldCat COPAC Lindblom, C. 1990. Inquiry and Change: The Troubled Attempt to Understand and Shape Society. New Haven, Conn.: Yale University Press. Google Scholar Google Preview WorldCat COPAC Lipsky, M., and Olson, D. J. 1977. Commission Politics: The Processing of Racial Crisis in America. New Brunswick, NJ: Transaction. Google Scholar Google Preview WorldCat COPAC Lynn, L. E. 1999. A place at the table: policy analysis, its postpositive critics and the future of practice. Journal of Policy Analysis and Management, 18: 411–24. 10.1002/(SICI)1520-6688(199922)18:33.0.CO;2-G Google Scholar WorldCat Crossref Mabry, L. 2002. Postmodern evaluation—or not? American Journal of Evaluation, 23: 141–57. Google Scholar WorldCat Majone, G. 1989. Evidence, Argument and Persuasion in the Policy Process. New Haven, Conn.: Yale University Press. Google Scholar Google Preview WorldCat COPAC Downloaded from https://academic.oup.com/edited-volume/28180/chapter/213055044 by Universiteit van Amsterdam user on 17 December 2024 Nagel, S. S. 2002. Handbook of Policy Evaluation. Thousand Oaks, Calif.: Sage. Google Scholar Google Preview WorldCat COPAC Nelson, R. R. 1977. The Moon and the Ghetto. New York: Norton. Google Scholar Google Preview WorldCat COPAC Pal, L. A. 1995. Competing paradigms in policy discourse: the case of international human rights. Policy Sciences, 28: 185– 207. 10.1007/BF00999675 Google Scholar WorldCat Crossref Palumbo, D. J., and Nachimas, D. 1983. The preconditions for successful evaluation: is there an ideal type? Policy Sciences, 16: 67–79. 10.1007/BF00138468 Google Scholar WorldCat Crossref Parsons, D. W. 1995. Public Policy: An Introduction to the Theory and Practice of Policy Analysis. Aldershot: Edward Elgar. Google Scholar Google Preview WorldCat COPAC Pawson, R., and Tilley, N. 1997. Realistic Evaluation. London: Sage. Google Scholar Google Preview WorldCat COPAC Portis, E. B., and Levy, M. B. (eds.) 1988. Handbook of Political Theory and Policy Science. Westport, Conn.: Greenwood. Google Scholar Google Preview WorldCat COPAC Radin, B. A. 2000. Beyond Machiavelli: Policy Analysis Comes of Age. Washington, DC: Georgetown University Press. Google Scholar Google Preview WorldCat COPAC Resodihardjo, S. L. forthcoming 2006. Institutional crises and reform: constrained opportunities. Ph.D. dissertation, Leiden University. Rose, R. 1993. Lesson‐Drawing in Public Policy. Chatham, NJ: Chatham House. Google Scholar Google Preview WorldCat COPAC —— , and Davies, P. 1994. Inheritance in Public Policy. New York: Oxford University Press. Google Scholar Google Preview WorldCat COPAC Rossi, P. H., and Freeman, H. E. 1993. Evaluation: A Systemic Approach, 5th edn. Newbury Park, Calif.: Sage. Google Scholar Google Preview WorldCat COPAC —— —— , and Lipsey, M. R. 1999. Evaluation: A Systemic Approach, 6th edn. Thousand Oaks, Calif.: Sage. Google Scholar Google Preview WorldCat COPAC Schön, D., and Rein, M. 1994. Frame Reflection. New York: Basic Books. Google Scholar Google Preview WorldCat COPAC Stone, D. 1997. Policy Paradox: The Art of Political Decision Making. New York: W.W. Norton.  Google Scholar Google Preview WorldCat COPAC Throgmorton, J. A. 1991. The rhetorics of policy analysis. Policy Sciences, 24: 153–79. 10.1007/BF00138058 Google Scholar WorldCat Crossref Van Eeten, M. 1999. Dialogues of the Deaf: Defining New Agendas for Environmental Deadlocks. Del : Eburon. Google Scholar Google Preview WorldCat COPAC Downloaded from https://academic.oup.com/edited-volume/28180/chapter/213055044 by Universiteit van Amsterdam user on 17 December 2024 Vedung, E. 1997. Public Policy and Program Evaluation. New Brunswick, NJ: Transaction. Google Scholar Google Preview WorldCat COPAC Weaver, R. K. 1986. The politics of blame avoidance. Journal of Public Policy, 6: 371–98. 10.1017/S0143814X00004219 Google Scholar WorldCat Crossref Weimer, D. L., and Vining, A. R. 1999. Policy Analysis: Concepts and Practice, 3rd edn. Upper Saddle River, NJ: Prentice Hall. Google Scholar Google Preview WorldCat COPAC p. 335 Weiss, C. H. 1998. Evaluation: Methods for Studying Programs and Policies. Upper Saddle River, NJ: Prentice Hall. Google Scholar Google Preview WorldCat COPAC Wildavsky, A. 1987. Speaking Truth to Power: The Art and Cra of Policy Analysis. New Brunswick, NJ: Transaction. Google Scholar Google Preview WorldCat COPAC Yanow, D. 2000. Conducting Interpretative Policy Analysis. Newbury Park, Calif.: Sage. Google Scholar Google Preview WorldCat COPAC