Public Law 2 Lecture 3 Slides - Data, Indicators & Rankings PDF

Vevox Session ID: 183-525- 593 Public Law 2 Lecture 3 – Data, Indicators and Rankings Dr Clare Williams Thursday 30th January 2025; WLT Quick recap: where are we? Lectures 1 & 2: Sovereignty beyond the State The Regula...

Vevox Session ID: 183-525- 593 Public Law 2 Lecture 3 – Data, Indicators and Rankings Dr Clare Williams Thursday 30th January 2025; WLT Quick recap: where are we? Lectures 1 & 2: Sovereignty beyond the State The Regulatory State Global governance by networks The erosion or transformation of sovereignty Neoliberalism and globalization Understanding these changes historically Understanding these changes conceptually (Foucault’s governmentality lens) Lecture 3: Governance through data, indicators and rankings Rise in quantification and commodification Data as constructing epistemic categories and presenting truth claims (shaping how we are able to understand the world) Lecture 4: Governance through data and surveillance In this lecture, we will… Establish how data operates as a neoliberal technology of governance Establish how data facilitates quantification, mathematization and comparison Establish a definition of indicators and look at some of their characteristics Identify how indicators function Look at two examples: The World Bank’s Business Enabling Environment (previously Doing Business) University league tables and rankings Some terms and definitions: Qualitative: data that is non-numerical (words, descriptions, etc) Quantitative: numerical data (numbers, For example: indicators, etc) In this lecture, we Commodification: a process by which something will be exploring is transformed into a commodity (something that how empirical data can be bought and sold in a marketplace) sets operate to Empirical: based on observation or experience quantify and (rather than purely theoretical) commodify social phenomena, Epistemic: relating to knowledge or to the degree transforming the of its validation world into distinct Commensuration: a process that entails turning epistemic qualities into quantities that share the same metric categories. (Espeland & Saunder, pp.91-2) 3 key take aways: 1. Quantification is reductive Nuance is lost Quantification produces epistemic categories 2. “What gets counted gets done” Indicators drive value Indicators drive resource allocation 3. As soon as you start keeping score, everybody wants to “win” Indicators and rankings can be “gamed” They alter the priorities of those being measured Data as a neoliberal technology of governance Data can inform/persuade Data quantifies and enables the comparison of (social) phenomena Data enables the commodification of social processes and objects Data presents quantified “facts” about the world as objective and scientific truth claims Data constructs epistemic categories and shapes how we understand the world Data can have a performative effect (BSM pricing model at the Chicago Board of Options Exchange experiment) What do we mean by “data”? Information collected, processed, analysed and retained By governments or agencies (public sector) By companies (private sector) (By the third sector) Not a new phenomenon: see the birth of statistics in the 1800s “in the late 20th century it led to the replacement of traditional forms of government through centralised and rule-governed processes by ‘goal-governed steering of outputs and outcomes, accompanied by the monitoring of targets’ (Ozga, 2009: 149).” [Soutero-Otero & Beneito-Montagut, 15] Recall the shift from government to governance, and the emergence of what Foucault calls governmentality Data as a tool of regulation Data can educate, inform, and persuade Soft power to influence behaviour “Docile bodies” (Foucault) that engage in self-regulation Statistics as a tool of processing and analysing data to inform policy making in the modern state Data sets and statistics (their analysis) as creating epistemic categories within society (what we are able to think about and value) Data can inform and/or persuade: Data helped the government consolidate and co mmunicate its authority during the pandemic. Governing through data: soft power Data collection, analysis, and presentation comprises one of the “soft power” tools available to states and other actors in a globalized world Recall that traditional governance mechanisms tend to be jurisdiction-specific Recall Slaughter’s New World Order An example: The Financial Action Task Force (FATF) which sets and monitors the rules of money laundering: Sets out “recommendations” for states but these apply also to non-members States that fail to comply can be “grey-listed” (effectively, financial sanctions) What are indicators? (1) Indicators, numerical and organisational ranking systems, are a calculative technique of New Managerialism. A purposive tool to change or influence how organisations Indicators are “a behave and shape how they make decisions. named collection of As Davis et al put it: “The name [of an indicator]…embodies a rank-ordered data presumption that it represents the phenomenon [it measures that purports to and captures] or that it has the power to produce represent the past or [objective or even “neutral”] knowledge about it” projected Indicators simplify, decontextualise and reduce to performance of selected data the complexity and nuance of the thing or different units.” phenomena they purport to mirror and rank (See Espeland, W. N., & Sauder, M., 2007). Davis et al (2012) Indicators are purposefully comparative and evaluative. Governance by Their purpose is to establish comparative hierarchies based Indicators on the relational position of units of different rank and value. They are “are placeholders for…“ideology”…[or] markers for larger policy ideas” (Davis, K., et al, 2012). What are indicators? (2) “Indicators represent a distinctive method of producing knowledge about societies” (Davis, Kingsbury & Merry, 2010) “they tacitly embody theories about both the appropriate standards against which to measure societies (or institutions) and the appropriate ways in which to measure compliance with those standards” (Davis, Kingsbury & Merry, 2010) Indicators are standard setting instruments that also evaluate Four key characteristics: (1) the significance of the name of the indicator and the associated assertion of its power to define and represent a phenomenon such as “the rule of law”, (2) the ordinal structure enabling comparison and ranking and exerting pressure for ‘improvement’ as measured by the indicator, (3) the simplification of complex social phenomena, and (4) the potential to be used for evaluative purposes Indicators as calculative techniques Indicators as “calculative techniques” (Espeland, W. N., & Sauder, M. (2007) Rankings and reactivity: How public measures recreate). They offer “quantitative accountability,” based on measurable and calculable outputs, such as performance indicators, league tables, targets, and rankings. Indicators are a tool of transparency, enabling detailed auditing and accountability in almost every walk of life. In this understanding of accountability, numbers give calculative techniques a form of legitimacy, reliability, stability, authority, and unassailability (see Lynch, K., Grummell, B. and Devine, D. 2012. New Managerialism: Screen grab from World Bank website of Commercialisation, Carelessness and Gender. Basingstoke, some of the latest data science tools for UK. Palgrave Macmillan). measuring legal systems. Image links to the resource. Recall the scientification of discourse and growth in expertise that characterise the regulatory state, and note that indicators and rankings embody the key trends, Characteristics of indicators Reductive Simplify and decontextualize data: numbers “supplant local cultures with rational methods” [De Souza (2022)] Evaluative Quantify and compare (the “what?” (conceptual) and the “why?” (normative) tools of neoliberal approaches) Establish comparative hierarchies based on the relational position of units of different rank and value Appear scientific and objective, but look more closely! Where are the value-decisions? (Look at the decisions about what is counted, when, how, by whom, and how the data comprising the indicators is analysed and presented). What are the assumptions that underpin those indicators? (What is the world view of the person doing the counting?) But data can be partial Data is rhetoric (or “humanly persuadable”) (McCloskey, 2007) Think of large data sets such as: UN Global Database on Violence Against Women OECD Violence against women How comparable are these data? What factors might influence the data available? Remember: what gets counted get done (and by extension, what is not counted tends to be ignored or under-valued) Data can be incomplete “In Ecuador, specialised criminal courts for violence against women (henceforth VAW) were established in 2013, VAW misdemeanours became criminal offences Recall data sets on violence against in 2014, and a new specialised law on VAW passed in 2018. Yet, the 2019 national survey on VAW shows an increase women: of the phenomenon compared to the rates Do each of these states collect the same of 2012: 64% of Ecuadorian women have experienced some form of gender-based data in the same way (using the same violence throughout their lives, with provinces where incidence is as high as method)? 79% (INEC, 2019). Around 36% of violent Do some states use proxies? deaths between 2014 and 2019 were femicides, mostly perpetrated by an Is there missing data in the dataset? intimate partner (Tupiza Aldaz & Guerra Páez, 2019). However, my empirical Partial or incomplete data sets can give research has shown that only 11.1% of all cases heard by the specialised courts partial or incomplete pictures of a between 2014 and 2019 resulted in a conviction. Almost 56% of the cases never particular phenomenon reach a resolution (see Chapter 6). Still, Note that data sets tend to have a Ecuador scored 100/100 in legally protecting women from violence, in a veneer of objectivity and scientific 2018 World Bank report (Iqbal, 2018).” validation (they represent truth Silvana Tapia Tapia (2022) “Feminisim, Violence Against Women, and Law Case study 1: Quantifying and comparing legal systems Why count legal systems? Belief in an “ideal investment climate” – what is best for business? (Perry-Kessaris, 2011) Application of market-based technologies of governance (quantification, mathematization, competition) to a global “marketplace” of legal systems But how can we count a legal system? La Porta, Lopez, Schleifer and Vishny (“LLSV”: series of papers in late-1990s and 2000s) Legal Origins literature Belief that certain types of legal systems promoted economic development Pistor & Wellons (1998): legal institutions are important for economic development in both allocative and procedural dimensions Acemoglu et al: “Colonial Origins of Comparative Development” Used proxies (mortality rates of colonial settlers) to measure strength of legal institutions But… is there really an “ideal investment climate”? Perry-Kessaris (2004): do investors even care about the legal system? But how can we count a legal ASSUMPTION 1: There is an “ideal system? investment climate” 2: ASSUMPTION Identify variables (what is important)? (E.g., number of processes required to start aForeign business) investors How do we know it is important before we count + compare? 1 are looking for the best legal Identify quantifiable aspects of each variable (Are all processes obvious? Do some run in parallel?) system How do we know that these aspect reflect the phenomena? ASSUMPTION 3: 2 Legal reforms will Score each variable according to predetermined ranking increase inward 3 How can we determine whether a legal rule is 3/5 or 4/5? investment ASSUMPTION 4: Inward Combine scores and rank according to total investment will How do we know that total score reflects overall experience of legal rule IRL? 4 boost the economy Use rankings to advise countries on how to “improve” their legal systems so that they can attract more inward investment (Does “improve” = “Westernize”? Is this what investors want? Does inward 5 investment boost economic growth?) But remember: correlation is not the same as causation! Examples of indicators: Business Enabling Environment (World Bank) Imagine: you are a business-owner and want to set up an overseas subsidiary or franchise. How can you find out which country might be the best for you to invest in? What are the local legal processes you need to know to start up a business there? WB Indicators can offer quick and effective summaries of jurisdictions, if presented transparently. Examples of indicators: Business Enabling Environment (World Bank) But wait! The World Bank’s Business Enabling Environment was previously called Doing Business. It was closed down amid a bullying scandal. It compiled annual rankings of countries around the world, quantifying and comparing their scores for indicators such as the rule of law, number of processes required to start a business, and time taken to start a business These indicators are based on the assumption that there is an “ideal investment climate”: an ideal set of legal rules that will persuade international investors to invest in that particular country (Perry-Kessaris, 2009 and 2011). Case study 2: University league tables  What happens when we quantify, compare, and rank higher education? Vevox poll: 183-525- 593 Examples of indicators: University league tables  University League tables are indicators, organisational ranking systems based on comparative numerical data and values, that purport to accurately capture and measure, and objectively present the relative quality and value of education as a commercial service to the student consumer segment.  They simplify, de-contextualise, standardise, and flatten the complexity and particularities of what they are named after (universities).  Their representations are based on contingent normative decisions. Choices have to be made in terms of what to include or exclude, and how to weight, or give relative value and significance, to what is included. University league tables: how do they work?  By focusing on or placing more weight on certain things, university league tables “The rankings do not influence and shape the decisions of particularly capture University administrators in ways that align what a lot of schools offer. Things like with what is being measured and included. commitments to  League tables can incentivise ranked diversity and missions are not taken into entities to play “the numbers game”. account. These are  The numbers game encourages universities qualitative factors, and to focus on what is “most manipulable” and if someone looks at the rankings they are just small variations or changes to improve going to knock out your league performance. school without even a  The numbers game may encourage grade cursory review, which is a shame. That’s what inflation. irritates me. It pressures  The numbers game restricts organisational you to play the numbers game…” choice and flexibility. University league tables: how do they work?  Compare the ranking of the University of Kent this across the following League tables. What position is the University in? How does this vary? On Vevox, rank the different league tables according to where Kent is placed overall.  The Guardian University Guide  The Complete University Guide  Times Higher Education Best University Guide  Now navigate to the following methodology pages for two prominent, online university Vevox poll: 183-525- league tables: 593  The Complete University Guide methodology  The Guardian University Guide methodology Thought experiment: you are establishing your own (new) league table ranking universities… What would you measure? What would you do differently? Number of students Are there other factors that you Ages would measure? Some of their characteristics Why do you think these are important? What else can we measure? How would you measure them? Time in the classroom (objective) What problems might arise? Number of hours studying (subjective) Number of modules (objective) Note that some of these variables are much easier to What other information might be relevant? measure than others. But Student satisfaction (subjective) each of these will involve you Equality and inclusion (both objective and subjective) making a value judgment at some point (deciding where to Quality of education (trickier – how can we measure this?) draw the line around what you are counting, and how). Measuring the quality of education (OfS) How can we measure the quality of education? Ask independent external experts? Ask academics (the instructors)? Ask students? Break it down into sub- indicators Academic support Learning resources The teaching on my course Assessment and feedback Student voice University league tables as rankings: Note which indicators record objective data, and which ask for a subjective assessment of students’ experiences (this can only give a snapshot at a particular moment). Note that natural biases can come into play (MacNell et al., 2014). What is not measured? What impact do other factors/circumstances have? Why do you think quality has improved over the last year? What other factors might have influenced this, and what does this tell us about the overall ranking? What’s the effect of university rankings? “Gaming” the rankings? Missing important factors that are not “counted”? Can the university justify putting resources into aspects of education that are not captured in the rankings? (such as the quality or price of the food on campus, quality of accommodation, etc) Who chose this list of indicators? Why did they think these were the most important? Can this approach give us a nuanced or personal view? What sort of incentives does this set for (a) students, (b) universities and (c) the regulator? Indicators: risk-based approach to regulation Risk-based approach to industry regulation through data analysis. Emphasis on transparency and accountability (characteristics of modern regulatory governance). Application of market principles to ensure “value for money”. Students as consumers (what effect does this have on values that are prioritized?) The commodification of education: what happens when a social good is transformed into a commodity (something that can be bought and sold)? Does it shift our focus or appreciation of what is important and what is not? Remember 3 key take aways: 1. Quantification is reductive Nuance is lost Quantification produces epistemic categories 2. “What gets counted gets done” Indicators drive value Indicators drive resource allocation 3. As soon as you start keeping score, everybody wants to “win” Indicators and rankings can be “gamed” They alter the priorities of those being measured (remember the importance of incentives) What’s next? Next week: Lecture 4 – Regulation by Data and Surveillance The following week: Seminar 2 – Technologies of governance: “Do university league tables mean anything?” Remember: Assessment 1 deadline - Friday 7th February