Qualitative Research Methods PDF

Qualitative Research Methods Goals, Tools, and Trade-offs 5 November 2024 Stigler Center Manuel Cabal 1 / 26  Agenda 1. Housekeeping Office hours: Fridays, 12:00-13:00, Wijnhaven 3.14A There may be changes in one or...

Qualitative Research Methods Goals, Tools, and Trade-offs 5 November 2024 Stigler Center Manuel Cabal 1 / 26  Agenda 1. Housekeeping Office hours: Fridays, 12:00-13:00, Wijnhaven 3.14A There may be changes in one or two sessions in the syllabus. I will let you know in advance. Quiz points: 50% is for participating (“mock” question) Quiz questions came from (modified) questions from last year’s final exam 2. Trade-offs in research design 3. Tools of (Positivist) Qualitative Research You studied this last semester. We review them to strengthen your grasp on these ideas. Stigler Center Manuel Cabal 2 2 / 26 Trade-offs in research design Brady and Collier. 2010. “Chapter 8: Critiques, Responses, and Trade-Offs: Drawing Together the Debate” Rohlfing. 2012. “Chapter 2: Case, Case Study, and Causation: Core Concepts and Fundamentals” Stigler Center Manuel Cabal 3 / 263  This part’s objectives: Understand better what (Positivist) Qualitative Research is good for, through the debate between KKV and BCS in the past three decades. ✓ How the analysis of one or a few cases contributes to the overarching goals of positivist research (i.e., shared with quantitative research). Specifically: descriptive and causal inference & concept formation (reconstruction). ✓ We cannot achieve everything with quantitative tools. Qualitative tools achieve specific goals that quantitative tools do not. Thus, a key part of research design is understanding the goal trade-offs inherent specific tools. Key terms ✓ Overarching goals, Intermediate goals, and Tools ✓ Set-relational causation: Sufficient causa, Necessary cause, Equifinality, Conjunctural causation, INUS conditions, and SUIN conditions. Stigler Center Manuel Cabal 4 4 / 26  To discuss methods, we must distinguish... Overarching goals (embedded in specific “methodologies”, per Schaffer). Shared by Positivist researchers: descriptive and causal inferences Important for Positivist Qualitative researchers: “Refining theory”, which can also contribute to the overarching goals of descriptive and causal inference. Intermediate goals (Goertz and Mahoney called them “immediate” goals). Different traditions choose different paths to pursue these overarching goals. Intermediate goals present trade-offs: “the pursuit of one particular objective may make it harder to achieve another”. In other words, I cannot achieve everything at the same time. Tools (Schaffer calls them “methods”, as opposed to methodologies): Practices and procedures for achieving intermediate goals. There are also trade-offs at the level of tools. No tool can achieve everything. Moreover, while a given tool may achieve a particular intermediate goal, it may make it difficult to achieve another one. Research design is about making “choices among potentially incompatible goals, or to Stigler Center Manuel Cabal 5 5 / 26 evaluate these trade-offs in light of alternative goals”.  Critiques and responses (1) Avoid “no variance” designs: KKV advised not to analyze just one case study (or just a few cases). You must focus on something that “varies”. Problem 1: This advice often gets in the way of doing relevant research. Why? You are limiting yourself to questions that can be addressed with statistical data. Problem 2: Some research practices can improve inference but not necessarily innovation. ✓ It’s not that by using quant methods you cannot innovative. It’s about practices. Sometimes, research questions come from methods, when it should be the other way around. ✓ The horse behind the car: Making questions that can be answered with a cross-national regression or with an experimental design. And skewing relevant questions that do not fit these methods. Stigler Center Manuel Cabal 6 6 / 26  Qualitative practices tend to be conductive to innovative questions and areas of research. ✓ For instance, identifying and studying deviant cases. ✓ KKV assumed that creativity and theoretical innovation are only the product of personal “genius” or “brilliance”. They are not. There is something dishonest about this perspective because a lot of innovations in quant political science and in economics come from qualitative research. ✓ My suggestion, let your question guide your method choice. And look for questions that are “puzzling”. Stigler Center Manuel Cabal 7 7 / 26 (2) Conceptualization and Measurement: KKV advised to focus your research on concepts that maximize measurement validity and reliability, and to avoid organizing data with typologies. Remember measurement validity and reliability (Halperin and Heath, p. 187, chapter 7: What is Data?): Validity Reliability Stigler Center Manuel Cabal 8 8 / 26  Problem 1: May turn into invitation to ignore questions involving important concepts. ✓ Again, the methods determining research agendas, and not the other way around. ✓ There are concepts that are very important normatively and theoretically, and yet are quite difficult to measure. For example, civil society, political culture, legitimacy, hegemony. Problem 2: Typologies, and more generally research on concept formation, can make important contributions to causal inference; for instance, by identifying causal heterogeneity. ✓ We will read a good example of how typologies contribute to causal inference in Session 7. Stigler Center Manuel Cabal 9 9 / 26 (3) Selection bias: KKV’s advise was to avoid selecting cases that do not represent the population (which is achieved by a large-enough sample chosen at random). Problem 1: Yes, a small number of cases selected in a purposive way (non-random selection) will not be representative of a population. But as we learned last week, there are other benefits that cannot be achieved with a large sample of cases. Representativeness is not the only possible (immediate) research goal. ✓ An important benefit achieved through qualitative methods is rigorous within-case analysis. Selection bias is not a concern since we have chosen a case that matters for a purpose other than representativeness. ✓ Other benefits (more on this in Levy’s article): hypothesis formation or theory refinement (with solid empirical research); concept formation through a few cases (session 7); analyzing cases that are actually comparable, without stretching their comparability through a large sample (e.g., countries within a historical region or subnational cases); Comparing cases that are not obviously “cases” (changing the unit of analysis, like historical regions across countries). Stigler Center Manuel Cabal 10 /10 26  Problem 2: A representative sample depends on how one defines the universe or population of cases For example, let’s say that I have a dataset of all countries, from 1800 to 2020. It does not need to be a random sample since in this case the sample equals the population. Thus, I have all the countries. However, I am making a series of assumptions that are debatable: ✓ Countries are perfectly comparable between each other, like countries in Western Europe and Africa. There are ways to model these differences to account for their effects on a DV in a regression, but it will always remain in a “black box” (i.e., undertheorized) exactly why these countries differ (if I don’t look inside them carefully). ✓ Different periods are comparable. Again, it can be modeled to account for time differences, but that will leave that undertheorized. Stigler Center Manuel Cabal 11 /11 26 (4) Models of Causation: KKV advise against using “deterministic” (set- relational) notions of causation on the grounds that the world is probabilistic. Problem 1: What about hypotheses and theories formulated in terms of set relations—How are we to evaluate them? These theories are often formulated, for example, in the context of comparing historical processes. Comparative case studies are good for analyzing these theories. Linear regression analysis is not very good at that. Stigler Center Manuel Cabal 12 /12 26  Set-relational causation The correct terminology is... Correlational VS Set-relational causation (Probabilistic vs Deterministic is a different distinction, which we do not revisit). We will not ask in the exam about the differences with correlational causation. That is, you do not need to distinguish it. What matters is that you understand the different forms of set-relational causation discussed in Rohlfing’s chapter. Last semester, you read about this in your textbook. Variety of set relations (i.e., set-relational causal relationships): ✓Sufficiency ✓Necessity ✓Necessity AND Sufficiency ✓Equifinality ✓Conjunctural causation ✓INUS conditions ✓SUIN conditions Stigler Center Manuel Cabal 13 /13 26 “Democratic peace” hypothesis (one version of it): Democracies do not go to war with other democracies. Dyad: Pair of countries X is sufficient condition of Y X is a necessary condition for Y X→Y X←Y Y: X: X: Y: Stigler Center Manuel Cabal 14 / 26 14  Equifinality: More than one path to the same X is a necessary and sufficient outcome. condition of Y Either X1 or X2 is individually sufficient for Y X Y X1 or X2 → Y Y: x1: x2: Stigler Center Manuel Cabal 15 /15 26 Conjunctural causation: Two conditions produce the outcome but only if they are together. X1 or X2 are together sufficient for Y: X1 and X2 → Y Stigler Center Manuel Cabal 16 /16 26 INUS conditions: When there is >1 causal paths to the same outcome. Equifinality meets Conjunctural causation. Each path is a conjunctural cause. The conditions in each conjunctural cause are insufficient but necessary for the conjuncture (IN-) The conjunctures are unnecessary but sufficient for the outcome (-US). Stigler Center Manuel Cabal 17 /17 26  INUS conditions are common in comparative-historical analysis (comparison of historical processes and sequences). We will not have this topic in this course... But I teach a course on this for third years (Comparative State Formation). Another example: Barrington Moore’s capitalist-democratic path in early modernity (book Social Origins of Democracy and Dictatorship) Bourgeois class Land-proprietor class France Strong Weak (perishes in revolution) Conjunction 1 United Kingdom Strong Allies with bourgeoisie Conjunction 2 United States Strong Allies with bourgeoisie Stigler Center Manuel Cabal 18 /18 26  SUIN conditions: When there are sequences of causes, this is a way to theorize them. Sufficient conditions for (subsequent) Necessary conditions. In the example from Barrington Moore’s capitalist-democratic path to early modernity, the bourgeoisie’s “political organization” is a sufficient cause of “strong bourgeois class”; in turn, the latter is a necessary condition of capitalist democracy. The same with “economic importance”. Bourgeois class Source of strength France Strong Political organization and Political Organization & Economic assertiveness Importance are each sufficient United Kingdom Strong Economic importance conditions for Bourgeoise Strength (which is a necessary condition for United States Strong Economic importance each Conjunction) Stigler Center Manuel Cabal 19 /19 26 Some tools from the Qualitative Research tradition Levy. 2008. “Case Studies: Types, Designs, and Logics of Inference” Stigler Center Manuel Cabal 20 / 26 20  This part’s objectives Remember what Qualitative Research tools can achieve Identify the main tools of Qualitative Research and their logics Key concepts ✓Case study objectives: ideographic, hypothesis generation, hypothesis testing, plausibility probe. ✓Selection criteria: crucial (most likely, least likely), deviant, extreme, variation ✓Comparison strategies: most similar, most different Stigler Center Manuel Cabal 21 /21 26  What is a case study? Distinguish from historical studies Historians embrace the richness of reality Social scientists focus on aspects of reality that relate to a theoretical discussion (or policy discussion, as there is scholarly research on policy). A case is often an instance of something else—“what is this a case of ?” Instance of a class of events, inside which we analyze in detail certain aspects to develop or test historical explanations that may be generalizable to other events. Not necessarily (Florida’s Panhandle, 2000) “Case” vs “observation” One or few cases... But many observations: pieces of data within the cases (like in the case study of Florida’s Panhandle) Case studies are not “narrative” studies: We can use many types of data within the case, including statistics... Remember that qualitative methods are not about “words” (vs numbers). Stigler Center Manuel Cabal 22 /22 26  Florida’s Panhandle case In 2000, George W. Bush won Florida (over Al Gore) by 537 votes. Florida gave Bush the presidential win in the Electoral College. Thus, Bush’s electoral mandate was considered weak by his adversaries. Moreover, Gore had overall more votes (as opposed to Electoral College votes). A scholar (John Lott) claimed that Bush’s victory in Florida was wider, but due to media manipulation, Bush lost as much as 10,000 votes. Lott estimated the number using past voting matters and regression analysis (a differences-in-differences design). Stigler Center Manuel Cabal 23 /23 26 The alleged manipulation: Some networks in the East coast hastily declared Gore a winner 10 minutes before polls closed in Florida’s Panhandle (northwest counties), which is an hour zone behind. As a result, people was discouraged to cast a vote given the election was supposedly settled. At stake: Bush’s victory was seen by many as “pure luck”, not decisive. Stigler Center Manuel Cabal 24 /24 26  Brady’s analysis 1. Assume flat turnout rate per minute: 1/72 of total 379k census voters (not even actually registered to vote) in these ten districts. Minus 20% of mail voting = 303k. 303k/ 72 = 4,200. Thus, Bush may have lost at most 4,200 votes (upper bound). However, interviews suggest that most people vote early, not in the last 10 min: the number is likely lower. 2. Additionally, polls indicate that at most 1/5 of adults watch tv news: 4,200*(1/5)=840. Accounting for this, the upper bound is not 4,200 but 840. 3. Of these, polls indicate that 2/3 tend to vote Republican in Panhandle: 840*(2/3)=560. 4. Past research suggests that at most 1/10 is affected. Many vote for reasons different than “the race is close”, such as self-expression and because other positions (like local ones) are in the ballot. Thus, (1/10)*560= 56, as an upper bound, would have been affected 5. Now, because Gore would also be affected by an early call, we do the same exercise for Gore. It yields 28 votes. Then, 28 to 56 votes lost for Bush. Stigler Center Manuel Cabal 25 /25 26 Typology by Objectives of Research Ideographic: Describe, explain, interpret, and/or understand a single case as an end in itself rather than as a vehicle for developing broader theoretical generalization. Like a historian’s case study. Inductive: Not structured by theoretical framework and concepts. Ex: A study describing what was going on the year WW1 started (1914), politically, economically, culturally, etc. Theory-guided: Explicitly structured by a well-developed conceptual framework that focuses attention on some theoretically specified aspects of reality and neglects others. Ex: A study on WW1 focusing on the chain of events happening in 1914 that led to a total war. Stigler Center Manuel Cabal 26 /26 26 Unlike ideographic case studies, others aim to generalize beyond the data (a broader scholarly discussion)... Hypothesis-generating: Purpose of proposing or refining a theory, for instance, through the specification of causal mechanisms. Ex: Arend Llijphart’s 1968 book The Politics of Accommodation. Examines how the Netherlands managed to achieve a stable and functional democracy despite deep social, religious, and ideological divides. In the early 20th century, when society was sharply divided into distinct "pillars" based on religion and ideology—Catholic, Protestant, Socialist, and Liberal communities. Each pillar had its own institutions, such as schools, media, labor unions, and political parties, creating a segmented society that rarely interacted across lines. Dutch leaders managed to create a stable democracy despite these divides, through a system that Lijphart calls "consociational democracy". This model is based on political elites working together to avoid conflict, often by making accommodations and compromises behind the scenes. Contradicts the conventional wisdom that liberal democracy does not work in contexts of deep societal divisions, at of decolonizations in Africa and Asia (when many democratic experiments failed). Stigler Center Manuel Cabal 27 /27 26  Hypothesis-testing: Some hypotheses can be tested through one or few cases, using process tracing, structured comparisons, crucial cases, etc. Plausibility probe: Exploratory case study that highlights elements that will be useful before engaging in a broader research endeavour. Provides initial propositions and questions. Often used along with formal modelling as illustrations. Beware of unsystematic cases that invoke this label to legitimize a poor job. Stigler Center Manuel Cabal 28 /28 26  Research Design: Case selection criteria Remember Dependent Variable = Phenomenon I want to explain Independent Variable = Explanatory factor “Causes of effects” kind of questions Purposeful sample: Studies with small number of cases are always nonrandom. We select with a purpose in mind. For example, strategies for exploration: Extreme: case shows clearly the values of interest in the IV or DV Variation: cases that illustrate/show relevant variation in IV or DV Stigler Center Manuel Cabal 29 /29 26  Why do we need a selection criteria or strategy? Selection bias risk Without a proper sense of scope (what kind of case is this?), it might over- or under-estimate theoretical claims or causal relationships Always useful to have a comparative design or even add “shadow cases” Ex: Democratic erosion in interwar Europe in Levitsky and Ziblatt’s How Democracies Die focuses on Germany and Italy as cases of democratic breakdown. Brings the Belgium and Finland as shadow cases. When is it not a problem? The broader conversation is clear: ✓ Falsify necessary causes (outcome present without IV) Ex: Revolutions will not occur in the absence of peasant revolts (Theda Skocpol)… What about the Iranian Revolution? ✓ Falsify sufficient causes (IV present but outcome absent) Ex: Democratic countries (dyads) never go to war with each other Finland-Allied Countries in WW2 (alliance with Germany); US-Chile in the 1970s Within-case analysis: Mechanisms Stigler Center Manuel Cabal 30 /30 26  Crucial case design Hardest and easiest cases for a hypothesis or theory... Most-likely case: If theory does not work here, not likely to be relevant. Least-likely case: If theory works here, it is likely relevant. Example: Development → Democracy ✓Identify priors: What makes me think the theory works/does not work in that case? ✓Define predictions: After a generation of sustained economic growth, pressures for democratization arise, and then free and fair elections are established. ✓Define empirical implications: GDP per capital growth, middle-class growth, civil society organizations, contentious politics, opposition parties.... Brazil Mexico Most likely (feeble autocracy) Least likely (robust autocracy) Stigler Center Manuel Cabal 31 /31 26  Deviant case design Example: Development → Democracy Anomalies of the theory India Case study asks: Why is it an anomaly? (Is it really?) Relatively poor country that democratized at independence, despite economic underdevelopment Refine the theory Singapore Case study asks: Under what (scope) conditions does the theory work? Rich country that has never (truly) democratized Stigler Center Manuel Cabal 32 / 26 32  Comparative designs Structured comparisons Most Similar Systems design (J.S. Mill’s method of difference) Most Different Systems design (J.S. Mill’s method of agreement) Context Hypothesized Outcome Source of leverage Condition Most Similar Different Different Since alternative conditions (context) are similar across Similar cases, they can’t explain outcome divergence Most Different Similar Similar Since alternative conditions (context) are different across Different cases, they can’t explain outcome convergence Analogous Control to... IVs IV DV Stigler Center Manuel Cabal 33 /33 26  Most Similar Systems design Marcus Kurtz. 2009. “The Social Foundations of Institutional Order: Reconsidering War and the "Resource Curse" in Third World State Building.” Politics and Society Most Similar initial conditions (alternative explanations) Colonial heritage Economic development Position in international division of labor (minerals) Exposed to regional geopolitical tensions (wars and invasions) Perú Different outcomes: Ineffective (Peru) // Effective taxation system (Chile) Explanation (Necessary condition): Hierarchical (Peru) // Non-Hierarchical (Chile) labor relations (inherited Chile from colonial times) Stigler Center Manuel Cabal 34 /34 26  Most Different Systems design Theda Skocpol. 1969. States and Social Revolutions Most Different initial conditions (alternative explanations) France Similar outcomes outcomes: Radical transformations of state and society in a relatively short period. Russia Explanation (INUS condition*): Inter-state rivalry → Liberal reforms → Peasant rebellion → Collapse of coercive apparatus → Radicals in power →Radical reforms China * Skocpol later identified alternative conjunctural conditions Stigler Center leading to Manuel Cabal 35 /35 26 social revolutions (in peripheral countries, like Mexico, Cuba, and Iran). Potential issues Comparability Cases do not need to be exactly similar (as if in an experimental comparison), but they must be ‘reasonably’ comparable. The author has the burden of showing that the cases are indeed comparable (Most Similar) or convergent (Most Different) in theoretically relevant ways. Complex causation When used for complex phenomena (e.g., macro-political processes), allow for “conjunctural causation” and “equifinality”....Here is where models of set-relational causation come in handy. Researchers are using more and more within-case analysis (process tracing) in combination with case comparisons. We will talk about process tracing in the next sessions. Stigler Center Manuel Cabal 36 /36 26