Module II - Research Design | Technische Universität München | PDF

Course structure i. Foundations ii. The empirical research process 1. Research question, theory, and hypotheses 2. Research design 3. Sampling and measurement 4. Data collection 5. Hypotheses testing 6. Dissemination iii. Research ethics...

Course structure i. Foundations ii. The empirical research process 1. Research question, theory, and hypotheses 2. Research design 3. Sampling and measurement 4. Data collection 5. Hypotheses testing 6. Dissemination iii. Research ethics Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 1 Learning goals of this module  Learning goals  … get acquainted with the concept of causality  … distinguish different types of experiments  … compare different types of experiments and quasi-experiments  Readings – Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002), Experimental and Quasi-Experimental Desings for Generalized Causal Inference, Houghton Mifflin, Boston – Trochim, W., Donnelly, J. P., & Arora, K. (2020). Research Methods: The Essential Knowledge Base. Conjoint. ly: Pyrmont, Australia. Chapters 8, 9, 10 Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 2  Technische Universität München Cause-effect relationships Criteria for establishing a cause-effect relationship Causality Covariation Temporal No rival of cause and precedence explanation effect  Internal validity: extent to which the evidence permits inference regarding the cause-effect relationship(s) of interest Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 4 Causality: covariation of cause and effect Happiness e.g., 1000 US$ additional salary is linked to an increase in happiness, on average Money Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 5 Causality: temporal precedence Cyclical Functions Money Happiness Time Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 6 Causality: no rival explanation 700 600 Messina 500 Ice cream sold (kg) Taormina Syracuse Sciacca 400 Licata Catania Gela 300 Modica Palermo 200 Napoli 100 0 100 120 140 160 180 200 220 240 260 Sunglasses sold (pieces) Y = -47.971 + 2.591x t = 4.64, p = 0.001 Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 7 Primary threats to internal validity  Confounders  Time trends  Pre-existing heterogeneity  Selection Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 8 Confounders Subject-related Situation-related + Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 9 Strategies to mitigate internal validity threats  Control group: One group of subjects receives a treatment (“treatment group”) and the other group does not (“control group”)  Random assignment: The sample is assigned into treatment and control groups by chance  Probabilistic equivalence Treatment group Control group Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 10 Overview of research designs Is random assignment used? Yes No Is it conducted in Is there a control a laboratory group? environment? Yes No Yes No Laboratory Quasi- Nonexperimental Field experiment experiment experiment design High internal validity Low internal validity Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 11 Terminology  Conditions or treatments: variables that are manipulated by researchers  Responses / outcomes / dependent variable: variable that we are interested in  Assignment method: procedure that determines which experimental unit receives which treatment Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 12  Technische Universität München Experimental designs Experimental research designs  Experiments are research designs, in which – … one or more variables are actively manipulated (treatment), –... subjects are randomly assigned into treatment and control groups, –... the change in an outcome variable is measured Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 14 Example of a laboratory experiment Proposition: Money + Happiness Random assignment of 100 individuals into 2 groups, receive payment in the laboratory by After one year, measure happiness researchers Control group Control group 50 individuals: Happiness = 5.6 No treatment Treatment group Treatment group 50 individuals: Receive EUR Happiness = 7.3 20,000 per year (lacobucci and Churchill (2010), p. 105) Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 15 Field versus Lab experiment Laboratory experiment Field experiment Advantages Time and cost High degree of authenticity High control over potential increases the possibility of confounding variables (high making generalizations about internal validity) experimental findings (higher Easy to replicate external validity) High level of control over Little bias caused by awareness manipulation of participation Disadvantages Little authenticity (low external Can be ethically problematic validity) Difficult to replicate Subjects are aware of taking Manipulation of independent part in an experiment variables often specific to the setting Access to organizations difficult (Homburg 2015, p. 282) Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 16 Field experiments in practice: A/B Testing Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 17 Manipulation  Important 1. Realism 2. Unobtrusiveness 3. Simplicity 4. Research ethics (see final module)  Experimental manipulation as a three-stages process Treatment Treatment Treatment delivery receipt adherence (Shadish et al. 2002, pp. 316)  Key issue: manipulation effectiveness  Manipulation checks  Realism checks  Confound checks Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 18 Experimental treatment levels  An experiment with one treatment and one control group may not tell everything we would like to know  We can use different treatments or levels of treatment  Example: Money + Happiness Control Group: 50 individuals: No treatment Treatment Group A: 50 individuals: Receive EUR 20,000 Treatment Group B: 50 individuals: Receive EUR 10,000 Treatment Group C: 50 individuals: Receive EUR 5,000 Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 19 Factorial notation  Notation: L1xL2x…xLn  Factors: several independent variables (each with at least two attribute levels) [n]  Level: different values that the independent variable n takes [L]  Cell: each combination of the factor levels corresponds to one experimental condition  Example: 2x2 experiment – A: Age (young, old) – B: Treatment (control, money) Control Money Young XA1B1 XA1B2 Old XA2B1 XA2B2 Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 20 Enhance internal validity through protocols  How was the independent variable manipulated?  How were participants assigned to groups?  What exactly were participants requested to do?  How was the independent variable measured?  How was the dependent variable assessed?  … Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 21 Specific threats to internal validity in experiments  Failed random assignment  Different drop-out rates  Extraneous events  Experimenter expectancy  Hawthorne effect Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 22  TUM School of Management Technische Universität München Question to you  A problem in experiments is participant drop-out. What could we do to minimize the drop-out of participants?  Minimize the amount of time between agreement to participate, manipulation, and measurement  Provide full details over all steps/parts of the experiment  Get consent of participants to participate in all possible experimental conditions  If drop-out occurs: compare drop-outs to other participants Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 23 Control group design: between-subject vs within subject Between-subjects design Independent Variable Dependent Variable Receive Happiness money DIFFERENCE Do not receive Happiness money Within-subjects design Independent Dependent Independent Dependent Variable Variable Variable Variable Do not receive Receive Happiness Happiness money money DIFFERENCE Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 24  TUM School of Management Technische Universität München Question to you  What are advantages and disadvantages of within-subjects designs?  Advantages – Fewer participants necessary (less costly) – No need to balance individual differences across group – Variability due to individual differences is eliminated  Disadvantages – No control group / solid benchmark: – All the general internal validity threats might create issues for inference – Bias due to previous conditions affecting outcome (carry-over effect), direct or indirect (e.g., fatigue, expectancy) – Time-related confounders Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 25  Technische Universität München Quasi-experimental designs Quasi-experimental designs  One group of subjects is exogenously treated by an event outside their control and that could not be foreseen, whereas another group of subjects is not affected by this event  No random assignment by researcher, no manipulation by the researcher  Examples of events – Policy changes – Court rulings – Natural causes (”natural quasi experiment”) Natural disasters, “acts of god” Lotteries, chance events Geographic variation – Discontinuities Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 27  TUM School of Management Technische Universität München Question to you  Why are quasi-experiments an important research method for management and economics research?  Lab experiments are in many cases not feasible: we cannot put entire companies into a laboratory  Active manipulation would be unethical / unfeasible  Tend to be less expensive because no manipulation needs to be carried out Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 28 Types of quasi-experimental designs  Difference-in-differences design (DID)  Instrumental variable design (IV)  Regression discontinuity design (RDD) Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 29 Difference-in-differences design  Difference in differences requires data about a treatment group and a control group at least one time period before “treatment” and at least one time period after “treatment”  Group assignment can be based on various strategies, for example: – A new law affects firms in one country but not in another country – A new regulation imposes restrictions for firms in one industry but not in others – A political change affects residents in one city but not in another city  One of the most applied quasi-experimental designs in business and economics Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 30 Difference-in-differences design Outcome 90 80 Ȳtreatment,after 70 60 ß1 diff-in-diff 50 40 Ȳtreatment,before 30 Ȳcontrol,after 20 Ȳcontrol,before 10 0 t=1 t=2 Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 31 DID: Card and Krueger (1994) example Minimum wage + Employment  Policy Change: in April 1992 – Minimum wage in New Jersey from $4.25 to $5.05 – Minimum wage in Pennsylvania constant at $4.25  Research Design: – Collecting the data on employment at 400 fast food restaurants in NJ (treatment group) in Feb. 1992 (before treatment) and again November 1992 (after treatment). – Also collecting the data from the same type of restaurants in eastern Pennsylvania (PA) as control group where the minimum wage stayed at $4.25 throughout this period Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 32 DID: Card and Krueger (1994) example Notes: Adapted from Card and Krueger (2000), Figure 1 Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 33 DID: Card and Krueger (1994) example Variable PA NJ Difference NJ-PA (i) (ii) (iii) 1. FTE employment 23.33 20.44 -2.89 before, all available (1.35) (0.51) (1.44) observations 2. FTE employment 21.17 21.03 -0.14 after, all available (0.94) (0.52) (1.07) observations 3. Change in mean -2.16 0.59 2.76 FTE employment (1.25) (0.54) (1.36) Notes: Adapted from Card and Krueger (1994), Table 3 Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 34 DID assumptions  Key assumption: „parallel trends“ before the treatment – Diff-in-diff does not identify the treatment effect if treatment and comparison groups were on different trajectories prior to the program Parallel trends No parallel trends Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 35 Instrumental variable (IV) design: overview Instrumental variable Z X Y Unobserved confounders Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 36 IV: example of Angrist and Krueger (1991) Education (years + Earnings of schooling) Unobserved confounders Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 37 IV: example of Angrist and Krueger (1991) Start School Born Turn 6 S Dec S Born Start Turn 6 16 Jan School 14 Years Completed Education 4 2 3 2 1 3 4 34 4 2 34 2 3 2 1 4 4 1 4 1 13.5 1 3 12 3 23 2 4 3 4 2 1 3 1 12 13 40 42 44 46 48 50 Year of Birth Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 38 IV: example of Angrist and Krueger (1991) 34 3 34 34 34 34 5.9 3412 34 234 2 4 2 12 2 12 2 1 4 1 41 2 1 12 34 234 1 13 13 12 1 123 2 42 1 34 Log Weekly Earnings 23 5.8 1 41 3 24 12 3 4 2 5.7 13 4 1 2 34 5.6 30 32 34 36 38 40 42 44 46 48 50 Year of Birth Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 39  Technische Universität München See exercise session “Research design” Questions Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 41