5 Slides Nature & Design Quantitative Research_UA.pptx

Quantitative Research Methods Prof. Dr. Bart Cambré Prof. Dr. Tim De Leeuw Antwerp Management School Outline 1. Quant research: research design 2. Data collection 3. Sampling 4. About concepts and measurement 2| Data Quality in Quantitative Research Data-quality = broad concept Refers main...

Quantitative Research Methods Prof. Dr. Bart Cambré Prof. Dr. Tim De Leeuw Antwerp Management School Outline 1. Quant research: research design 2. Data collection 3. Sampling 4. About concepts and measurement 2| Data Quality in Quantitative Research Data-quality = broad concept Refers mainly to research design and to measurable quality of data (% no opinion, differentiation in response patterns, item nonresponse…) 1. Quantitative Research: Linking to the research design (aka the Turtle) Epistemology and Ontology: continuum Ontology: what can we know Objectivism Constructionism Epistemology: how can we know Positivism Interpretivism Theory in relation to research Deductive Inductive Methodology: how can we find out Quantitative Qualitative Deductive and inductive approach - Deductive approach - Inductive approach - Quantitative methods - Qualitative methods Observations Theory /Findings Observations Theory /Findings Deductive and inductive approach Quantitative methods Qualitative methods -Data as numbers -Data in the form of words -From instruments (e.g., surveys) -Data from the field -Statistics and methods -Interviews, documents, observations, audio-visual materials -Researcher is the instrument -Emergent design -Data Themes Research cycle Deductive approach process: 2. 3. Data 1. Theory Hypothesis collection 5. 6. Revision Hypothesis 4. Findings of theory confirmed? Research designs  Experimental design  Cross-sectional design  Longitudinal design  Case study design  Comparative  Design science Main preoccupations of quantitative research Measurement Causality Generalization Replication  As always, reliability and validity Criticisms of quantitative research Quantitative researchers fail to distinguish people and social institutions from `the world of nature' The measurement process possesses an artificial and spurious sense of precision and accuracy The reliance on instruments and procedures hinders the connection between research and everyday life The analysis of relationships between variables creates a static view of social life that is independent of people's lives Quantitative measures: pitfalls Goodhart's Law : “When a measure becomes a target, it ceases to be a good measure” Campbell’s law: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.” E.g., high-stakes testing; teaching to the test 12 | Sources of error 1. Sampling (-related) error 1. Sampling error: difference between sampling estimate and population parameter 2. Sampling related error: size, selection, … 2. Data collection error 3. Data processing error Variation and Error variation = true variation + variation due to error A variable A variable with little error True variation Variation due to error A variable with considerable error True variation Variation due to error Measurement error Measurement error: = how far is the measured variable free of measurement error (ME) Measurement invalidity true value of the construct of Measure does not capture concept interest Systematic error/bias + systematic error (validity issue) Measurement unreliability: Inconsistency under repeated uses + random error (reliability issue) Random measurement error = total value of the construct 15 Measurement error (ME) & validity At empirical level: distinction between - the variable we want to measure (T) and - the observed variable (X) i.e. the measure of T Applies to both: multiple indicators for latent variables and to single indicators X=T + I + e observed value true value invalidity random error systematic error 16 Sources of measurement error (ME) I- term: invalidity or systematic error Sources of (systematic) ME: - response scales (different scale points) - wording of items - order of response categories - context of question - response styles… In cross-national research: - Natural language differences - Unjustified differences in the translations - Differences in interpretation of the questions in different countries 17 | Quality Indicators in Quantitative Business Research Reliability – are measures consistent? Replication/ replicability – is study repeatable? Measurement (or construct) validity – do measures reflect concepts? Validity – are conclusions well- Internal validity – are causal relations between variables real? founded? External validity – can results be generalized beyond the research setting? Ecological validity – are findings applicable Types of Reliability Stability → is the measure stable over time? e.g. test–retest method Internal reliability → are the indicators consistent? e.g. split-half method Inter-observer consistency → is the measure consistent between observers? External reliability → replicability Types of validity Face validity: intuitive process Construct validity (or measurement validity): extent to which a measure performs according to theoretical expectations Internal validity: refers to causality; is X really causing Y; linked to confounding variable bias External validity: generalization Ecological validity: are research findings applicable? “Falsehood flies, and truth comes limping after it” (Swift, 1710) “An idiot can create more bullshit than you could ever hope to refute” (Fanelli) Brandolini’s principle (2014): “The amount of energy needed to refute bullshit is an order of magnitude bigger than that needed to produce it.” Example: falsehood that vaccines cause autism Andrew Wakefield (1998) made this claim in The Lancet Wakefield was found guilty of professional misconduct, lost his license, paper was fully retracted by The Lancet, … But until today, the false conclusions can be found everywhere Principles (Bergstrom, C.T. & West, J.D., 2020 ) on data bullshit : 1. Bullshit takes less work to create than to clean up 2. Bullshit takes less intelligence to create than to clean up 3. Bullshit spreads faster than effort to clean it up 21 | 2. Data Collection Data collection Primary data: Collected by the researcher conducting the research Sources:  Interviews (e.g., managers)  (Structured) observations  Survey(s)  E-mail/communication/LinkedIn/network data  Experimental outcomes  Focus group data  Instrument (readings) data Data collection Secondary data: Collected by someone other than the researcher Sources:  Content analyses of (archival) texts (e.g., meeting minutes)  Academic papers  Secondary databases (e.g., stock market data) Level of measurement Remember statistics … Nominal Ordinal Metric (interval/ratio)  important when collecting data e.g. developing questionnaires  measure at highest possible level ! Level of measurement Different types of data: 3. Sampling Sampling Draw a sample (e.g., random; convenience) Can the answers of the sample be used to describe the population? How precise is the data? Use formula for confidence interval) Sampling Sampling: no error Male Female O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O N=50 O O O O O O O O O O O O O O O O O O O O Sampling Sampling: some error Male Female O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O N=50 Sampling Sampling: lot of error Male Female O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O N=50 Categorization of sampling techniques (Saunders et al., 2012) Types of Probability Sample Simple random sample Selection by equal probability of inclusion Systematic sample Selection from sample frame at a constant interval Stratified sample Proportional representation of population characteristics Multi-stage cluster sample Selection from groupings of population units Types of Non-Probability Sample Convenience sampling By accessibility Snowball sampling By contact Quota sampling By proportion, but not at random Limits to Generalization › Findings can only be generalized to the population from which the sample was taken › Findings may be specific to the characteristics of the population › Findings may be locality specific › Findings may be temporally specific Methodological considerations 1. Sample Size No basic rules, but … (online calculator) Absolute size more important than relative size: size does matter! Time and cost Non-response: random? Heterogeneity of the population Kind of analysis to be conducted Strategic considerations: all employees? ‘why me?’ Depends on problem statement: -Exploratory research: until saturation -Explanatory research: until sufficient explanation of hypotheses Methodological considerations 2. Self selection BIAS?! Compare respondents with population data 37 Methodological considerations 3. Non-response BIAS?!  Compare respondents with non-respondents o e.g., call non-respondents, ask key questions, compare  Comparing early with late respondents 38 Methodological considerations 4. Response rate  On average for externals: between 3-8%  Compare your response rate within your field of study  Within own organization much higher (e.g., > 50%)  From respondents and response rate to selection: o If e.g., 400 respondents are needed o Response rate would be estimated to be 25% o Sample should al least be = 1600 39 Statistics ‘In case of doubt, visit a statistician’ › Field, A. (2009). Discovering Statistics Using SPSS. Sage. › https://www.discoveringstatistics.com/ › https://www.datacamp.com/tracks/statistics-with-r › “DataCamp is the first and foremost leader in Data Science Education, offering skill- based training, pioneering technical innovation, and courses from the world's best educators” › www.khanacademy.org › “Our mission is to provide a free, world-class education to anyone, anywhere.” › https://eu.qualtrics.com/ControlPanel/ This is the end …