Sampling and Measurement Module

Course structure i. Foundations ii. The empirical research process 1. Research question, theory, and hypotheses 2. Research design 3. Sampling and measurement 4. Data collection 5. Hypotheses testing 6. Dissemination iii. Research ethics...

Course structure i. Foundations ii. The empirical research process 1. Research question, theory, and hypotheses 2. Research design 3. Sampling and measurement 4. Data collection 5. Hypotheses testing 6. Dissemination iii. Research ethics Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 1 Learning goals of this module  Learning goals – … get to know the different data types and sources – … understand the concept of sampling – … learn what we mean by operationalization, measurement, level of measurement/ scale, and variable – … understand measurement error and assess the reliability and validity of a measurement – … select the suitable sampling method – … find measures for constructs  Readings – Trochim, W., Donnelly, J. P., & Arora, K. (2020). Research Methods: The Essential Knowledge Base. Conjoint. ly: Pyrmont, Australia. Chapters 4, 5, 6 Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 2  Technische Universität München Sampling Terminology: population and sample Theoretical Population Accessible Representativeness population and External validity Sample Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 4 Sampling strategies Population of interest Census Sampling Probability sampling Non-probability sampling Simple Stratified Cluster Convenience Quota Snowball Judgement Theoretical random sampling sampling sampling sampling sampling sampling sampling sampling (based on lacobucci and Churchill 2010) Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 5 Probability sampling: Random sampling  (Simple) random sampling – Each member of the population has an equal probability of being selected – The use of random numbers applied to a list of the entire population, subsequent lottery-like number draw Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 6 Probability sampling: Stratified sampling OBJECTIVE: Divide the population into nonoverlapping groups (strata) N1, N2, N3,…. Ni, such that N1+N2+N3+…+Ni = N. Then do a simple random sample of f = n/N in each strata.  Assures that you will be able to represent not only the overall population, but also key subgroups of the population Random Population Stratified Population Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 7 Probability sampling: Cluster sampling 1. Divide population into clusters (usually along geographic boundaries) 2. Randomly sample clusters – Natural, heterogeneous clustering of population – Common cluster variables: geographical area, buildings, schools, etc. – Can be more economic Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 8 Non-probability sampling: Convenience sampling  Convenience sampling – The respondents are selected, in part or in whole, at the convenience of the researcher (no or little effort to achieve representativeness) – Inference is difficult – Can provide useful information, especially in a pilot study – Inexpensive, easy to execute – Used for exploratory research Pollster Population Sample Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 9 Non-probability sampling: Quota sampling  A convenience sample, with an effort made to ensure a certain distribution of demographic variables  Identification of population characteristics relevant for interpreting study outcomes, e. g. age, religion  Sampling must match population quotas: ensures representativeness of sample and involvement of small groups in the population  Done for economic reasons Symbol Age Group No. 11-21 years 10 23-31 years 14 32-41 years 10 42-51 years 10 Total 11-51 years 44 Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 10 Non-probability sampling: snowball sampling  Respondents refer other respondents Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 11  TUM School of Management Technische Universität München Question to you  When could snowball sampling be a proper sampling strategy?  Usually employed to access hard-to-reach populations (e.g., study of rare diseases, homeless people)  When it is of interest to obtaina sample of highly similar units (share the same traits)  Simple and cost-efficient  Little planning, little knowledge required Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 12 Non-probability sampling: judgement sampling  The researcher uses his/her judgement in selecting the units from the population for the study  If the population to be studied is difficult to locate, or if some members are thought to be better, more knowledgeable, or willing  Similar to convenience sampling, but expert judgement is used to identify respondents Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 13 Non-probability sampling: theoretical sampling  Data collection sequential and choice of respondents informed by the analysis of existing data  Selection of extreme cases  Usually employed in qualitative research Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 14 Assessing external validity Less similar Gradients of similarity Settings Less similar Times Your Study Places Less similar People Gradients of similarity Less similar Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 15 Sample size: considerations  Time and costs  Tolerable sampling error  Heterogeneity within the population  Statistical power needed for hypothesis testing  Expected response rate Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 16 Example: response rate in an online survey 1500 Not contactable 3400 5500 Did not Contacted respond 140 Incomplete 150 Disqualified 600 Responded 310 Usable responses Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 17  TUM School of Management Technische Universität München Question to you  Which approaches can we implement to increase the response rate?  Short questionnaire  Financial incentives  Non-financial incentives  Preliminary notification  Personalization of the request  Follow-up requests  Affiliation with research organizations/charities  Emotional appeals (sympathy/making difference)  Guarantee confidentiality Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 18  Technische Universität München Measurement Construct measurement  Operationalization: process in which a measure for a construct is defined Construct Happiness Attributes Measure Smiley happiness scale Very bad bad neutral good Very good Values Variable Happiness value 1 2 3 4 5 Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 20 Sources of measures  Prior research – Study the method section of existing papers  Search engines – Example: https://inn.theorizeit.org  Books – Examples: “Handbook of Marketing Scales” or “Marketing Scales Handbook”  Develop a new measure Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 21 Example of scale search engines: Theorizeit Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 22 Design choices for measurement  Response – Self-reported: Study subject provides the data – Observation: Data over the study subject comes from observing the study subject (e.g., visual/auditory senses)  Observability – Manifest: Can directly be measured (e.g., temperature, speed, mass, revenue, or prices) – Latent: Can only be inferred indirectly via a measurement model (e.g., values, attitudes, intentions, interests, emotions, or personality traits)  Frequency of observation – Snapshot / cross-sectional: measurement at a single point in time – Panel: repeated measurement over time Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 23 Cross-sectional data: example Selected Observations on Test Scores and Other Variable for California School District in 1998 Observation District average test Student-teacher Expenditure per % of students (district) number score (fifth grade) ratio pupil ($) learning English 1 690.8 17.89 $6,385 0.0% 2 661.2 21.52 5,099 4.6 3 643.6 18.70 5,502 30.0 4 647.7 17.36 7,102 0.0 5 640.8 18.67 5,236 13.9.......... 418 645.0 21.89 4,403 24.3 419 672.2 20.20 4,776 3.0 420 655.8 19.04 5,993 5.0 Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 24 Panel data: example Selected Observations on Cigarette Sales, Prices, and Taxes by Sate and Year for U.S States, 1985-1995 Observation State Year Cigarette sales Average price per Total taxes number (packs per capita) pack (including (cigarette excise tax taxes) + sales tax) 1 Alabama 1985 116.5 $1.022 $0.333 2 Arkansas 1985 128.5 1.015 0.370 3 Arizona 1985 104.5 1.086 0.362............ 47 West Virginia 1985 112.8 1.089 0.382 48 Wyoming 1985 129.4 0.935 0.240 49 Alabama 1986 117.2 1.080 0.334............ 96 Wyoming 1986 127.8 1.007 0.240 97 Alabama 1987 115.8 1.135 0.335............ 528 Wyoming 1995 112.2 1.585 0.360 Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 25  TUM School of Management Technische Universität München Question to you  What could be advantages but also challenges of using panel data over cross-sectional data?  Advantages of panel data: – Cross-section data is merely a snapshot and prone to capturing a temporal or accidental effect  panel data provides richer information – Allows accounting for external influences that are constant within units over time  Challenges for using panel data: costs, time, drop-out – Solution: use existing panel data available for free (EU, …) – May also be bought: GfK Consumer Panel etc. Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 26 Objectives of the operationalization Reliability Operatio- Construct Replicability nalization validity Availability Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 27 Reliability: definition and measurement error  Reliability: degree to which a measure consistently measures a construct—both across (sub)measures and time points 𝑋 =𝑇+𝐸 𝑋 = 𝑇 + 𝐸𝑟 + 𝐸𝑠  X: observed value for the measure of a construct  T: true value of a construct  Random measurement error (Er): A component or part of the value of a measure that varies entirely by chance  Systematic measurement error (ES): A component or part of the value of a measure that varies consistently Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 28 Reliability: measurement error Random measurement Systematic measurement error error The The distribution of x with distribution no systematic error of x with random Frequency error Frequency The distribution The of x with distribution systematic of x with error no random error Random error does not affect Systematic error does affect the average, only the the average variability around the average Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 29 Reliability: reducing measurement error  Reuse proven scales  Pilot tests  Repeated measurement  Train researchers  Double-check data entry  Use statistical procedures to adjust for measurement error Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 30 General tests for reliability  Test-retest reliability: is used to assess Same measure the consistency of an observation from one time to another Time 1 Time 2  Inter-rater reliability: is used to assess the Object or phenomenon degree to which different raters give consistent values for a measure ? = Observer 1 Observer 2  Parallel test reliability: is used to assess Form A = the consistency of the results of two tests Form B constructed in the same way from the same content domain Time 1 Time 2 Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 31 Validity: definition and types Construct validity Translation Criterion-related validity validity Face validity Content validity Convergent Discriminative Concurrent Predictive validity validity Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 32 Reliability vs validity Reliable, Valid, Neither Both not valid not reliable reliable nor reliable and valid valid Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 33 Measuring manifest vs latent constructs Manifest constructs Latent constructs Examples temperature, speed, mass, values, attitudes, intentions, revenue, or prices interests, emotions, or personality traits Measurement Direct observation Scale Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) Scales  A scale is a set of items (questions) that jointly measure a construct 1. In general, I consider myself: 1 2 3 4 5 6 7  Example not a very a very happy happy person Subjective Happiness person Scale 2. Compared to most of my peers, I consider myself: Instructions to participants. For each of the following 1 2 3 4 5 6 7 statements and/or questions, less more please circle the point on the happy happy scale that you feel is most appropriate in describing 3. Some people are generally very happy. They enjoy life regardless of what is going on, getting the most out of everything. you. To what extent does this characterization describe you? 1 2 3 4 5 6 7 not at a great all deal 4. Some people are generally not very happy. Although they are not depressed, they never seem as happy as they might be. To what extend does this characterization describe you? 1 2 3 4 5 6 7 not all a great all deal Lyubomirsky and Lepper (1999) Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 35  TUM School of Management Technische Universität München Question to you  It is typically recommended to reuse existing scales—why?  Reliability  Comparability  Developing novel scales is time-intensive because they require extensive validation  Minor wording tweaks are considered appropriate in some fields Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 36 Capturing responses to scales: Likert scale  Items are statements; scale captures agreement STRONGLY DISAGREE NEITHER AGREE STRONGLY DISAGREE DISAGREE AGREE NOR AGREE I am satisfied with my income. The money I earn is enough for living. – Checkboxes may be replaced by smilies (Kunon-scale) Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 37 Capturing responses to scales: semantic differential  Two ideally contradictory words (usually adjectives) are presented  Simple example: Nike’s Air Max 200 are.. outdated __ __ __ __ __ novel helpful __ __ __ __ __ useless Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 38 Specific tests for scale reliability  Inter-rater reliability: degree to which different raters give consistent estimates of the same measure (e.g., Cohen‘s Kappa)  Internal consistency: correlation between items measuring a construct (e.g., Cronbach’s Alpha, item-to-total correlation) Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 39 Internal consistency tests: Cronbach‘s Alpha n × 𝑐𝑜𝑣 α= 𝑣𝑎𝑟 + (n − 1) 𝑐𝑜𝑣  Measures the internal consistency of a set of items – Relates shared variance to total variance – Idea: Variance shared by the items reflects the variance of the phenomenon  Interpretation – Ranges between 0 and 1, threshold value 0.70 – Increases if inter-item correlation increases – If alpha increases, its fraction attributable to measurement error decreases  x₁, x₂, …., xn are the items of a scale – n is the number of items – 𝑐𝑜𝑣 is the average inter-item covariance among the items – 𝑣𝑎𝑟 equals the average variance of the items Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 40 Internal consistency tests: Item-to-Total correlation  Idea: provide diagnostic information at the item-level  Item-to-Total correlation – rit = r (xi,y) – Where: x1, x2,…, xm items and y = σm i=1 x i  Corrected Item-to-Total correlation – Problem with small number of items: overestimation of r since xi contributes twice – ro, t(o) = r(xo,y) – Where: x1, x2, … xo,…, xm used items xo item, for which the corrected Item-to-Total correlation is computed Scale total: y = (σm i=1 xi) – xo  No established threshold values Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 41 Internal consistency tests: Item-to-Total correlation Mean (Standard Item-to-Total- Construct Items Deviation) Correlation Perceived Price Our business unit/firm… Aggressiveness …counters competitors’ price 3.44 (1.36) 0.70 cuts in a very offensive way …tries to attack competitors’ 3.03 (1.37) 0.78 prices very often …engages in a very aggressive 2.86 (1.41) 0.82 pricing strategy …tries to offer our customers 2.29 (1.26) 0.63 the lowest prices in the market …puts price pressure on our 2.91 (1.39) 0.71 competitors Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 42 Variables  Variable: contains the assigned numerals (i.e., numerals vary)  Types of variables – Dichotomous: only two values – Discrete: finite number of values over a range, and there is a positive minimum distance to the nearest next permissible value (i.e., countable) – Continuous: if it can take infinitely many, uncountable values, even values that are arbitrarily close together  Determined by the operationalization Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 43  TUM School of Management Technische Universität München Question to you  What are examples of dichotomous, discrete, and continuous variables for HAPPINESS?  Dichotomous: happy/unhappy, yes/no  Discrete: very happy / somewhat happy / unhappy  Continuous: degree of happiness between 0 and 1 Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 44 Level of measurement  The level of measurement describes the relationship between the numerical values on a measure (also referred to as scale of measurement) - Ratio scale Numerical - Interval scale (metric) Increasing hierarchy / Increasing scope of - Ordinal scale Categorical information - Nominal scale (non-metric) Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 45 Nominal scale  Used for – Assigning objects to categories – Lowest level of measurement denoted with the minimal requirement  Minimal suitability for statistical analysis (countable) Example: Manager survey Please state the industry of your company: Automobile Education Computer/Electronic/Telecommunication Media/Publishing/Entertainment Energy Retail and wholesale Finance/Insurance/Real Estate Tourism Health care Other Is your company listed on the stock market? Yes No Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 46 Ordinal scale  Used for – Ranking between the different scale points (categories) (A > B > C)  Inferences about the distances between the ordered categories are not possible  Description of ordinal data by measures of location (median, percentile) Example: Employee survey How satisfied are you with the… …training opportunities Very dissatisfied Very satisfied …management skills of your direct supervisor Very dissatisfied Very satisfied Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 47 Special case of ordinal scale: Likert Scale (five-level) Example: Company survey Neither Strongly Strongly Disagree agree nor Agree disagree agree disagree When the new product was planned it was very important for the company The new product was supposed to bring a lot of attention to the company Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 48 Interval scale  Measuring based on constant units (i.e., even space between the units of the measure)  Statements about the distance between two categories are possible  No naturally occurring zero point; distance between each unit is arbitrarily selected, therefore no multiplication or division possible Example: Employee survey Date of survey (Day. Month. Year) Time of survey (Hour: Minutes) Further examples: temperature scale, IQ-scale Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 49 Ratio scale  The scale of measurement has constant units with a fixed, natural zero point  Multiplications and divisions are allowed; ratios can be computed (A is a multiple of B)  Highest form of measurement  Application of all statistical methods possible  Transformation to lower forms of measurement possible (i.e., discretization, binarization) Example: Employee survey Please state your… …age …net income (EUR) Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 50 Level of measurement Ratio Absolute zero Interval exists Distance is meaningful Ordinal Attributes can be Nominal ordered Attributes are only named; weakest Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 51 Level of measurement and permissible statistics Scale of measurement Mathematical operation Permissible statistics Mode Nominal =, ≠ Number of cases Median Ordinal >, < Percentiles Mean Interval +, − Deviation Ratio x, / Coefficient of variation Stevens (1946) Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 52  Technische Universität München See exercise session “Measurement” Questions Professorship for Innovation & Digitalization (Prof. Dr. Jens Förderer) 54