Questionnaire Design PDF

MARKETING DATA COLLECTION & ANALYSIS Renaud Lunardo, PhD, HDR Office: 1421 Mail: [email protected] THE STEPS FOR AN EFFICIENT DECISION-MAKING 1. Decide whether existing inf...

MARKETING DATA COLLECTION & ANALYSIS Renaud Lunardo, PhD, HDR Office: 1421 Mail: [email protected] THE STEPS FOR AN EFFICIENT DECISION-MAKING 1. Decide whether existing information is adequate or whether additional information is required 2. If necessary, collect more information in a reasonable and thoughtful way 3. Summarize the available data in a useful and informative manner 4. Analyze the available data 5. Draw conclusions, make decisions, and assess the risk of an incorrect decision. 2 THE STEPS FOR AN EFFICIENT DECISION-MAKING Design of the Managerial study (choice of Data collection Data analysis Recommendation question methods) 3 METHODS OF DATA COLLECTION 4 DATA COLLECTION Firsthand, original data or Interview/ Personal Observation Mechanical information that is directly collected from the source. This data is typically raw, Mail unprocessed, and gathered Telephone for the specific purpose Surveys Personal of the research. Web Primary data collection Secondary data collection Web Information that has Internal Sources Accounting records been collected, Marketing databank processed, and interpreted by others. It involves analyzing data that External Periodicals already exists, often from Sources Reports primary sources. EXAMPLES OF PRIMARY DATA 1. Field studies http://transforming.com/wp-content/uploads/2014/09/Image-45.png 2. Experiment 3. Online data 6  Field studies Experiments Independent Measured Controlled / Manipulated variable X Moderating Measured Controlled / Manipulated variable Z Advantage Association / Correlation Causality Advantage Reflects reality Lack of reality Thomke S. and Manzi J. (2014), The discipline of Business Experimentation, Harvard Business Review, December. 7 EXAMPLE 8 TYPES OF EXPERIMENTS: BETWEEN VS. WITHIN BETWEEN-SUBJECTS designs: Independent samples (participants are randomly assigned to different conditions) WITHIN-SUBJECTS designs: Paired samples (the same participant tests all conditions) Charness, G., Gneezy, U., & Kuhn, M. A. (2012). Experimental methods: Between-subject and within-subject design. Journal of Economic Behavior & Organization, 81(1), 1-8. 9  HOW IT WORKS Participants are randomly assigned to one Each participant experiences all conditions condition.  RATIONALE FOR EACH DESIGN TYPE 1. At least one control group and one experimental group 1. Researchers test the same participants repeatedly differ on a variable to assess differences between conditions. 2. Every experimental group is given an independent 2. There are no control groups in within-subjects designs variable treatment that the researcher believes will because participants are tested before and after have some effect on the outcomes, while control groups independent variable treatments. are given no treatment or a fake treatment 3. The pretest is similar to a control condition where no 3. You compare the dependent variable measures between independent variable treatment is given yet, while the groups to see whether the independent variable posttest takes place after all treatments are manipulation is effective. If the groups differ administered. significantly, you can conclude that your independent variable manipulation likely caused the differences.  PROS AND CONS FOR EACH DESIGN TYPE PROS 1. Fewer threats to internal validity* 1. Fewer participants compared to between- because it prevents carryover effects subjects design. (i.e., being in one experimental condition on 2. No Individual differences across treatments a subsequent condition) 2. Shorter duration → Prevents fatigue effects CONS 1. Requires more participants 1. Longer duration → Fatigue effects 2. Individual differences may threaten validity 2. Hawthorne effect (changing one’s behavior) → Threats to internal validity *: Internal validity examines whether the study design, conduct, and analysis answer the research questions without bias. CONFOUND EFFECTS What are the effects of bottle elongation on perceived wine quality? 13 DESIGNING A QUESTIONNAIRE (way less easy than one would imagine…) 14 STEPS FOR A QUESTIONNAIRE STEP 1: INFORMATION What information will be sought? What information do you want from the survey ? → The hypotheses determine what information will be sought and from whom, because they specify what information relationships will be investigated. What are the questions I want my survey to get answers to ? → Write every single question !! MODELLING OUR QUESTIONS Independent Dependent variable(s) variable(s) X Y 16 MODELLING OUR QUESTIONS Independent Dependent variable(s) variable(s) X Y Moderating variable(s) Z 17 MODELLING OUR QUESTIONS Mediating variable(s) Independent M Dependent variable(s) variable(s) X Y Moderating variable(s) Z 18 METHODS OF ADMINISTRATION STEP 2: ADMINISTRATION What method of administration to use? After specifying the basic information that will be sougt, you need to specify how it will be gathered 19  STEP 3: CONTENT What content? What form? What wording? What sequence? ✓ Is the question necessary ? → Check your model ! ✓ Are several questions needed instead of one ? Avoid writing your own questions → For complex variables, use existing questions/scales from literature Look for scales in articles OR in scales handbooks (“Handbook of Marketing Scales”, Bearden et al., 2012) 20 ASKING THE QUESTIONS PROPERLY ❑ Do respondents have the necessary information ? Can the respondent answer ? ❑ Will respondents give the information ? Even though they have the information, there’s always a question if they will share it. Social desirability bias is a type of response bias that is the tendency of survey respondents to answer questions in a manner that will be viewed favorably by others Examples: politics, drugs use, sex habits, … 21 ASKING THE QUESTIONS PROPERLY Once the content of the individual questions is determined, you need to decide on the particular form of the response. ❑ Open-ended questions ❑ Multichotomous question ❑ Dichotomous questions ❑ Scales 22 ASKING THE QUESTIONS PROPERLY ❑ Open-ended question : respondents are free to reply to open-ended questions in their own words rather than being limited to choosing from a set of alternatives. ❑ How old are you ? ❑ What commercials do you remember seeing on TV last night ? ❑ Why did you purchase a Sony brand plasma TV ? 23 ASKING THE QUESTIONS PROPERLY ❑ Multichotomous question : it’s a fixed-alternative question ; respondents are asked to choose the alternative that most closely corresponds to their position on the subject. How old are you ?  Less than 20  40-49  20-29  50-59  30-39  60 or over How many long-distance telephone calls do you make in a typical week?  Fewer than 5  5-10  More than 10 24 ASKING THE QUESTIONS PROPERLY ❑ Dichotomous question : it’s also a fixed-alternative question but one in which there are only two alternatives. « Do you intend to purchase an automobile this year » ? ❑ Yes ❑ No → Very useful when you have to use « dummy variables » in a regression analysis 25 ASKING THE QUESTIONS PROPERLY ❑ Scales : it’s another fixed alternative question. 2 kinds of scales : the Likert-type scale and the Osgood-type scale. When responding to a Likert question, respondents specify their level of agreement to a statement. Often five ordered response levels are used, although many psychometricians advocate using seven or nine levels. Please circle the number that represents the extent to which you agree with the statement: I totally I disagree totally agree I’m satisfied with the computer software I’ve 1 2 3 4 5 6 7 been using 26 ASKING THE QUESTIONS PROPERLY Involves the phrasing of each question → Critical : poor phrasing of a question will cause respondents to skip it and not answer it, or they’ll answer the question incorrectly. ❑ Use simple words ❑ Avoid ambiguous words and questions ❑ « How often do you tape programs for later viewing ? » Never; Occasionnally; Sometimes; often. ❑ Avoid implicit assumptions ❑ Avoid generalizations and estimates ❑ Avoid double-barreled questions ❑ « What is your degree of satisfaction toward this book and this magazine? » 27 BIASES DUE TO POOR PHRASING 1. Will you marry me? 2. Are you really sure you want to spend all your days, nights, week-ends and holidays with me, while you could have much more fun doing something else with someone else? MEASUREMENT BIAS When the method of observation tends to produce values that systematically differ from the actual value in some way → for example with the use of an improperly calibrated scale, or when questions on a survey are worded in a way that tends to influence the response MEASUREMENT BIAS THEN, SEQUENCING THE QUESTIONS 1. Presentation of the study 2. Introductory / opening questions 3. Specific questions (testing the model) 4. Identification 31  STEP 4: DESIGN Physical characteristics of the questionnaire ✓ Acceptance of the questionnaire ❑ Make the questionnaire reflect the importance of the study → Avoid sloppy questionnaires. ❑ The introduction is also important. The cover letter introduces the study, it must convince the respondents to cooperate. ✓ Questionnaire length is important: smaller questionnaires are better than larger ones. 32 https://kedgebs.eu.qualtrics.com/ 33  STEP 5: PRETEST ❑ Examine each word of every question to ensure the question is not confusing, ambiguous, offensive, or leading. ❑ Get peer evaluations of the draft questionnaire. 34 FINALLY, PRETEST AND REVISE ❑ Use personal interviews among respondents similar to those to be used in the actual study, and obtain comments to discover any problems with the questionnaire ❑ Eliminate questions that do not provide adequate information and revise questions that cause problems. 35 SAMPLING AND BIASES 36 THEN, SAMPLING 37 THEN, SAMPLING Population The entire collection of all observations of interest to the researcher ≠ Sample A representative portion of the population which is selected for the study 38 SIZE DOESN’T MATTER 39  Common misconception that if the size of a sample is small compared to the population size, the sample can’t possibly accurately reflect the population 40 Thomke S. and Manzi J. (2014), The discipline of Business Experimentation, Harvard Business Review, December, p.5 41 THEN, SAMPLING Probability samples : Every member of the population has a known and equal chance of being selected. Nonprobability samples : ❑ Convenience sample : the researcher selects the most accessible population members from whom to obtain information ❑ Quota sample : the researcher interviews a prescribed number of people in each of several categories (ex: 50 women and 50 men) 42 2 BIASES IN SAMPLING 43 SELECTION BIAS When the way the sample is selected excludes some part of the population of interest ≠ For instance when your research focuses on wine lovers whilst your sample involves non-drinkers… NON-RESPONSE BIAS When respondents who choose not to respond may be inherently different than the population 45 CODING THE DATA Raws represent observations; so, if you have collected data from 100 respondents, your dataset will contain 100 raws Columns represent questions; so, if you have 10 questions in your questionnaire, your dataset will contain 10 columns 46