Questionnaire Design PDF

Questionnaire Design 1-1  Why Decision Makers Like Survey Research Gives insights about why people do or do not do something o e.g. why do people not buy pet insurance? Sheds light on how customers make decisions The need to know who makes the decisions...

Questionnaire Design 1-1  Why Decision Makers Like Survey Research Gives insights about why people do or do not do something o e.g. why do people not buy pet insurance? Sheds light on how customers make decisions The need to know who makes the decisions and who plays an influencing role 6-2 Occasions for Using Questionnaires Executive Interviews Mall E-mail Intercept surveys Interviews Telephone Interviews (CATI) 6-3  Executive Interviews Interviewing business people online or at their offices about industrial products or services Very expensive o Must locate authorized decision makers and executives o Must persuade them to agree to be interviewed and then schedule it o Interview must take place at the scheduled time: Long waits are frequent and cancellations are common 6-4  Mall-Intercept Interviews Intercepting shoppers in malls or other high-traffic locations and interviewing them face-to-face in a centralized location(e.g. research office) (+)Easy and simple personal data collection approach (-)Not allowed / less ideal for health risks such as COVID (-)Many malls did not permit marketing research interviewing even pre-COVID (-)Sample representativeness issues 6-5  Telephone Interviews / CATI It was once the most popular survey form Advantages: o Relatively inexpensive o Quick to make calls o High reach potential: Almost everyone owns a phone o Asking follow up questions: e.g. which other brands have you tried? o Probabilistic sampling: Software makes calls randomly (robo-calling) Disadvantages o No answer / hangs up o Questionable representativeness o Must keep it short 6-6  E-Mail Surveys Two types: 1. Ad hoc (also known as one-shot) 2. E-mail panels: Longitudinal, pre-screened 6-7  Pros and Cons of E-Mail Surveys -High rate of non-response and systematic error in ad-hoc surveys Nonresponse is not a problem as long as everyone has an equal probability of not responding o However, certain types of people have a greater probability of not responding. Examples? Response rates may be as low as 2% in ad-hoc surveys 6-8 (Stromp 2021)  How to Increase E-Mail Survey Response Rates Personalization and well-written cover / introduction letter Follow-up email (the sooner the better) Images (a button to click on, an image of the incentive) Incentives o Monetary (e.g. gift card, larger in B2B) o Freemiums (notebook, keychain, pencil, etc.) Promise of contribution to favorite charity (effective in B2B) Entry into drawings for prizes (increasingly common) Emotional appeals (used especially by non-profits, charities) Affiliation with universities or research institutions 6-9  Determining the Type of Questionnaire/Survey Budget o High sample size = Low survey error = High Budget Need to expose respondents to various stimuli o Taste tests, simulated market tests, product prototype evaluation tests usually require face- to-face contact o Online contact is okay for ad copy tests Length o Long questionnaires are difficult to do by mail, over the phone, or in person Incidence rate: is it easy to find participants? 6-10  Length of the Questionnaire The trick is to match the data collection method to the length of the questionnaire Recommended maximum lengths: o Telephone 8 minutes o Mall-intercept 10 minutes o Online 15 minutes o Individual depth interview 60 minutes 6-11  Question Scale Types Graphic Rating Itemized Rating Rank Order Paired Comparisons Constant Sum Semantic Differential Stapel Likert Purchase Intent 6-12 Graphic Rating Scales 3-13  Itemized Rating Scales Respondents must select from a limited number of ordered categories They do not allow for the detailed distinctions Easy to understand and administer Arguably more reliable 3-14  Rank-Order Scales The participant is asked to compare two or more items (e.g. brands, products, ads) and rank each Realistic, easy to understand, and implement Helps understand the most important/preferred alternative o Please rank the following auto brands in terms of their reliability. _______Mitsubishi _______Toyota _______GMC _______Opel Disadvantages: o If all of the alternatives are not included (e.g. Ford), the results could be inconclusive o The item being ranked may be outside the participant’s choice set (e.g. Opel) o Only yields ordinal data 3-15  Paired Comparisons The respondent makes a series of paired judgments between items o Which of the following is more important to you when selecting a Pair 1 Hypoallergenic Smells nice deodorant? Pair 2 Smells nice Affordable Pair 3 Does not stain clothing Smells nice Advantages: o It is easier for people to select one item from two than to rank from a large set Disadvantages o # of paired comparisons increases rapidly as more variables are 3-16 added  Constant Sum Scales Participants are forced to reveal their preferences by dividing a number of points (e.g. 100) among two or more attributes based on their importance Points allocated to each alternative indicates the real ranking of the item (e.g. product attribute, investment alternatives) Disadvantage: Allocating the points becomes difficult if there are too many items Number of points Hypoallergenic Kid friendly Does not shed Easy to train 100 points 3-17  Semantic Differential Scales Semantic differential scale: Respondents rank a concept using dichotomous pairs of words / adjectives on a scale (usually 1 to 7) The mean of the responses for each pair of adjectives is computed, and the means are plotted as a profile or image 3-18  Semantic Differential Scales Advantages and Disadvantages Advantages: 1. Quick and efficient for online surveys (not for CATI) 2. Sufficiently reliable and valid for decision making and prediction in marketing and the behavioral sciences 3. Statistically robust Disadvantages: 1. Suffers from a lack of standardization 2. Having a midpoint on the scale is problematic 3. The halo effect If all the positive adjectives are on the right, responses may be dominated by the extreme alternatives if the participant generally has a positive overall impression 3-19  Stapel Scales A modification of the semantic differential scale Respondent rates on a scale ranging from +5 to −5, how closely and in what direction an adjective fits a given concept (e.g. brand) Advantages: 1. Researcher avoids the arduous task of creating bipolar adjective pairs 2. It provides more detailed insights into measuring attitudes Drawbacks: 1. Choice of adjectives (positive or negative) has been shown to affect the scale results and the participant’s ability to respond 2. Not commonly used in commercial research projects 3-20 Stapel Scale Used to Measure a Retailer’s Web Site Annoying 3-21  Likert Scales Respondent specifies a level of agreement or disagreement with statements expressing either a favorable or an unfavorable attitude Very popular, quick and easy to develop 1. The researcher assembles a large number of statements concerning the concept 2. Each test item is classified by the researcher as generally “favorable” or “unfavorable” 3. Each response is given a numerical weight (1 to 5) 3-22  Purchase-Intent Scales Used to measure a respondent’s intention to buy or not buy a product First asked during concept testing. Then, evaluated at each stage of development for refining the demand estimates Can be a fair predictor of consumer choice for frequently purchased and durable consumer products 3-23  Response Formats: Open-Ended Questions Open-ended questions: (+)Respondent replies in her or his own words; may suggest alternatives not listed in a closed-ended response format Examples: 1. What do you think is most in need of improvement on this page?____ 2. Why do you say that brand [the one you use most often] is better?____ (-)Editing and coding can consume great amounts of time and 4-24  Response Formats: Closed-Ended Questions Participants make a selection from a list of responses (+) No or minimal coding Which sections on our website need improvement? (+) Reading response 1. Navigation pane 2. Submission buttons 3. Search function alternatives may help explain 4. Product videos 5. Text elements the question and generate 6. Other____ realistic responses (+)Measurement error (e.g. wrong or missing entries) is minimized 4-25  Response Formats: Dichotomous Questions Typical yes / no questions Evokes quick response Examples: 1. Did you have to heat your meal before serving it? 2. Do you think that the price of bitcoin in 2023 will be greater or less than it was in 2021? Neutral response may be added o Example? 4-26  Response Formats: Multiple-Choice Questions A type of closed ended question: Presents alternative(s) that correctly expresses the participant’s preference, feelings or opinions (-)Limited insights, list may not be exhaustive (-)Position bias: some people may choose 4-27  Response Formats: Scaled-Response Questions Scaled-response questions: Closed-ended questions that allow us to measure the intensity of respondents’ answers Can use much more powerful statistical tools with some scaled-response questions 4-28  Question Wording Wording guidelines… 1. Clear and simple 2. Specific and unambiguous 3. Avoid biasing the respondents 4. Considers respondent’s willingness to answer 5. Answerable 4-29  Make Sure the Wording Is Clear, Simple, Specific and Unambiguous Question must be easy to understand and mean the same thing to all o What is your total per annum remuneration from all sources combined? Avoid ambiguity. Tailor wording to the target respondent group if needed o How much juice do you think this Tropicana bottle contains? Be specific o What is your annual income? $40,000-$60,000, $60,000-$80,000 Avoid double-barreled questions: Ask one question at a time o Would you agree that Tropicana Orange juice tastes 4-30  Avoid Biasing or Leading the Respondent Question wording can bias the respondent’s answers o How many charities do you support? o “Do you often shop at low class retailers such as ValuStore?” o “Do you believe that the brand Protector is manufactured by Procter and Gamble?” o “What kind of fiction (e.g. mysteries) do you read?” o “Do you intend to buy this product?” Making the purpose obvious early in the interviewing process may distort answers o Mix other brands/firms in questions to temporarily disguise the specific purpose 4-31  Consider the Respondent’s Willingness to Answer the Question Embarrassing, sensitive, or self-concept threatening questions may not be answered If they are answered, the answer will be distorted in a socially desirable direction o E.g. How frequently have you bought condoms in the past 12 months:__________ Alternative: o Have you bought any of the following products in the past 12 months? Toothpaste, Milk, Microwave, Condoms, Potato Chips, Hair Dryer, Vase, etc. o If yes: How frequently have you bought condoms?_____ 4-32  Ways to Deal with the Respondent’s Willingness to Answer the Question (cont’d) Potentially embarrassing questions must be written carefully to minimize measurement error One technique is to ask the question in the third person (e.g. someone you know, most people, etc.) Use counter biasing statements o E.g. At least one in five Canadians complain about nail fungus 4-33  Ask Answerable Questions Is it reasonable to expect that the participant can provide the information? o “How much did you spend on groceries last year?” o “How many hours of Netflix did you watch last month?” Reword questions to make them easier to answer Consider including a “Don’t know” option Avoid black and white answers when the answer may be grey 4-34 o  Ideal Questionnaire Flow and Layout Questionnaires need a logic to the positioning of each section 1. Use screening questions to identify only qualified respondents 2. Begin with a question that gets the respondent’s interest 3. Ask general questions first 4. Ask more complex, mentally taxing questions in the middle 5. Position sensitive, threatening, and demographic questions at the end 4-35 Screening Questions 4-36 How a Questionnaire Should Be Organized 4-37  Quality of Data The quality of data required is another important determinant of the survey method Data quality is measured in terms of validity and reliability o Validity: The degree to which a survey result reflects the true characteristics / opinions / behaviors of target customers o Reliability: whether the survey consistently produces the same results with the same (or comparable) populations Many factors beyond the interviewing method affect data quality 6-38 Types of Survey Error 6-39  Random Error The difference between the sample value and the true values observed in the population o e.g. customers rate your restaurant 4/5 but your satisfaction survey finds 3.8/5 o It is possible to estimate its potential range at a particular level of confidence (e.g. 95%) o It can not be eliminated but it can be reduced. How? 6-40  Systematic Error Results from omissions or mistakes in the research design, measurement or from flaws in the execution of the sample design Dangerous! Results obtained from the sample show a consistent tendency to vary in one direction (up or down) from the true value of the population It is a much more serious threat to survey result accuracy than is random error! Two types: o Sample design error: Results from a problem in the sample design or sampling procedures o Measurement error: Variation between the information being sought and the information actually obtained by the survey questions e.g. two interviewers asking the same question differently causing different answers by similar participants; or entering the responses incorrectly e.g. having leading questions in the survey e.g. having participants that deliberately give misleading answers 6-41  Strategies for Minimizing Errors 1. Random error Increase the sample size. 2. Systematic error Minimize sample design and measurement errors. 2a. Sampling error Define the target population accurately. Do quality control checks to maintain the accuracy and completeness of the sampling frame. Ensure randomness by following proper participant selection procedures 2b. Measurement Select and train interviewers carefully (CATI, face to face) error Design your questionnaires carefully and do pretesting Avoid leading questions and data entry mistakes Encourage all potential participants to join 6-42  Online Software (Free versions) for Designing Surveys Number of Active Surveys 3 Unlimited Unlimited Questions per survey 30 10 Unlimited Max. number of participants 500 25 Unlimited Saved surveys limit Unlimited 100 Unlimited Advanced features Branch logic, (skipping, randomization and piped No No randomization, piping) text Data analysis Summary reports, filtering and cross tabs Trended data and 1 filter No Export Raw Data file Yes No Yes 6-43  Research Ethics: Participant Rights Participants have the right to… Know the name of the person and the research company contacting them, and the nature of the study Verify the legitimacy of the research study (via MRIA or the research director’s contact info) Ensure they will not be sold anything or asked for money Confirm their privacy will be protected in line with the organization’s privacy policy and applicable laws Learn the approximate duration of the study and request to be re-contacted at a later time Decline to participate or answer specific questions, and choose to discontinue at any time Be assured that the highest standards of professional conduct will be upheld at all stages of the study Source: Marketing Research and Intelligence Association of Canada 6-44