Quantitative Primary Research Design PDF

CHAPTER 6 QUANTITATIVE PRIMARY RESEARCH DESIGN LEARNING OBJECTIVES Create surveys that employ best practices in survey development, including universal survey components, question sequencing, and length. Identify and avoid common survey development mistakes, such as complex, leading, loaded, am...

CHAPTER 6 QUANTITATIVE PRIMARY RESEARCH DESIGN LEARNING OBJECTIVES Create surveys that employ best practices in survey development, including universal survey components, question sequencing, and length. Identify and avoid common survey development mistakes, such as complex, leading, loaded, ambiguous, and double-barreled questions. Identify the types of categorical and continuous scales of measurement for the purposes of response option design and quantitative analysis. INTRODUCTION TO SURVEY INSTRUMENTS Surveys are the MOST COMMON form of PRIMARY RESEARCH and CONTINUE to be an attractive option, in part due to the prevalence of online survey tools. Surveys are mainly quantitative in nature because they are composed of mostly closed-ended questions. Surveys may include open-ended questions, which are qualitative in nature. Survey design is the most critical stage in the survey research process. SURVEY DESIGN is ONE OF THE MOST (if not the most) CRITICAL STAGE in the survey research process. A survey is to be CLEAR and CONCISE and MEASURE what it is purported to measure COMMON UNIVERSAL SURVEY COMPONENTS “OVERALL MEASURES” Include “overall measures” to gauge the sentiment regarding the subject being evaluated. Overall measures can be used for Tracking results over time Correlation analysis DEMOGRAPHICS QUESTIONS Although demographic questions do not help address the research objective, they are necessary for most surveys. Demographic questions: Help describe sample representation Used to segment results Since they tend to be personal nature, use sparingly; only include questions that will help determine sample representation and/or will be useful in segmentation analysis: Age Geography Education level Household income Employment status Housing (rent, own, condo, house) Ethnicity Marital status Gender QUESTION SEQUENCING TYPES OF APPROACHES Funnel approach: to gather top-of-mind responses before more specific questions are asked order bias can occur certain questions could reveal ideas or concepts too soon Pathway approach: organizing a survey in the order experiences occur (customer journey) used for gathering levels of satisfaction for service experiences helps respondents recall events in a logical order and reduces confusion TYPES OF APPROACHES 1 Screener questions: Placed before the core survey questions in order to determine if a potential respondent qualifies to take the survey. Please indicate how often you purchased a toaster pastry product in the last six months. 0 boxes [terminate interview] 1-2 boxes [terminate interview] 3-5 boxes [continue] 5 boxes or more [continue] Unsure [terminate interview] TYPES OF APPROACHES 2 Branching questions: Placed before a sequence of questions that only applies to a subset of respondents. For instance, a hospital satisfaction survey may ask, “Did you receive any xrays or radiology procedures?” If respondents answer “no,” they will skip the sequence of related questions Demographic question placement. Go at the end of the survey unless being used as screener questions; tend to be more personal and could be jarring if first in the sequence SURVEY LENGTH DETERMINING SURVEY LENGTH Time respondents will devote to a survey is linked to (1) the survey format (telephone, mail, online, mobile) and (2) the importance of the topic to respondents Most surveys should run no more than 5-7 minutes; generally, the shorter the survey, the better; long surveys risk incomplete responses and termination. Telephone surveys can run a bit longer Topics in which respondents are invested can afford longer surveys STACKABLE AND MULTI-CHANNEL SURVEYS Stackable surveys are longer surveys broken up into a series of smaller surveys dispersed over time. Multi-channel surveys are large surveys broken into smaller surveys and distributed across multiple platforms. RESPONSE OPTIONS & SCALING Categorical measurements use values that can be counted as parts of distinct groups. The groups can be based on a characteristic or label. Continuous measurements use values that can be expressed numerically and placed in a logical order. Additionally, there is equal distance between each data point. CATEGORICAL MEASUREMENTS: NOMINAL Nominal: A nominal measurement has two or more categories, but there is no natural order to the categories. Why are you considering Shield University? (select all that apply.) ❏ Academic reputation ❏ Sports teams ❏ Small class size ❏ Online courses available ❏ Diverse student body ❏ Residence halls available CATEGORICAL MEASUREMENTS: ORDINAL Ordinal: An ordinal measurement has two or more categories that can be placed in a clear order. What is the highest level of education you have obtained? ❏ High school diploma ❏ Associate ❏ Bachelor's ❏ Graduate or master’s ❏ Doctorate CONTINUOUS MEASUREMENTS: INTERVAL Interval: An interval scale of measurement has a clear order and also displays the exact difference in value along the scale. These distances are called “intervals.” Rate your overall satisfaction with your visit. 1. Poor 2. Fair 3. Average 4. Good 5. Excellent CONTINUOUS MEASUREMENTS: RATIO Ratio: Like an interval measure, a ratio has an equal distance between each data point. The difference is there is an absolute “zero point” that is treated as the point of origin. How much would you expect to pay for concept A? $____. LIKERT SCALES Likert scales are survey responses that offer a range of options from one extreme to another. They may include a neutral midpoint. Likert scales that only display labels on either end of the scale are referred to as anchored Likert scales (such as only labeling 1 and 5 on a 5-point scale) LIKERT SCALE EXAMPLE – Very satisfied – Somewhat satisfied – Neither satisfied nor dissatisfied – Somewhat dissatisfied – Very dissatisfied BALANCE OF SCALES For the most part, it is recommended that Likert scales be balanced. 1. Strongly disagree 2. Disagree 3. Neutral/Neither agree nor disagree 4. Agree 5. Strongly agree USE SAME SCALE FOR MEANINGFUL COMPARISONS When gathering ratings on items, features, or competitors, use the same scale for meaningful comparisons. VISUAL WEIGHT If the visual weight at certain points of a scale are stronger than others, respondents will be more likely to select those options than they otherwise would have. ORDER OF RESPONSE OPTIONS It is the tendency of respondents to choose options at the top of a list more often than items at the bottom of a list. This phenomenon is referred to as order bias. Therefore, it sometimes makes sense to randomize categorical response options so that different respondents see them in a different order. Randomization is a way to combat order bias within response options. Randomization can be easily employed in electronic surveys. Imagine if “billboard” were always the first option in the list below. Because it is the first item respondents see, it would likely result in more respondents selecting the option, giving it an erroneous, inflated score. ORDINAL RESPONSES / INTERVAL VARIABLES Ordinal responses, such as education level, should be presented in their most logical order so as not to confuse the respondent. For interval variables like the Likert scale that measures intensity, scales can either go from positive to negative or negative to positive. “NOT APPLICABLE” OPTIONS Include “not applicable” options. In most instances, researchers shouldn’t force respondents to rate items they are not familiar with. Doing so can result in frustrated respondents and skewed results. It is good practice to include a “not applicable,” “don’t know,” or “unsure” option where appropriate. OVERLAPPING RESPONSE OPTIONS Avoid overlapping response options. Options should be mutually exclusive so that respondents can make an appropriate choice. Consider the example below. If a respondent expects to pay $700, which option would he or she choose in the first list of responses? How much would you expect to pay for a 13” water resistant tablet? OPEN-ENDED QUESTIONS Open-ended questions are used to: Gather strengths and weaknesses of a concept or experience Collect top-mind thoughts Act as a catch-all for collecting items the research team may not have thought of (such as features in a new product development survey). DEVELOPMENT MISTAKES TO AVOID Mistake Description and Example Complex Multiple components and/or stipulations; example: “If you questions attended the art museum more than five years ago, rate how well the museum compares to your visit today. Please exclude visits to the art museum in the past two to three years.” Leading Slanted to imply preference to a particular response, thus questions biasing the result; example: “How do you rate our exceptional oil change service?” Loaded Suggest respondents give a socially desirable answer; example: questions “Rate how likely you are to declaw your next cat, considering that declawing is deemed inhumane by most veterinarians.” Ambiguous Vague; respondents interpret meaning on their own; example: “Is questions the doctor knowledgeable?” Double-barreled Two (or more) questions in one. They can usually be recognized questions by the inclusion of “and” in the statement or by lists of several items. Can be triple or quadruple- barreled, although still referred to as “double barreled.”; example: “Rate your awareness of the American Cancer Society and the services it offers.” CONCLUSION Surveys are the most common form of quantitative primary research. Common universal survey components are (1) overall measures and (2) demographic questions. Most market research surveys follow either a funnel (start broad and get specific) or pathway approach (organized by the order of an experience). Scales of measurement apply to all types of quantitative data, not just survey data, so understanding each is important, especially when it comes to data analysis. Categorical ○ Nominal ○ Ordinal Continuous ○ Interval ○ Ratio When crafting survey response options, there are many considerations to take into account in relation to the objective: The use of categorical or continuous scales of measurement Balance of scales Using the same scale for meaningful comparisons Visual weight Order of response options Avoiding overlapping options Inclusion of not applicable/don’t know options The use of open-ended questions Question development has several pitfalls to avoid, including: Complex questions Leading questions Loaded questions Ambiguous questions Double barreled questions EXTRA NOTES FROM SLIDES: SURVEY COMPONENTS: WHAT TO ASK? New Product Development To gain feedback from a sample of a target market about new product direction Brand Perception To gather unbiased perceptions; almost always kept blind Satisfaction To gauge feedback on current products and services as well as a means to understand levels of engagement for employees or customers BRAND PERCEPTION SURVEY COMPONENTS Line of Rationale Questioning Unaided Unaided awareness question asks respondents to list all of the brands, awareness products, or services they can recall in a category; example: “Which movie streaming services are you aware of? What others?” Aided Aided awareness questions use a predefined list to obtain definitive awareness read; example, “How familiar are you with [insert brand a, b, c, etc. ] on a scale of 1-5 with 1 being not at all familiar and 5 being very familiar?” Descriptors Responses use words or phrases; top-of-mind comments for understanding company perception and ways to position itself; example: “What words or phrases would you use to describe [insert sub shop]?” Brand attribute Determining degree to which company “owns” each attribute in the ratings marketplace; example: “Tell me how much you agree or disagree with these statements on a scale of 1 to 5, where 1 means strongly disagree and 5 means strongly agree... [insert company] is focused on quality [insert company’s] products are comfortable to wear [insert company] is committed to wellness [insert company] is innovative  [insert company] is trustworthy” Avoiding Mistakes 1. Simpler is better 2. Avoid leading and loaded questions 3. Avoid ambiguity: be as specific as possible 4. Avoid double-barreled items 5. Avoid making assumptions 6. Avoid taxing respondents’ memory. Check your questions as we go through!! → Example of Taxing Respondents Memory “How did this year’s event compare to the last event 5 years ago?”