Experimentation and Test Markets PDF

Marketing Experiments and Test Markets 1-1  What Is an Experiment? One variable is manipulated and the effect on another variable is observed o Variable being manipulated is called an experimental, treatment, independent, or...

Marketing Experiments and Test Markets 1-1  What Is an Experiment? One variable is manipulated and the effect on another variable is observed o Variable being manipulated is called an experimental, treatment, independent, or explanatory variable (e.g. ad spending, number of distributors, price) o The variable being observed for an effect is called the dependent variable (e.g. sales) Experimental variables are typically marketing mix elements (price, advertising, product, distribution levels, promotions, etc.) It is the only type of research that has the potential to demonstrate that a change in one variable likely causes a predictable change in another 11-2  Demonstrating Causation To demonstrate that A likely caused B, you must show: 1. Correlation or concomitant variation 2. Appropriate time order of occurrence 3. Elimination of other possible causal factors Causal relationships are always inferred and never demonstrated conclusively! o Increased advertising may not be the real cause of an increase in our sales, o It could be our competitors hiking their prices, or stocking out 11-3  Lab vs. Field Experiments Lab Experiments Field Experiments Controlled Setting Actual Market Environment Helps control extraneous factors (e.g. no Many extraneous factors are at play (e.g. competitive intervention) competitors price actions) High internal validity Low internal validity Low external validity outside the lab High external validity, more generalizable conditions in real life 11-4  Field Experiment Examples Online Advertising’s Impact on Purchases o Lambrecht and Tucker (2013): Dynamic retargeting with online ads increases propensity to purchase only for customers close to making a purchase. Message Effectiveness o Kronrod et al (2012): Google AdWords experiment showing assertive messages (e.g. Protect the Mediterranean) are more effective than generic messaging when customers believe in the cause. Optimizing Paid Search Spending o Yang and Ghose (2010): When paid search is present, so that customers can see both paid and organic search results, the combined click-through rate (CTR) and conversion rates were 5.1% and 11.7% higher compared to no paid search advertising. Sales Force Optimization o Kumar et al.(2008): Coordinating sales calls from different sales teams specializing in different products in a B2B technology firm lead to higher ROI and profits. 11-5  Experimental Design, Treatment, and Effects An experiment may include a treatment group, and a control group o A control group is a group in which the independent variable is not changed during the course of the experiment o A treatment group is a group that is exposed to manipulation (change) of the independent variable (e.g. improved product, better ad message) Experimental effect refers to the effect of the treatment variable on the dependent variable o The goal is to determine the effect of each treatment condition (level of treatment variable) on the dependent variable 11-6  Experimental Notation X :The exposure of an individual or a group to an experimental treatment O: taking measurements on the individuals or groups(e.g. observations, survey) Different time periods are represented by the horizontal arrangement of the Xs and Os o O1 :Conversion rate % at time t1 O1 X O2 O2 :Conversion rate % at time t2 X: Changing the call to action from “Try Now” to “Try Now for Free” The Xs and Os can be arranged vertically to show simultaneous exposure and measurement of different individuals or groups o X1 O 1 X1: Using a celebrity actor (e.g. Seth Rogen) in our You Tube ad X2 O 2 O1 :Group 1’s liking of the YouTube ad (use the same scale) X2: Using an unknown, anonymous actor in our You Tube ad O2 :Group 2’s liking of the same YouTube ad (use the same scale) 11-7  Pre-experimental Designs The one-group pretest–posttest design is the design employed most frequently for testing changes in established products (e.g. changing the product features) or marketing strategies (e.g. changing the ad message) o O1 X O2 The fact that the product was on the market before the change provides the basis for the pretest measurement. Disadvantages: (-) History is a threat to the internal validity of this design (-) Maturation is another threat to this type of design (-) This design has only one pretest observation 11-8  Pre-Experimental Designs : A/B Test Example Marketing Problem: Modifying the message sent to target customers for promoting Zynga’s new online game o Original: The Wizard of Oz Magic Match takes you back to your childhood vs. o Modified: Follow the Yellow Brick Road: The Wizard of Oz Magic Match is the most fun matching puzzle game around! Zynga show the modified ad to 10% of its target market on Facebook # of conversions (before and after viewing the modified ad) are compared The higher conversion message is shown to the rest of Zynga’s target market 11-9  True Experimental Designs We randomly assigns treatments to randomly selected participants o the random assignment is denoted by (R) Randomization makes the results of true experimental designs more valid than those of pre-experimental designs Two types of true experimental designs: o Before and after with control group design o After-only with control group design 11-10  The Before and After with Control Group Design Involves random assignment of participants to experimental and control groups, and pre- and post-measurements of both groups o Experimental group: (R) O1 X O2 X: Using a POP display in the store O3 :Group 2 store sales at o O1 :Group 1 store sales at time t1 time t1 Control group: (R) O3 O4 O2 :Group 1 store sales at time t2 O4:Group 2 store sales at time t2 Due to random assignment, both groups are likely subject to the same extraneous causal factors o Only the treatment (POP display) is different between the two (-)Mortality could be a problem if some participants drop out during the study and the remaining participants differ systematically from the ones that left (-) History could be a problem if factors other than the treatment variable affect one group or the other 11-11  The After-Only with Control Group Design Random (R) assignment of subjects or test units to experimental and control groups, but no premeasurement of the dependent variable X: Using a POP display in the store O1 :Group 1 store sales in a week o Experimental group: (R) OX2 :Group O1 2 store sales in a week o Control group: (R) O2 Random assignment should produce experimental and control groups that are approximately equal in regard to the dependent variable Same disadvantages as the Before and After with Control Group Design 11-12  Quasi-Experiments When you lack complete control over the timing of treatments or must assign respondents to treatments in a non-random fashion More feasible in field settings than in true experiments Examples of quasi-experiments: 1. Interrupted time-series designs 2. Multiple time-series designs 11-13  Interrupted and Multiple Time-Series Designs Interrupted: Repeated measurements of an effect both before and after a treatment o O1 O2 O3 O4 X O5 O6 O7 O8 o E.g. Using a retailer purchase panel to determine the sales effects of a coupon for Ben & Jerry’s Ice Cream (coupon introduced after Measurement 4 and stayed) The multiple time-series design: An interrupted time-series design with the addition of a control group o Experimental group (store 1): O1 O2 O3 X O4 O5 O6 (coupon stays) o Control group (store 2):O O O O4 O5 O6 (no coupon at all) 1 2 3 (+)Many measurements allow better understanding of the effects of extraneous variables (-) Weaknesses (of both designs): o Inability to fully control history (e.g. competitors’ sales promotions, ads, or new products) o Repeated testing and evaluation may create consciousness on participants 11-14  Test Markets Real-world testing of a new product or some element of the marketing mix using an experimental or quasi- experimental design Estimates of new-product failure rates vary all over the place and range to more than 90 percent! Test marketing can help avoid failure 11-15  Uses of Test Markets Estimating a new product’s market share and volume Estimating the effect of a new product launch on the sales of firm’s existing products Characteristics of consumers who buy the product The actions of competitors during the test 11-16  Traditional Test Market The new product is introduced in one or more selected or test markets Sales results are tracked across markets for an extended period (at least a month) Length of time for a traditional test market is dependent on several factors but especially on the purchase cycle of the product Disadvantages/challenges: o Selecting an appropriate city or cities o High cost (advertising and distribution) o Alerting the competition o Potential brand damage if it fails 11-17  Simulated Test Market Simulations of traditional test markets (offline or virtual) Quick Cheaper Can be highly predictive of what will actually happen 11-18  Virtual Test Markets Consumers are recruited to go online and enter a virtual shopping world where are they click through a virtual shopping process Benefits: o Less expensive and more flexible o Engaging o Fast o Easy to iterate o Convenient and comfortable for participants o Less likely to damage the firm’s reputation 11-19  Controlled Test Markets To learn the background of potential customers or assess trial rates Research firms manage a panel of consumers who are tracked regarding their purchase of various products (e.g. in supermarkets) (+)Fast (+)lower cost than traditional test markets (+)concealing the marketing strategy from competitors (-) Panel participants may not be representative of target shoppers 11-20  Steps in a Test Market Study 1. Define the objectives o Also consider competitor reaction speed and cost of the study 2. Select an approach 6. Analyze the test results 3. Develop detailed test procedures o Purchase data (trial rate and repeat rate) 4. Select test markets o Awareness data o At least two markets o Competitive response o Geographically dispersed o Source of sales o Representative of the market o Variety of media outlets 5. Execute the plan o Must continue until repeat purchase rate stabilizes 11-21  Four Major Factors in Determining to Use a Test Market 1. Weigh the risk of failure against the probability of success 2. Can competitors quickly copy your product and introduce it on a national basis? 3. Consider the investment required to produce the product for a traditional test market 4. How much damage an unsuccessful new product launch would inflict on your firms reputation? 11-22