Consumer Insights & Analytics - Week 1 PDF

Consumer Insights & Analytics - Week 1 Prof David De Franza & Prof Marius Claudy www.smurfitschool.ie www.smurfitschool.ie MOTIVATION www.smurfitschool.ie “… activities, set of institutions, and processes for creating, communicating, deli...

Consumer Insights & Analytics - Week 1 Prof David De Franza & Prof Marius Claudy www.smurfitschool.ie www.smurfitschool.ie MOTIVATION www.smurfitschool.ie “… activities, set of institutions, and processes for creating, communicating, delivering and exchanging offerings that have value for customers, clients, partners and society at large.” (American Marketing Association) www.smurfitschool.ie www.smurfitschool.ie www.smurfitschool.ie Context “If the chief marketing officer used to be the most creative person, today it’s the person with the math degree” Patrick Kennedy, Former CEO Paddypower www.smurfitschool.ie Context www.smurfitschool.ie www.smurfitschool.ie www.smurfitschool.ie  Learning Objectives Understand how customer insights and the use of analytics can help marketers achieve strategic business objectives. Develop a critical understanding of various customer analytics tools and methods, recognizing their potential benefits and limitations in different marketing contexts. Transform raw data into meaningful information and actionable insights that can inform and enhance marketing strategies and decisions. www.smurfitschool.ie  Think T – A – E ✓ Theory – knowing and understanding the core range concepts and ideas in your discipline ✓ Application – understanding how these might operate in practice through example, class experiences and discussion ✓ Evaluation – developing informed views through discussion and critique www.smurfitschool.ie 12 Course Outline www.smurfitschool.ie Assessment www.smurfitschool.ie www.smurfitschool.ie Our Expectations Prepare by engaging with the material before the class Contribute: When discussing case studies and exercises Engage with the weekly tasks! This way you will get the most out of each week and you won’t scramble for time at the end! www.smurfitschool.ie Our Expectations Please keep in mind that this is a Masters-Level Class! We assume a basic proficiency in quantitative analysis and statistics as well as knowledge in MS Excel! We won’t be covering these issues in-depths, so if you feel you are not up to speed, now is the time to catch-up! The course outline directs you to some useful resources! Set yourself a goal and spend 10 minutes every day covering some of the key concepts. The resources are nicely structured and give you some ‘bite-sized’ learnings. www.smurfitschool.ie Introduction www.smurfitschool.ie  Customer Insights Customer insights are defined as the firm’s “understanding of current customer needs, the reasons behind these needs, and how these change over time” (Hillebrand et al. 2011, 595) www.smurfitschool.ie https://hbr.org/2016/03/branding-in-the-age-of-social-media www.smurfitschool.ie www.smurfitschool.ie www.smurfitschool.ie Nike registers $35B in revenues — $15B domestically and $20B abroad. Two-thirds of Nike consumers are under the age of 35. A younger consumer who can afford $150 Flyknit racers likely has substantial disposable income and lives in a city. The term for this cohort? Progressive. Of the $20B international customer base, how many believe the US is currently a “beacon on a hill” and is handling race issues well? I’ll speculate, none. Nike has risked $1-3B in business to strengthen their relationship with consumers who account for $32-34B of their franchise. The math? Nike just did it. Scott Galloway 2018 www.smurfitschool.ie  Analytics Analytics refers to “the extensive use of data, statistical analysis, explanatory and predictive models, and fact-based management to drive (marketing) decisions and actions” (Davenport & Harris 2007) www.smurfitschool.ie www.smurfitschool.ie  Analytics Prescriptive/Optimization What’s the best that can happen? Competitive Advantage Predictive modelling What will happen next? Analytics Forecasting/extrapolation What of these trends continue? Statistical analysis Why is this happening? Alerts What actions are needed? Access, Question/drill down Where exactly is the problem? reporting and Ad hoc reports How many, how often, where? metrics Descriptive/Standard reports What happened? www.smurfitschool.ie www.smurfitschool.ie  To what end? blob:https://hbsp.harvard.edu/dc9ca12a-6df5-424c-84ac- 190fee5c4e5e www.smurfitschool.ie Marketing Experiments: A/B Testing www.smurfitschool.ie Marketing Experiments: A/B Testing www.smurfitschool.ie www.smurfitschool.ie STATISTICS www.smurfitschool.ie  Why Statistics? Vast amount of information is contained in data, but this information is often not immediately accessible – statistics helps you extract and understand this information. www.smurfitschool.ie 2-36 Why Statistics? “A change in the role of corporate CMO is considered in which the mining and analysis of data on consumer preferences and behavior has become the most essential task in marketing… CMO’s who want a seat at the table will have to harness customer data and leverage it….” (Zmuda 2012) www.smurfitschool.ie  Is it easy? Nope! But… Much easier to become an expert user of statistics than it is to become an expert statistician… www.smurfitschool.ie 2-38 Statistics & Probability www.smurfitschool.ie Imagine we meet in a year's time. Me: Great to see you, how are you doing? You: Fantastic, it’s been an amazing year… www.smurfitschool.ie 2-40  A Small Exercise Spend 5 minutes to think about what an “amazing” year would look like. What would make you feel amazing in a year's time? Don’t just think about academic achievements. Think about your personal growth, family, friends, experiences, sports etc. www.smurfitschool.ie 2-41  A Small Exercise Now translate this into specific goals (e.g., be fitter & healthier). For each goal get specific on how you would like to measure success (e.g., run 5k under 20 min). Identify inhibitors and enablers that will prevent or help you to achieve your goals (personal and external). Determine two actions you can take to either mitigate an inhibitor or maximize an enabler. www.smurfitschool.ie 2-42 Week 2 Segmentation, Targeting and Positioning Associate Professor Marius Claudy 4 Principles of Marketing Strategy Palmatier et al. 2021 Segmentation, Targeting, Positioning Segmentation, Targeting, Positioning Market Segmentation Market segmentation is the division of a market into different groups of customers with distinctly similar needs and product/service requirements. Purpose of market segmentation: Leverage scarce resources. To ensure that the elements of the marketing mix are designed to meet particular needs of different customer groups. Focus on specific customers’ needs, in the most efficient and effective way. Identify opportunity for growth Process of Market Segmentation Aim is to identify segments where: identifiable differences exist between segments (segment heterogeneity) similarities exist between members within each segment (members homogeneity) Segmentation Criteria Segmenting Consumer Markets Segmentation Example Neil Patel 2021 Segmentations: Emotions Emotional motivators vary by category and brand. Emotional motivators vary across customer segments. Emotional motivators for a given brand or industry vary with a person’s position in the customer journey. Emotional-connection-driven growth opportunities exist across the customer experience, not just in traditional brand positioning and advertising. Magids et al. (2015) Targeting Who should we target? It depends on how we evaluate market segments: Distinct/Effective– is each segment clearly different from other segments Accessible – can buyers be reached through appropriate promotional programmes and distribution channels Measurable – is the segment easy to identify and measure? Profitable – is the segment sufficiently large to provide a stream of constant future revenues and profits? Targeting Hyper Targeted Marketing Age Sex Location Interests Education Birthday Profession Likes Hyper Segmentation Segment Attractiveness Factors Rating approach for different segment attractiveness factors: Segment growth Segment profitability Segment size Competitive intensity within the segment Cyclical nature of the industry Each of these attractiveness factors can be rated (e.g., on a scale of 0-10) and loosely categorized as high, medium or low in attractiveness. Positioning The act of designing the company’s offering so that it occupies a meaningful and distinct position in the target consumer's mind. Effective positioning is the act of linking products and services to the solutions that customers seek. Positioning Two fundamental elements: Physical attributes: the functionality and capability that a brand offers Communication: the way in which a brand is communicated and how consumers perceive the brand relative to other competing brands in the marketplace. Position a brand functionally and/or expressively (symbolically) Positioning Strategies Functional Features Quality Use Expressively User Benefit Heritage Example 19 Positioning Positioning requires a marketer to define and communicate similarities and differences between their brand and competitors. 1. Choosing a frame of reference by identifying the target market and relevant competition 2. Identifying Points of Parity and Difference. 3. Choosing and communicating on Points of Difference (and Parity if applicable) Positioning: Comparative Positioning Statement Once decided on frame of reference, PODs and POPs need to communicate offerings positioning A step towards this is to create a positioning statement Positioning Statement (x) is the brand for (target audience) that satisfies (category need) by offering (benefit) 1. Satisfy why people need it 2. Provide a motivating reason to consider it 3. Emphasise (in advertising) an optimum benefit 4. Have a focus consistent with the motivation driving behaviour in the category “Pillsbury is better than any other brand of baked goods at helping moms to provide their family with emotional warmth because it has years of baked goods experience” Positioning Statement Quiz “For busy professionals XXX offers, a fast, convenient way to rent the right type of car at an airport” “For safety conscious upscale families, XXX offers the safest, most durable automobile in which your family can ride” Summary The process of market segmentation is based on the principle that not all customers are the same. They are many different variables which can be used for segmenting consumer and industrial markets. The 5 criteria for successful segmentation are: effective, measurable, accessible, actionable and profitable. Positioning is the key tool for conveying how companies meet customer needs with products and services. 25 Part II – Segmentation with Pivot Tables Open the data set: kmeans_example.xlsm Go to the sheet, called ‘variables explained’ What do you find? Part III - Cluster Analysis Cluster analysis aims to identify and classify similar entities, based upon the characteristics they possess. It helps the researcher to understand patterns of similarity and difference that reveal naturally occurring groups. The primary input for cluster analysis is a measure of similarity between customers, such as (a) correlation coefficients, (b) distance measures, and (c) association coefficients. Formulating the Problem Perhaps the most important part of formulating the clustering problem is selecting the ‘right’ variables. Inclusion of irrelevant variables may distort clustering solutions. The variables selected should describe the similarity between objects in terms that are relevant to the marketing research problem. The variables should be selected based on past research, theory or a consideration of the hypotheses being tested. K-Means Clustering for Marketers K-Means Clustering is a type of unsupervised learning used to segment data into groups, or "clusters.“ Each group shares similarities within the cluster but is different from other clusters. Example: Think of grouping customers based on their spending habits, demographics, or engagement. K-Means Clustering for Marketers Segment customers into different groups based on characteristics like age, spending, and behavior. Target marketing efforts more effectively (e.g., personalized emails, ads). Increase customer retention by understanding their needs and behaviors. Example: Identify high-value customers versus price-sensitive customers and target them with different offers. K-Means Clustering for Marketers – How Does it Work Step 1: Choose the number of clusters (k). Step 2: Assign data points (customers) to clusters based on proximity to the "centroid" (center of a cluster). Step 3: Recalculate centroids as more points are added. Step 4: Repeat until clusters stabilize. K-Means Clustering for Marketers – Some Terminology Cluster: A group of similar data points (customers). Centroid: The center point of a cluster. Distance: How close a data point is to a centroid (typically measured using Euclidean distance). An ideal clustering solution Centroid Clusters Distance … vs. reality! The Algorithm does the following… The Distance or Similarity Measure The most used measure of similarity is the Euclidean distance or its square. The Euclidean distance is the square root of the sum of the squared differences in values for each variable. Euclidean distance Simple Example of Euclidean distance measure for two individuals: Joe and Sam, on the basis of n variables. Euclidean distance By using the Euclidean distance measure, we can create a pairwise distance matrix. Imagine three people Joe, Sam, and Sara who are being clustered based on their preference for (a) premium offers, and (b) a loyalty scheme. Importance ratings are presented in table below: Name Importance Score Premium Offers Loyalty Scheme Joe 4 7 Sam 3 4 Sarah 5 3 Euclidean distance The pairwise distance matrix would look like this. They are the Euclidian distances for each pairwise combination of the three individuals. Joe Sam Sara Joe 0 3.2 4.1 Sam 0 2.2 Sarah 0 This pairwise matrix is then provided as an input to a clustering algorithm. Euclidean distance The k-means algorithm The cluster center (centroid) is simply the average of all the points in that cluster. Think again about Joe (4;7), Sam (3;4), and Sara (5;3). If we assume that they belong to the same cluster then the centroid is obtained as: How do I choose the number of clusters? You must specify the number of clusters required before the analysis can be run Theoretical, conceptual or practical considerations may suggest a certain number of clusters. The ratio of total within-group variance to between-group variance can be plotted against the number of clusters. The point at which an elbow or a sharp bend occurs indicates an appropriate number of clusters. The relative sizes of the clusters should be meaningful. How do I choose the number of clusters? Elbow Method: A method to find the optimal number of clusters by plotting the intra-cluster variance against the number of clusters. How do I choose the number of clusters? Interpreting and Profiling the Clusters The “elbow method” should allow you to specify the appropriate number of clusters To be sure, you can run the algorithm several times with different starting values (i.e. order of cases in sample) Split the sample randomly into two halves and run the analysis separately. Results are robust if number of clusters and size of different clusters are similar. Interpreting and Profiling the Clusters Interpreting and profiling clusters involves examining the cluster centroids. The centroids enable us to describe each cluster by assigning it a name or label. It is often helpful to profile the clusters in terms of variables that were not used for clustering. These may include demographic, psychographic, product usage, media usage or other variables. When to use K-means clustering? Customer Segmentation: Identify key customer groups. Product Recommendation Systems: Group similar users based on preferences. Personalized Marketing Campaigns: Target each group with tailored content or offers. Sales Forecasting: Use customer segments to predict purchasing trends. Challenges with K-means Choosing k: Selecting the right number of clusters can be tricky. Sensitive to initial placement: Initial centroids can affect results. Data scaling matters: Features with larger ranges can dominate clustering unless data is normalized. Assess reliability and validity 1. Perform cluster analysis on the same data using different distance measures. Compare the results across measures to determine the stability of the solutions. 2. Use different methods of clustering and compare the results. 3. Split the data randomly into halves. Perform clustering separately on each half. Compare cluster centroids across the two subsamples. 4. Delete variables randomly. Perform clustering based on the reduced set of variables. Compare the results with those obtained by clustering based on the entire set of variables. 5. In nonhierarchical clustering, the solution may depend on the order of cases in the dataset. Make multiple runs using different order of cases until the solution stabilizes. Chase Sapphire: Creating a Millennial Cult Brand © Business eLearning Chase Sapphire: Creating a Millennial Cult Brand Learning Understand how Chase Objectives segmented the market to identify highly profitable target customers How it effectively marketed its product to create differentiated value for customers and the firm. Learning Evaluate the factors that made its Objectives product launch and subsequent AER successful Use the CLV method to calculate the ‘worth’ of customer segments “ Q1: How would you describe the target market for Chase Sapphire Reserve? Target Market  Deliver relative, relevant, and Remember: resonant value to target customers Deliver real value to the company Differentiate from the value offered by the competition. “ Q2: What value does the Chase Sapphire Reserve Card offer to its target customers? Value Creation - Customers Value Category Examples Great Rewards (3 points per dollar $ travel/dining); Economic value 1.5% conversion ratio; 100k points sign-up bonus 300$ travel credit (after one year) 1000 points after 3 months Customer Service (24/7) Experiential value Transparent; easy to use app/ website Personal service – phone (no bots); resolving issues fast Stand out card – metal core; thump; sleek design Ego-expressive value Stand out → im interesting; not rich (but really I am rich) Exclusivity / individuality Lounge access; access to experience; chase experience Social value platform; VIP; social events Credit; pay; Functional value Value Creation - Competition Differentiation Ego-Expressive Chase Value Sapphire Reserve CITI AMEX Economic Value “ Q3: How does this translate into value for JP Morgan Chase? Value Creation - Company Sales targets exceeded in two weeks + Connected with core target market + POTENTIAL to enhance CLV ? Loyalty ? Top of wallet → 4k/ 3months + Economics – short term costs vs. long ? term gains Marketing comms + Social proof + “ Q4: What are the economics of the 100,000 bonus point offer?  Year 1 Transactors Revolvers Churners Assumptions Annual Spend $16,000 $16,000 $4,000 % of Balance on Revolve 0% 50% 0% Economics of Interchange fee on spend (p.13) 1.5% 1.5% 1.5% Interest Rates on Revolve Balance 20.5% 20.5% 20.5% customer (midpoint) (p.18) Customer Revenue segments Annual Card Fee (p.1) Interchange fee revenue on spend (spend x fee) Interest revenue on unpaid balance (spend X percent on revolve x interest rate) Acquisition Expenses Card acquisition expenses (p.2) Bonus point expense (@100k) (1.5% x 100k) Margin Break-Even  Year 1 Transactors Revolvers Churners Assumptions Annual Spend $16,000 $16,000 $4,000 % of Balance on Revolve 0% 50% 0% Economics of Interchange fee on spend (p.13) 1.5% 1.5% 1.5% Interest Rates on Revolve Balance 20.5% 20.5% 20.5% customer (midpoint) (p.18) Customer Revenue segments Annual Card Fee (p.1) 450 450 450 Interchange fee revenue on spend 240 240 60 (spend x fee) Interest revenue on unpaid balance 0 1640 0 (spend X percent on revolve x interest rate) Acquisition Expenses Card acquisition expenses (p.2) -375 -375 -375 Bonus point expense (@100k) (1.5% -1500 -1500 -1500 x 100k) Margin -1185 455 -1365 Break-Even =2.7 years = 0.8 years = 3.7 years (375+1500)/ (450+240) = 2.72 “ Q5: How can Chase use marketing to address the profitability of its customer base? Maximize Customer Lifetime value through AER strategies Manage Acquisition: Discourage churners from joining (e.g., 5/24 rule) Identify and attract revolvers (e.g., product design decisions) Lower sign-up bonus Maximize Customer Lifetime value through AER strategies Manage Expansion: Encourage to use the card more (e.g., new ways to earn points) Provide more rewards for spending (spend X; get Y) Surprise customers with additional bonus points Work with partners to provide unique offers Offer more attractive interest rates Reduce costs (e.g., self-service options). Maximize Customer Lifetime value through AER strategies Manage Retention: Monitor spending and push offers if spending declines Increase engagement Loyalty offers Focus retention efforts on most valuable customers Key take aways What happened? Data shows that: 95% of customers are active spenders Spending is up 13% Revenue up 3% Renewals > 90% and churn a lot lower than expected Average income is $180K and average FICO is 785 4,300 applied for a new mortgage – doubled from the previous year. Key take “These are customers that everyone aways wants to acquire. We now have them, and we intend to deepen relationships with them”. Marianne Lake (CFO) “ Good products need to deliver value to customers, and firms, and be competitively differentiated to provide a sustainable competitive advantage. Key take aways Identifying and acquiring profitable customers is a vital first step to maximize CLV. Attracting the wrong type of customers makes it difficult to retain them Retention is key as some consumers only become profitable in the long term. Sales promotions can be a substitute for MKT COMMS and serve to generate “buzz” and trial of new products. There is no substitute for truly knowing your customers Customer insights are not only vital for product development but also for other marketing activities (e.g. communication) Discussion Who are the most profitable customers in your organizations? How would describe these customers (e.g. Demographic; behavior; needs; motivations) What is your organization doing to attract (more of) these customers and to retain them? Based on the learnings today, what could your organization do to further maximize customer lifetime value? Week 4 Customer Centricity: CLV Associate Professor Marius Claudy Back to the Basics Palmatier et al. 2021 Example Opportunity-Based Segmentation Opportunity-Based Segmentation Customer Centricity Customer centric firms recognize that: Some customers are more valuable than others Firms should build their growth strategy around these customer Enhancing their value, extracting some of it, and acquiring more customers like them It is not about putting the product (or related technologies) first It is not about treating all customers equally to drive volume (and lower costs) Customer Lifetime Value “80% of profits are usually derived by 20% of customers” Pareto’s Principle Customer–Product Profitability 8 Customer Profitability It is not always the company’s largest customers who yield the most profit. The largest customers demand considerable service and receive the deepest discounts. The smallest customers pay full price and receive minimal service, but the costs of transacting with small customers reduce their profitability. 9 Customer Profitability A profitable customer is a person, household, or company that over time yields a revenue stream exceeding by an acceptable amount the company’s cost stream for attracting, selling, and serving that customer. The emphasis is on the lifetime stream of revenue and cost, not on the profit from a particular transaction. 10 Customer Lifetime Value Customer Lifetime Value (CLV) describes the net present value of the stream of future profits expected over the customer’s lifetime purchases. CLV calculations provide a formal quantitative framework for planning customer investment and helps marketers to adopt a long- term perspective. 11 Customer Lifetime Value What customer segments Which customers have the should we focus on? How greatest potential to much should we spend on increase “share of acquisition? wallet”? Expand Acquire Retain Are there segments that we can “fire”? Who should we retain? Customer Lifetime Value Where: N = Number of years over which the relationship is calculated Ma = Margin the customer generates in year a Ca = (Marketing) costs directed at customer in year a r = Retention rate, where r(a-1) is the survival rate for year a i = Interest rate AC = Acquisition cost Customer Lifetime Value 2700/ (1+0.1)^1 =2700/(1.1^1) CLV for Year 1 = -1,545.45 CLV for year 2 = 966.95 3040/ (1+01)^2 … 14 Modelling CLV in Excel A few basic rules… Clearly state inputs and assumptions Separate model inputs from analysis Use an easy-to-debug step-by step format Design a model that can be understood and used by others Modelling in Excel Important questions you need to answer before you start: What is the decision? What are the decision alternatives? What are the decision criteria? What is the system (model) that converts inputs into outputs? CLV-to-CAC Ratio CLV-to-CAC Ratio= Customer Lifetime Value Customer Acquisition Cost Exercise: Nutri Sport Exercise: Nutri Sport On-the-Go Everyday Gym Athletes (51% of customers) (23% of customers) (14% of customers) Lifestyle Busy professionals, between 21-35 Ambitious fitness enthusiasts, who Endurance athletes (e.g. marathon looking for a nutritious meal are looking for high-performance runners; triathletes) looking for replacement for on-the-go. These protein shakes to improve muscle gain personalized nutrition plans to consumers are active and health and recovery times. Customers in this improve their overall performance conscious. While customers exercise segment are between 25-45 years of and recovery times. Customers are on regularly, nutrition and calory-reduction age and train between 4-6 times a average 35+ and competitively train 5- are prime motivations for these week. 7 times per week. customers. Product(s) Meal-replacement shakes Performance nutrition and protein Personalized performance nutrition powders Price 49 Euro (2.3kg), approx. monthly supply 35 Euro (2.7kg) approx. monthly 110 Euro (mix of bars; gels powders) supply for a monthly supply Gross Profit ~ 55% ~ 68% ~ 42% Margin Orders per Year ~ 6x per year ~ 9x per year ~ 8x per year Cost of Acquisition Mainly through referrals; social media Direct sales (e.g. gyms); search engine Events (e.g. marathon races); SEO & advertisement; website. Approx. 220 advertisement (SEA) and search SEA; Youtube; 340 Euro (year 0) Euro engine optimization (SEO). Approx. 250 Euro Yearly Marketing- Mainly direct mail and discounts. Tailored discounts and referral Free testing; personal consultations. Related Costs Approx. 8 Euro programs. Approx. 25 Euro per year. Approx. 120 Euro a year (from year 1) Average Retention 65% 75% 88 % Rate Exercise: Nutri Sport 1. Calculate CLV for all three segments. Which segment has the highest CLV? 2. Calculate the Customer Lifetime Return on Investment (i.e. CLV/ Costs of Acquisitions). What do you find? Exercise: Nutri Sport 3. NutriSport has secured a major partnership deal with a large Gym franchise. The CMO believes that this partnership could possibly lower acquisition cost for the Everyday Gym segment by 20% and further result in an increased retention rate of 80%. How do these changes effect CLV? Should NutriSport partner with the Gym franchise 4. Furthermore, NutriSport has considered to launch a national TV campaign to target On-the-Go customers. Estimates show that the campaign would double acquisition costs, but that increased awareness could result in average orders to grow from 6 to 7 and increase retention the rate to 70%. Would investing in this campaign pay-off from a CLV perspective?  APPENDIX RFM Analysis aka CLV for dummies ;-) Recency, Frequency, Monetary Value Segmentation A marketing analysis tool used to identify a firm's best customers by measuring certain factors. The RFM model is based on three quantitative factors: Recency - How recently did the customer purchase? Frequency - How often do they purchase? Monetary Value - How much do they spend? RFM analysis often supports the marketing adage that "80% of business comes from 20% of the customers." 23 Recency, Frequency, Monetary Value Segmentation Customer Sales Data RFM Scoring (Recency, Frequency, Monetary Top 10 % You can use any number of Next 10 % “buckets”. In this example I am using 10 i.e. deciles. Next 10 % Sort in descending … order* Bottom 10% *For Recency use ascending order 24 Recency, Frequency, Monetary Value Segmentation The idea is that customers that rank highly in these characteristics are more likely to be repeat buyers. In this example, we assume that they are also more likely to respond to a direct mail campaign. Let’s test this assumption! We know from the dataset that the average redemption rate is about 4% and our aim is to increase this rate, and thus increase ROMI. 25 Recency, Frequency, Monetary Value Segmentation Acquisition, Retention and Development Most firms use Cost Per Acquisition (CPA) to determine Casting a net Throwing a spear marketing spend on acquiring new customers. Would you use this metric for other kinds of acquisition activities (e.g. employees, technology, lawyers)? Firms should focus instead on Value per Acquisition (VPA), which is CLV! Acquisition, Retention and Development Acquiring new customers can cost five Throwing a spear times more than the costs involved in satisfying and retaining current customers. It requires a great deal of effort to induce satisfied customers to switch away from their current suppliers. The average company loses 10% of its customers each year. Acquisition, Retention and Development A 5% reduction in the customer defection rate can increase profits by 25% to 85%, depending on the industry. Profit rate tends to increase over the life of the retained customer due to increased purchases, referrals, price premiums, and reduced operating costs to service. 29 Acquisition, Retention and Development Share of wallet is the amount an existing customer spends regularly on a particular brand rather than buying from competing brands. It is useful to look for development opportunities, but the upside to these activities is more limited than most managers think. Due to massive customer heterogeneity, there is more opportunity to “move the needle” via acquisition than development. Still, it’s important to pursue development tactics, but think about them as “icing on the cake” Acquisition, Retention and Development According to Prof Peter Fader, retention and development are important but at the margin smart acquisition “wins”… Consider the conventional wisdom: “it costs 5- 10 times more to acquire a new customer than to retain one, so work hard to keep the ones you have…” This may be true, but it misses the point: it focus on costs instead of value. Acquisition, Retention and Development Customer centricity can only succeed by “celebrating heterogeneity” The greatest upside to improve customer profitability arises through “smart acquisition” Don’t overspend on retention –the flighty customers will fly away no matter what you do (or they’ll be unprofitable) View development as “icing on the cake.” Attempting to turn persistent “detractors” into “promoters” is a difficult task, and the resources required to do so can be better invested elsewhere Introduction to Marketing Experiments MKT46150: Lecture 5  Which video is best? Every year Baker Creek publishes a 500-page seed catalog for gardeners Seeds for sale Plant information Recipes Garden photos Customer stories Promote traditional content marketing stock via TikTok Created 3 videos (Danilkovich, 2023) Analyzing campaign cost Highest cost Video 1 was most costly Video 2 was least costly What about revenue? (Danilkovich, 2023) Lowest cost Comparing purchases and ROAS Video 2 is best! Purchases Return on Advertising Spend (Danilkovich, 2023) A B Which ad is best? MKT42400 5  325% more clickthrough! A B MKT42400 6 Using data to make decisions Data driven decision-making But… associated with 3-5% increase Data quality is often poor in output and productivity Data and results can be hard to Quality data that informs interpret marketing innovation leads to improved marketing 32% of businesses report performance difficulty in finding tangible value from data (Brynjolfsson et al., 2010; Sanéz et al., 2022; Stackpole, 2021) Experiments as a decision-making tool Randomized experiments answer questions with data Refine questions to be narrow and specific ✘ Does advertising work? ✔ Is ad treatment A better than treatment B ✘ How should we position our new product? ✔ Should we position our new product with a promotion or prevention orientation (Graves, 2023; Hauser & Luca, 2015) Experiments as a decision-making tool Randomized experiments answer questions with data Allows data to support intuition Folk knowledge: Losses loom larger than gains “Our customers are risk avoiding” “Promotions never work for us” How do we know? Experiments as a decision-making tool Randomized experiments answer questions with data Allows data to support intuition Provides paradigm for quality data collection Experiment data is collected systematically Hypotheses and motivation are documented Experiments as a decision-making tool Randomized experiments answer questions with data Allows data to support intuition Provides paradigm for quality data collection Uses data to inform decision- making What is a random experiment? A random experiment, trial, or event is the An observation or outcome is the result of a process of making an observation of a random random experiment. variable. The sample space is the set of all possible A random variable is a variable whose value is outcomes. unknown (𝑋) and the function that links possible values to an observed value. If we flip a coin three times, the sample space 𝑆 is: 𝑆 = { 𝐻, 𝐻, 𝐻 , 𝐻, 𝐻, 𝑇 , 𝐻, 𝑇, 𝐻 , 𝑇, 𝐻, 𝐻 , 𝐻, 𝑇, 𝑇 , 𝑇, 𝐻, 𝑇 , 𝑇, 𝑇, 𝐻 , (𝑇, 𝑇, 𝑇)} Where data sets come from Population: Collection of all possible units we are interested in Sample: A subset of units taken from the population  Why is sampling important? A random experiment! We infer the average of a series of random experiments… By conducting a single experiment. Types of sampling Non-probability Probability Convenience sampling Simple random sampling Mall intercept surveys Each visitor is randomly selected to see an ad or not Self-selection sampling Visitors opt-in to taking a web- Stratified sampling based survey Each visitor within an age group randomly sees an ad Snowball sampling Age groups are sized proportionally Asking each participant to to population recommend 5 additional participants What is a randomized experiment? A random experiment, trial, or Random assignment is the event is the process of making an process of dividing units (e.g., observation of a random variable. consumers) into groups, whereby each unit has an equal chance of being placed in any one group. A randomized experiment is a comparison of two groups which are formed through random assignment. Random assignment in practice 1. A unit is selected for the experiment Drawn from a database of past 𝑝 = 0.5 customers Visits a website Conducts a targeted search 2. A unit is assigned to a group or 𝑝 = 0.5 condition Probability of group assignment is equal Benefits of random assignment What if some people are in a bad mood? What if some people don’t like tablets? What if some unknown difference drives preference? Accounts for Random assignment controls for individual and unknown differences Ensures differences are distributed in the proportion they exist in the population Ideally, randomized experiments are contemporaneous A randomized experiment is a A contemporaneous comparison is contemporaneous comparison of one in which data for all groups is two groups which are formed collected at the same time. through random assignment. Random assignment is the process of dividing units (e.g., consumers) into groups, whereby each unit has an equal chance of being placed in any one group. Controls for external influence during an experiment Contemporaneous experiments control for external events: An influencer is seen with the product Seasonal events influence intent A search provide changes the algorithm Bad weather impacts everyone’s mood People suddenly start spending more time at home Developing an experiment: Generate an idea Where do experiment ideas come from? Generate Consumer behavior research literature an idea Do losses really loom larger than gains? Analysis of existing data Most visitors view 10 or more pages Industry best practice Showing a progress bar on checkout Intuition and experience Our customers always respond to a price discount The “Highest Paid Person’s Opinion” The CEO doesn’t like red Arguments The creative director wants high-resolution photos The head of IT wants faster page speed Questions What if we showcased reviews above the product description? Developing an experiment: Identify a hypothesis A hypothesis is an informed prediction about the Generate outcome of an experiment. an idea A simple formula for hypotheses is if-then-will: If seeing keywords appeals to user Identify a search intent, then bolding keywords in hypothesis search ads will increase ad clickthrough by 10%. Developing an experiment: Create variations A/B/n tests (“Sequential testing”): Each variation features only one small change Bold CTA button text or not bold CTA button text Generate Red CTA button or blue CTA button The “A” variation is often the baseline or control an idea Existing design of the page/ad/element Identify a hypothesis Create variations  In sequential tests, change one characteristic at a time… TEST 2 TEST 1 Winner becomes new control CONTROL NEW CONTROL CTA message change… CTA color change… VARIATION ✓ VARIATION Developing an experiment: Create variations A/B/n tests: Each variation features only one small change Bold CTA button text or not bold CTA button text Generate Red CTA button or blue CTA button The “A” variation is often the baseline or control an idea Existing design of the page/ad/element Pros: High degree of experimental control (validity) Allows for more confident statements of causality Cons: Identify a Slower to identify impact hypothesis Results in many inconclusive results Often ignores interactions between elements Multivariate tests: Changes are combined Bold-Red CTA Scarcity Headline Red CTA Scarcity Headline Bold-Blue CTA Scarcity Headline Bold-Red CTA Discount Headline … Create variations  How many variations does each customer see? In multivariate tests, all possible combinations are compared simultaneously Developing an experiment: Create variations A/B/n tests: Each variation features only one small change Bold CTA button text or not bold CTA button text Generate Red CTA button or blue CTA button The “A” variation is often the baseline or control an idea Existing design of the page/ad/element Pros: High degree of experimental control (validity) Allows for more confident statements of causality Cons: Identify a Slower to identify impact hypothesis Results in many inconclusive results Often ignores interactions between elements Multivariate tests: Changes are combined Bold-Red CTA Scarcity Headline Red CTA Scarcity Headline Bold-Blue CTA Scarcity Headline Bold-Red CTA Discount Headline … Create Pros: variations Faster to identify impact Often produces an identifiable result Captures interactions between changes Cons: Requires a larger sample size Often prohibits statements of causality Developing an experiment: Analyze results Generate an idea Which variation was associated with the most sales? Variation A: Identify a Mean sales: 25 hypothesis 95% CI: [22, 28] Are they meaningfully different? Variation B: Mean sales: 30 We will return to this question 95% CI: [27, 33] next week… Analyze Create results variations  Developing an experiment: Analyze results Generate an idea Make the winning variation the new default Deploy Identify a and start the process again. changes hypothesis Analyze Create results variations Develop an A/B test plan You have a restaurant that Develop an experiment to test provides delivery this question with Google PPC What kind of restaurant is it? ads You think your customer have Variations should modify the headline and/or body either a promotion or prevention orientation What characteristics differentiate Work individually, with your these two? friend/neighbor, and/or with a generative AI agent Analyzing Marketing Experiments MKT46150: Lecture 6 Restaurant Grades What is RG interested in understanding? How effective advertising packages are in helping client restaurants. What’s the key performance indicator? Understanding digital marketing measures A measure is a quantified observation at the unit level Clicks, impressions, sales, order value A metric is a measure or combination of measures which are linked to performance Clickthrough rate, average order value From measures to objectives Objectives and key results: Collection of KPIs tied directly to objective outcomes OKR Key performance indicator: Metric used as representation KPI of business health or campaign success Metrics Measure or combination of measures linked to performance Measures Quantified observations Choosing the right metrics Metrics are often a proxy for the true outcome of interest! 1. Metrics should be easy to understand If calculations are too complicated, interpretation becomes difficult 2. Metrics should be consistent Different marketing groups should use the same metrics across campaigns For example, avoid using “net ads” in one campaign and “gross ads” in the next 3. Metrics should be easily replicated Capture similar sets of metrics at regular intervals (e.g., monthly) 4. Metrics should provide insight, linked to objectives E.g., Avoid choosing an engagement metric if the goal is conversions 5. Determine the one metric that will indicate success or failure in advance Use other metrics and measures to understand this primary metiric E.g., Low conversion rate (primary metric) is explained by low bounce-rate and high time-on-site What might be happening here? Restaurant Grades Ultimately, what is the outcome of interest? Restaurants want to know how advertisements impact sales. Restaurant Grades Do we have enough data? What other data would you collect? Restaurant YES Do we have enough data? NO Grades Covers all restaurants in Not every customer calls or sample orders though the app Three independent proxies of No metrics are a direct Do we have enough data? behavior measure of sales Proxies indicate demand Metrics don’t indicate what is purchased It’s difficult to get more data No information on cost or revenue (i.e., no profit)  1. Pageviews? Demonstrates changes in awareness Restaurant Gap between interest and conversion Grades 2. Calls to the restaurant? 3. Reservations? What is the key Farthest down the funnel performance indicator? When was the last time you made a reservation to get a spice bag? Why? Could we use all three? How would we weight them? Calculate the average treatment effects Calculate typical values for each treatment group Calculate the typical value for each metric 𝑥1 +𝑥2 +⋯+𝑥𝑛 1 Mean: 𝑋ത = 𝑛 = 𝑛 σ𝑛𝑖=1 𝑥𝑖 =AVERAGE() Use a pivot table! Calculate lift: 𝐴𝑐𝑡𝑢𝑎𝑙 −𝐵𝑎𝑠𝑒𝑙𝑖𝑛𝑒 https://youtu.be/UsdedFoTA68 𝐵𝑎𝑠𝑒𝑙𝑖𝑛𝑒 ∗ 100 Calculate the heterogeneous treatment effects Calculate typical values for each treatment group and restaurant type What is the lift? Advertising is more effect for chain What can you conclude from restaurants this table? Restaurant Grades Which ad package is most effective? Why? Restaurant Grades Which ad package is most effective? For all restaurant types? Restaurant CONFIDENT NOT CONFIDENT Used control and treatment Using proxies instead of Grades groups measuring outcome of interest Effect is (mostly) consistent Results may not generalize across treatment groups How confident are you in these results? Large sample (n = 30,000) Estimates may be noisy Large effect size/magnitude No information on cost or revenue (i.e., no profit) Statistically significant? Customers may see multiple ads What is variability? Variability is the degree to which data differs from each other (or from the typical value). Also referred to as diversity, dispersion, spread, and uncertainty Conditional Excel functions IF Formulas COUNTIF, AVERAGEIF Formulas https://youtu.be/KkTaQ5OjAGc https://youtu.be/AZuBNWMh7VM  Measures of variability: Standard deviation The standard deviation is a Calculate the standard deviation: summary of how far from the mean 1. Calculate difference between values in the data are. each value 𝑥 and the mean 𝑋ത The average difference from the mean 2. Square each difference 3. Add all differences 1 𝑆= σ𝑛𝑖=1 𝑥𝑖 − 𝑋ത 2 4. Divide by 𝑛 − 1 𝑛−1 5. Take the square root of the In Excel: =STDEV.S() quotient Interpreting the standard deviation  A thought experiment Draw 30 random restaurants from the data Calculate the lift in pageviews for treatment 1 and 2 Now how confident are you? The observed result varies from the typical value  Constructing the sample distribution Random Observations Sample Mean Experiments SAMPLE ഥ 𝑿 Random ഥ SAMPLE 𝑿 Variable The 𝑿 SAMPLE ഥ 𝑿 Population Central limit theorem: The distribution of the sample means ഥ will approach a normal SAMPLE 𝑿 distribution, given enough samples (𝑛 ≥ 30). SAMPLE ഥ 𝑿 SAMPLE ഥ 𝑿  Characterizing the sample distribution Typical value of the sample distribution: 1 𝜇𝑋ത = 𝑛 σ𝑛𝑖=1 𝑋ത𝑖 The mean of the sample means Variability or uncertainty of the sample distribution: 𝑆 𝜎𝑋ത = 𝑛 𝑆 is the standard deviation of the sample means 𝑛 is the total number of samples The standard error of the means Note that 𝝁 and 𝝈 indicate we are calculating from the population or sample distribution Standard deviation or standard error? Standard Deviation Standard Error 𝑆 𝑛 𝜎= 1 𝑛 𝑆= ෍ 𝑥𝑖 − 𝑋ത 2 𝑛−1 𝑖=1 Indicates how far the sample mean is from the population mean Indicates, on average, how far each observation is How far is the mean of the sample from the from the mean population mean 𝜇? How far is each observation, 𝑥𝑖 , from the mean 𝑿 ഥ? Level of the statistic Level of the unit or observation  Inferring the sample distribution Random Observations Sample Mean Experiments SAMPLE ഥ 𝑿 In practice, we only conduct one random Random ഥ SAMPLE 𝑿 experiment, and we infer Variable the properties of the The sample distribution. 𝑿 SAMPLE ഥ 𝑿 Population SAMPLE ഥ 𝑿 SAMPLE ഥ 𝑿 SAMPLE ഥ 𝑿  What are inferential statistics? To infer is to form an opinion or Inferential statistics is the guess that something is true process of using the because of the information that information (observations or you have. data) available to form an opinion about the underlying (unobserved sample or Descriptive statistics summarize population) distribution of that the characteristics of the data or individual sample. information. Comparing descriptive and inferential statistics Descriptive Inferential 𝑛 𝑆 1 𝑆𝑋ത = 𝑋ത = ෍ 𝑥𝑖 𝑛 𝑛 𝑖=1 Estimated standard error of the sample Mean of the sample Since we haven’t measured the 1 𝑛 complete sample distribution, we’re 𝑆= ෍ 𝑥 − 𝑋ത 2 making a guess or estimate of the 𝑛 − 1 𝑖=1 𝑖 sample error. Standard deviation of the sample Whenever we make a guess, we know there’s a chance of being wrong. How confident are we in our guess?  Recall the normal distribution There’s a 95% chance an observation is There’s a 5% chance an observation is not within 2 standard deviations of the mean within 2 standard deviations of the mean At what standard deviation value is there exactly a 95% of being included in the data? 2.5% chance an 2.5% chance an observation is less than 2 observation is more than 2 standard deviations standard deviations Scaling our confidence to the data If the data is not standard normal… We want to scale our confidence With very small 𝑛, based on the data 95% of the data is much more than 2 The more data we collect, the more standard deviations! confident we can be Thus, we scale according to 𝑛 𝑛=3 𝑛 = 10 𝑛 = 50 𝑛 = 100  How the distribution scales… With very small 𝑛, 95% of the data is much more than 2 standard deviations! As 𝑛 increases, the distribution approaches the standard normal, with standard deviation of -1.96 and 1.96 bounding 95% of the data. A historical aside… Known as Student’s t distribution A variation of the normal Scales based on 𝑛 Invented by William Sealy Gosset Working at the Guinness factory in Dublin Concerned with variation in the chemical properties of barley Signed his 1908 paper “Student” Estimating the number of standard errors from a probability of the t distribution At what standard error value t is there exactly a 95% chance an observation occurs in the data? “2 tail” T.INV.2T(probability, deg_freedom) ? TAILS  What are degrees of freedom? If a theoretical population mean 𝝈 ≈ ഥ and the sample 𝒏 = 𝟑, two values 𝑿 of 𝑿 can vary but the third must ensure 𝑺𝑼𝑴 𝑿 = 𝑺𝑼𝑴(𝑿 ഥ ). The degrees of freedom are the The mean or average is the number of values in the calculation of a statistic which are free to vary. total value of items divided evenly across all items. 𝑋 = 5,10,15 𝑆𝑈𝑀 𝑋 = 5 + 10 + 15 = 30 𝑋ത = 10 𝑆𝑈𝑀 𝑋 = 30 5 + 10 + 15 𝑋ത = IF X = 12, 10, 𝑥 IF X = 10, 15, 𝑥 3 THEN: 12 + 10 + 𝑥 = 30 THEN: 10 + 15 + 𝑥 = 30 22 + 𝑥 = 30 25 + 𝑥 = 30 30 𝑥 = 30 − 22 𝑥 = 30 − 25 = = 10 𝑥=8 𝑥=5 3 Estimating the number of standard errors from a probability of the t distribution, resolved At what standard error value t is there exactly a 95% chance an observation occurs in the data? “2 tail” T.INV.2T(probability, deg_freedom) deg_freedom = 𝑛 − 1 TAILS Determining a confidence interval 1. Determine desired confidence level Typically, 95% confidence 2. Calculate the mean (17.64) and standard deviation (0.21) 3. Determine degrees of freedom 𝐷𝐹 = 𝑛 − 1 = 30 − 1 = 29 4. Calculate standard error 𝑆 0.21 0.21 𝑆𝑋ത = 𝑛 = 30 = 5.48 = 0.038 5. Calculate t-critical value 𝑡 =T.INV.2T(1-0.95,29)= 2.05 6. Calculate Lower 95% confidence bound 𝑋ത − 𝑆𝑋ത 𝑡 = 17.64 − 0.038 × 2.05 = 9.85% 7. Calculate Upper 95% confidence bound 𝑋ത + 𝑆𝑋ത 𝑡 = 17.64 + 0.038 × 2.05 = 25.43% If we keep repeating the experiment, we expect the mean to fall within 9.85 and 25.43% of the time. Restaurant Grades What is RG interested in understanding? How effective advertising packages are in helping client restaurants. What’s the hypothesis? Restaurant Grades If consumers are motivated by seeing relevant dishes then the new algorithm emphasizing food images will generate more reservations Also known as: Research hypothesis Accepted given sufficient statistical evidence Alternative to what? Alternative hypothesis Restaurant Grades If consumers are motivated by seeing relevant dishes then the new algorithm emphasizing food images will generate more reservations The “null” hypothesis Using the new ad algorithm, the number of booked Null meaning no reservations will be the same as the baseline (population). difference or no effect.  Picking a hypothesis Null hypothesis Alternative hypothesis Using the new ad algorithm, the If consumers are motivated by number of booked reservations will be seeing relevant dishes the same as the baseline. then the new algorithm emphasizing food images The default assumption will generate more reservations Only rejected with convincing statistical evidence Only accepted if the null is rejected Otherwise, accepted Demands convincing evidence Thus, the null can be accepted without convincing evidence “Test” the hypothesis using the data  How have we characterized random variation of the mean? An alternative in practical terms In practical terms: Are averaged booked reservations from the new algorithm showing food… We can’t know the population Equal to the averaged booked mean 𝜇 reservations from the baseline We’re interested in the representing the population (no difference between a sample advertising) mean 𝑋ത and some baseline 𝜇0 𝑋ത − 𝜇0 = 41.68 − 33.96 = 7.72 Yes! There are 7.72 more reservations We test: on average using the new algorithm 𝐻0 : 𝑋ത = 𝜇0 Could this difference be the result of random variation?  Is 𝑋ത truly different from 𝜇0 ? What does this mean? 1. Determine desired confidence level Compare 𝑋ത (41.68) to the 95% confidence Typically, 95% confidence interval of 𝜇0 : 2. Determine degrees of freedom 𝐷𝐹 = 𝑛 − 1 = 10000 − 1 = 9999 If 𝑋ത is outside the confidence interval 3. Calculate standard error Reject 𝐻0 𝑆 6.59 6.59 𝑆𝜇0 = = = = 0.066 𝑛 10000 100 Otherwise: 4. Calculate t-critical value Fail to reject 𝐻0 𝑡 =T.INV.2T((1-0.95),9999)= 1.96 Implicitly: Accept 𝐻1 5. Calculate Lower 95% confidence bound 𝜇0 − 𝑆𝜇0 𝑡 = 33.96 − 0.066 × 1.96 = 33.96 − 0.129 = 33.831 6. Calculate Upper 95% confidence bound 𝑋ത + 𝑆𝜇𝑜 𝑡 = 33.96 + 0.066 × 1.06 = 33.96 + 0.129 = 34.09 Recall the normal distribution 95% chance an observation occurs here What does the region between the red lines represent? ~Two (1.96) standard deviations of the mean The area of highest concentration of the data Values with the highest probability < 5% chance an observation occurs here. Could we be more precise about the likelihood of a value occurring in the distribution? Calculate the p value What is the probability 𝑝 that 𝑋ത falls within the distribution of 1. Calculate the t statistic for the 𝜇0 ? difference between 𝑋ത and 𝜇0 ത 0 𝑋−𝜇 7.72 𝑡𝑋−𝜇 ത 0 = = = 116.97 𝑆𝜇0 0.066 Interpretation 2. Find the probability of the t statistic There is a very small chance T.DIST.2T(116.97,𝑛 − 1)≈ 0 (𝑝 < 0.0001) mean reservations booked under Note: A p-value can never be equal to the new algorithm are equal zero, but it can be very small. Excel will to those of the baseline truncate small values to zero for computational convenience. The best way to report a very small (≈ 𝟎) 𝒑-value. Interpreting a p-value There is a very small chance (𝑝 < OR: 0.0001) mean reservations There is 𝑝 < 0.0001 chance we booked under the new algorithm have observed a difference are equal to those of the baseline between the sample and (population). population means, when in fact there was no difference. Type I error t-test for differences between groups Using the new ad algorithm, the number 1. Calculate 𝑆𝑆 by combining the of booked reservations will be different (squared) standard error than the old ad algorithm. 𝑆𝑆 = 𝑆𝑋2ത1 + 𝑆𝑋2ത2 Using the new ad algorithm, the number 2. Calculate the t-statistic of booked reservations will be the same A measure of difference between the as the old ad algorithm. two groups 𝑋ത1−𝑋ത2 𝑡𝑠𝑡𝑎𝑡 = 𝑆𝑆 3. Calculate degrees of freedom We have two means to account for now! 𝐷𝐹 = 𝑛1 + 𝑛2 − 2 4. Calculate the 𝑝-value T.DIST.2T(𝑡𝑠𝑡𝑎𝑡 , 𝐷𝐹) t-test for differences between groups Using the new ad algorithm, the number of Determine difference in means: booked reservations will be different than the old ad algorithm. 𝑋ത𝑛𝑒𝑤 − 𝑋ത𝑜𝑙𝑑 = 41.68 − 34.02 = 7.66 1. Calculate 𝑆𝑆 by combining the (squared) standard error Using the new ad algorithm, the number of booked reservations will be the same as the old 𝑆𝑆 = 𝑆𝑋2ത𝑛𝑒𝑤 + 𝑆𝑋2ത𝑜𝑙𝑑 = 0.082 + 0.072 = ad algorithm. 0.0064 + 0.0049 = 0.0113 = 0.106 2. Calculate the t-statistic There is 𝑝 < 0.0001 chance the difference of 𝑋ത𝑛𝑒𝑤 −𝑋ത𝑜𝑙𝑑 41.68−34.02 7.66 7.66 reservations is due to random variation. 𝑡𝑠𝑡𝑎𝑡 = 𝑆𝑆 = 0.106 = 0.106 = 72.26 3. Calculate degrees of freedom 𝐷𝐹 = 𝑛1 + 𝑛2 − 2 = 10000 + 10000 − 2 = 19998 How confident are we? 4. Calculate the 𝑝-value T.DIST.2T(72.26, 19998)≈ 0 Confidence intervals for differences between groups Using the new ad algorithm, the number 1. Determine confidence level of booked reservations will be different Typically, 95% confidence than the old ad algorithm. 2. Calculate degrees of freedom 𝐷𝐹 = 𝑛𝑛𝑒𝑤 + 𝑛𝑜𝑙𝑑 − 2 = 10000 + 10000 − 2 = 19998 3. Calculate the pooled variance Using the new ad algorithm, the number A measure of variation between the two groups of booked reservations will be the same 2 2 𝑛𝑛𝑒𝑤 −1 𝑆𝑛𝑒𝑤 +(𝑛𝑜𝑙 𝑑−1)𝑆𝑜𝑙 𝑆𝑝2 = 𝑑 𝑛𝑛𝑒𝑤 +𝑛𝑜𝑙 𝑑−2 as the old ad algorithm. = 10000−1 ×8.152 + 10000−1 ×6.502 10000+10000−2 664658.2+422971.5 1087629.703 = = = 54.39 19998 19998 4. Calculate the t-critical value There is 𝑝 < 0.0001 chance the 𝑡 =T.INV.2T((1-0.95),19998)= 1.96 difference of 7.66 reservations is due to 5. Calculate the margin of error random variation. 𝑆𝑝2 𝑆𝑝2 54.39 54.39 𝑀𝐸 = 𝑡 × + = 1.96 × + = 0.0054 + 0.0054 = 𝑛𝑛𝑒𝑤 𝑛𝑜𝑙 𝑑 10000 10000 0.011 = 0.105 We are 95% confidence the true 6. Calculate the lower bound difference lies between 7.56 and 7.77 (𝑋ത𝑛𝑒𝑤 −𝑋ത𝑜𝑙𝑑 ) − 𝑀𝐸 = 7.66 − 0.105 = 7.56 mean reservations. 7. Calculate the upper bound (𝑋ത𝑛𝑒𝑤 −𝑋ത𝑜𝑙𝑑 ) + 𝑀𝐸 = 7.66 − 0.105 = 7.77  What does it really mean when p > 0.05? If we repeat the experiment again… Difference Between Means Across Experiments When p < 0.05 Difference Between Means Across Experiments When p > 0.05 18.00 8.00 Difference Between Treatment 0 and Treatment 2 Difference Between Treatment 0 and Treatment 1 16.00 6.00 14.00 4.00 12.00 10.00 2.00 8.00 0.00 6.00 0 5 10 15 20 25 30 4.00 -2.00 2.00 -4.00 0.00 0 5 10 15 20 25 30 -6.00 Random Experiment (sample) Random Experiment (sample) Notice the difference is > 0 for all random experiments… Notice the difference varies above and below zero… And the moving average converges toward zero! Bonus Excel functions FILTER Formulas INDEX Formulas https://youtu.be/1mHAVptUKAk https://youtu.be/F264FpBDX28 Analyzing ROI & Campaign Performance MKT46150: Lecture 8 Rocket Fuel What is Rocket Fuel interested in understanding? How effective is advertising for their client, TaskaBella Inc. Rocket Fuel Did Rocket Fuel utilize an experiment? Yes. A portion of consumers were randomly assigned to a control condition. What was the control? A generic PSA ad. What kind of experiment was this? Multivariate test design! Control Variation Rocket Fuel Why this design? Why A/B test design? Or more generally, why not test two different TaskaBella Ads? How do consumers become customers? Awareness: A Consumer must first Awareness become aware of a category, product, service, or brand Interest: The consumer becomes Interest interested in brand benefits Desire: The consumer develops a favorable attitude toward the brand Desire Action: The consumer forms a purchase intention and/or makes a purchase Action AIDA model of purchase decision-making Often referred to as a “funnel” The AIDA model is: Awareness Hierarchical Phases are ordered by the relative number of prospects Interest E.g., there are more potential customers at the awareness phase than the desire phase Desire Linear Consumers move from one stage to the next Action AIDA in action: Mammoth Mountain Ski Resort You see this billboard while driving to work (Awareness) Thinking about the billboard, you begin researching ski holidays (Interest) You compare other ski resorts to Mammoth Mountain (Desire) You book a ski holiday in California, and choose Mammoth Mountain (Action) AIDA in action: Grocery store shelves In the grocery store, a product catches your eye (Awareness) You compare the product to others on the shelf (Interest) You decide the flavor of the first product sounds good and unique (Desire) You add the product to your basket (Action)  How is this relevant to Rocket Fuel? Consumer decision journey 3. Brands added and removed during evaluation Active evaluation Initial 4. Brand selected and consideration purchased set 6. Repurchase via “loyalty loop” Trigger Moment of 2. Immediately recalled 5. Post-purchase purchase brands experience 1. Stimulus that initiates search “I want a snack”  Buy why? One-half the money I spend for advertising is wasted, but I have never been able to decide which half ~John Wannamaker, 1890s  Advertising pushes consumers down the marketing funnel! Advertising and the marketing funnel Awareness: Advertising helps create Awareness awareness of an otherwise unknown brand or product Interest: Advertising provides Interest information about the benefits of a brand Desire: Advertising shapes attitudes Desire toward a brand or product Action: Advertising encourages purchase through e.g., promotions Action  A purchase is generated without any advertising exposure! The organic marketing funnel Awareness: A consumer hears Awareness about a brand from a friend Interest: They seek out information directly from a brand Interest website Desire: A second friend recommends the brand Desire Action: The consumer decides to purchase based on immediate needs Action To measure campaign effectiveness, we must remove the baseline demand How do we attribute a conversion to our campaign? An existing customer Makes a purchase A consumer sees our ad And makes a purchase Establishing a baseline for comparison Year-on-year Experimental baseline Compare change over last Compare ad treatments to a year’s revenue for the same control or placebo period AKA, “ghost ads” However, such baselines are Relies on random assignment affected by: Controls for exogenous Industry seasonality influence through Offline paid media contemporaneous comparison Earned and owned media Market trends Rocket Fuel Was the advertising campaign effective? What was the KPI?  Establishing campaign effectiveness Utilize a Pivot Table What’s the lift? KPI: Conversions per customer AKA: Average conversions Row RowLabels Labels Count ofCount test ofSum of tot_impr Sum test Sum of ofconverted tot_impr Conversions per unique customer Sum of converted 0 23524 582481 420 0.0179 10 564577 23524 14014701 582481 14423 0.0255 420 1 Total Grand 588101 564577 14597182 14014701 14843 14423 Grand Total 588101 14597182 14843 What would happen if you repeated the experiment? Rocket Fuel Did additional consumers convert because of the ad campaign? How much money did TaskaBella make by running the campaign? Was the campaign profitable? What was the ROI? Calculating Return on Investment Basic formula: 𝑅𝑂𝐼 = 𝑁𝑒𝑡 𝑝𝑟𝑜𝑓𝑖𝑡 𝐼𝑛𝑣𝑒𝑠𝑡𝑚𝑒𝑛𝑡 Profit formula: 𝑃𝑟𝑜