Week 5: The Surprising Power of Online Experiments (Kohavi & Thomke, 2017) PDF

Stuvia - Koop en Verkoop de Beste Samenvattingen Week 5: The Surprising Power of Online Experiments: Getting the Most Out of A/B and Other Controlled Tests (Kohavi & Thomke, 2017) Start-ups and companies without digital roots have discovered that an “experiment with everything” approach has surpris...

Stuvia - Koop en Verkoop de Beste Samenvattingen Week 5: The Surprising Power of Online Experiments: Getting the Most Out of A/B and Other Controlled Tests (Kohavi & Thomke, 2017) Start-ups and companies without digital roots have discovered that an “experiment with everything” approach has surprisingly large payoFs. - - When a Irm develops the soLware infrastructure and organizaonal skills to execute them, it will be able to assess not only ideas for websites, but also potenal business models, strategies, products, services, and markeng campaigns - all inexpensively However, too many organizaons go about conducng experiments haphazardly, do not know how to conduct scienIc tests, or conduct far too few. The value of the A/B tests A/B tes ng = the experimenter sets up two experiences: "A" the control (usually the current system and is considered the "champion"), and "B" the treatment, which is a modiIcaon that tries to improve something (the "challenger") This can be: - Online, the modiIcaon could be a new feature, a change to the user interface A dierent business model: such as an oFer of free shipping. Whatever aspect of operaons companies care most about—be it sales, repeat usage, click-through rates, or me users spend on a site—they can use online A/B tests to learn how to opmize it. The ability to access large customer samples, to automacally collect massive amounts of data of user interacon, and to run simultaneous experiments, gives companies the ability to evaluate many ideas quickly, with great precision, and at negligible cost. What managers need to understand: 1. Tiny changes can have a signicant impact A greater investment does not bring a larger impact: Online, where success is more about ge9ng many minor changes right. 2. Experiments can guide investment decisions Online tests can help managers Igure out how much investment in a potenal improvement is opmal. Build a large-scale capability - The majority of new ideas fail in experiments. All this goes to show that companies need to kiss a lot of frogs (that is, perform a massive number of experiments) to Ind a prince. 18 Gedownload door: matsmolenberg | [email protected] Dit document is auteursrechtelijk beschermd, het verspreiden van dit document is strafbaar. ¤ 912 per jaar extra verdienen? Stuvia - Koop en Verkoop de Beste Samenvattingen A company’s experimentaon employees can be organized in three ways: 1. Centralized model In this approach a team of data sciensts serve the enre company - Advantages: they can focus on long-term projects, such as building beEer experimentaon tools and developing more-advanced stascal algorithms. Disadvantages: the business units using the group may have diFerent priories, which could lead to conicts over the allocaon of resources and costs. Another con is that data sciensts may feel like outsiders when dealing with the businesses and thus be less aEuned to the units’ goals and domain knowledge, which could make it harder for them to connect the dots and share relevant insights 2. Decentralized model Distribung data sciensts throughout the diFerent business units. - The beneIt of this model is that the data sciensts can become experts in each business domain. The main disadvantage is the lack of a clear career path for these professionals, who also may not receive peer feedback and mentoring that help them develop. 3. Centre of excellence model Data sciensts in a centralized funcon and others within the diFerent business units (Like MicrosoL) - - Advantage: A centre of excellence focuses mostly on the design, execuon, and analysis of controlled experiments. It signiIcantly lowers the me and resources those tasks require by building a companywide experimentaon plaAorm and related tools. Disadvantage: lack of clarity about what the centre of excellence owns and what the product teams own, who should pay for hiring more data sciensts, and who is responsible for investments in alerts. Small companies typically start with a centralized model In companies with mulple businesses, managers who consider tesng a priority may not want to wait unl corporate leaders develop a coordinated organizaonal approach; in those cases, a decentralized model might make sense, at least in the beginning. And if online experimentaon is a corporate priority, a company may want to build experse and develop standards in a central unit before rolling them out in the business units. Address the denion of success Every business group must deIne a suitable evaluaon metric for experiments that aligns with its strategic goals. 19 Gedownload door: matsmolenberg | [email protected] Dit document is auteursrechtelijk beschermd, het verspreiden van dit document is strafbaar. ¤ 912 per jaar extra verdienen? Stuvia - Koop en Verkoop de Beste Samenvattingen Idenfying which short-term Igures are the best predictor of long-term results is diRcult. Ge9ng it right – coming up with an overall evaluaon criterion (OEC) – takes oLen extensive internal debate and requires close cooperaon between senior execuves who understand the strategy and data analysts who understand metrics. Looking at diFerent metrics is crucial because it helps teams discover whether an experiment has an unancipated impact on another area. Beware of low-quality data - Use A/A tes ng = tesng something against itself to ensure that about 95% of the me the system correctly idenIes no stascally signiIcant diFerence. The best data sciensts are scepcs and follow Twyman’s law: any Igure that looks interesng or diFerent is usually wrong. Managers should also be aware when some segments experience much larger or smaller eFects than others (= heterogeneous treatment eects) In certain cases, a single good or bad segment can skew the average enough to invalidate the overall results. Experimenters looking at an average eFect may dismiss a good idea as a bad one. Avoid assumpons about causality - - Because of the hype over big data, some execuves mistakenly believe that causality isn’t important. In their minds all they need to do is establish correlaon, and causality can be inferred = Wrong! Observaonal studies cannot establish causality. Including too many variables in tests also makes it hard to learn about causality. Example: Yahoo did to assess whether display ads for a brand, shown on Yahoo sites, could increase searches for the brand name or related keywords. The observaonal part of the study esmated that the ads increased the number of searches by 871% to 1,198%. But when Yahoo ran a controlled experiment, the increase was only 5.4%. If not for the control, the company might have concluded that the ads had an enormous impact and wouldn’t have realized that the increase in searches was due to other variables that changed during the observaon period. The online world is oLen viewed as turbulent and full of peril, but controlled experiments can help us navigate it. They can point us in the right direcon when answers are not obvious, or people have conicng opinions or are uncertain about the value of an idea. Finally, to really understand the value of an experiment, look at the diFerence between the expected result and the actual result. - If you thought something was going to happen and it did, you haven't learned much. If it did not happen, you learned something important (posive or negave). By combining the power of soLware with the scienIc rigor of controlled experiments, your company can create a learning lab. The returns you reap—in cost savings, new revenue, and improved user experience—can be huge. If you want to gain a compeve advantage, your Irm should build an experimentaon capability and master the science of conducng online tests 20 Gedownload door: matsmolenberg | [email protected] Dit document is auteursrechtelijk beschermd, het verspreiden van dit document is strafbaar. ¤ 912 per jaar extra verdienen?