6 Build on Tiny Bets from Slides

• Agenda items include:
  • AI Investigation Recommendations
  • Conducting your own AI Investigation
• Stages to investigate AI include:
  • Stay Calm and Focus on the Problem space
  • Feel their Pains, Learn their Gains
  • Stake Out an AI Playing Field
  • Stand on the Shoulders of Giants
  • Unpack Assumptions & Unknowns
  • Build on Tiny Bets for Big Wins
  • Socialize an Outcomes Blueprint
• Decisions can be treated as explicit bets to avoid emotional traps and to learn and seek the truth.
• Bets have a payout, with an expectation of meaningful progress at the end of the cycle.
• Teams get uninterrupted time to work on selected projects through bets.
• Bets have a cap on downside; for example, the maximum loss is the time allocated (6 weeks.)
• +20% of new products or features have a big impact.
• 60% have little or no lift, and -20% hurt the business
• The most expensive way to test your idea is to build production quality software.

Small AI Bets

• Low upfront investments
• Limited scope
• Quick to prototype
• Low impact if it fails
• Easy to scale

Big AI Bets

• High upfront investment
• Broad scope
• Longer development cycle
• High impact if it fails
• Complex to scale

AI "Haul" of Shame

• Failures happen when hype and developed capabilities lead to unmet expectations and market misfit.
• Anki overbuilt without validating the demand for consumer robots and lost approximately $200 million.
• Jibo had repeated delays and functionality overpromises eroded confidence and lost approximately $73 million in 6 years.
• IBM Watson overpromised on AI decision-making; couldn't overcome data, regulatory, and customer adoption hurdles, and lost about $4 billion in 6 years.
• Rethink Robotics assumed robots could meet high precision demands, but performance fell short with loses of $149.5 million over 10 years.
• Walmart's Bossa Nova Robotics overestimated robots' abilities to match human workers in efficiency leading to ~$65M wasted for 5 years.
• Lowe's LoweBot underdelivered on AI's customer service potential and low adoption and lost around $30M with 2 years invested
• Utrip overbuilt a travel AI solution without a sustainable revenue model costing at least $4 million over 7 years.
• Aria Insights pivoted too late and struggled to scale with losses of $39 million in funding over 11 years.
• Bloomberg's LLMs overinvested and failed to scale keeping up with broad innovations, specifically GPT-40 and lost approximately 1.3 GPU hours

Starting Small

• Starting with problem you're solving, then ask key questions for AI development.
• Questions include:
  • Have the right data?
  • Use a prebuilt or custom model?
  • Use a private or public model?
  • Leverage existing APIs?
• Stay flexible in a dynamic environment.
• Involve dev and data teams.

Fight to Keep the Scope of POCs TIny

• Scope the Pilot:
  • Pick the Outcome
  • Pick the Data
• Set the Foundation:
  • Problem Focus
  • Relevant Data
• Execute the Pilot:
  • Run a Simulation
  • Run a Limited Trial
• Evaluate and Expand:
  • Quantify ROI
  • Socialize Outcomes
• Scale Responsibly:
  • Prioritize Data
  • Land and Expand
• Product discovery separates good ideas from bad ideas creating a validated product backlog.
• Elements of Value Include: Learn, Data, Valuable, Feasible, Viable, Measure, Build Product.

Get the Right People

• Ask engineers & DevOps about scope, effort, and impact.
• Use simple metaphors when translating complexities to stakeholders.
• Clarify expectations on timelines, costs, and resources.
• Base discussions on informed, validated assumptions.
• Collaborate with finance when defining budgetary amounts.
• Link past successes to predict future successes.

Risks & Assumptions

• Minimize risk by validating assumptions early.
• Choose strategies that allow learning without heavy commitment.
• Buffer validated demand before through a landing page before building
• Webvan based their assumptions on untested growth and overbuilt the infrastructure
• Amazon used A/B testing to minimize impact of bad features.
• Solution Hypothesis proposes a solution and predicts its impact with TAD experiments to validate.
• Create measurable success metrics and a deadline to effectively assess the hypothesis

AI & Tiny Acts Discovery (TADs)

• AI TADs for Viability (should we do this?)
  • Conduct Interviews with users, and use synthetic data to explore simulated behaviors, alongside data mining for feature support and traits.
  • Demonstrate likely outcomes to potential customers using market sizing with CoPilot, and through the Monte Carlo test.
• AI TADs for Desirability (do people want this?)
  • Al User Journeys: Generate user scenarios with Al.
  • Conduct quick, informal Guerrilla Interviews
  • Use Al for Social Listening with social media
  • Analyze Data and do short Observations

Sample Success Metrics

• Quick to Assess: Speedy evaluation for swift decisions.
• Minimal Resources: Requires few resources to test.
• Clear Direction: proceed, pivot, or stop.
• User-Focused, and offers initial Early Proof
• Some measures include demo sign-ups, quick feedback, early use, willing users and usability insights.

Framing Your Bet as a Hypothesis

• Create a solution using "If... then..." format.
• Create a plan to test the hypothesis, like a prototype or an user story.
• Define specific target metrics to validate the hypothesis with experiments.

Validating through Lo-fi experimentation

• Get valuable feedback quickly and cheaply
• Ways to experiment include: concierge tests, wireframes, "Wizard of Oz" testing, explainer vids, landing pages.

Storyboarding

• Tell the story to help solve the problem.
• Illustrate the solution with a storyboard
• Use images and text to create the storyboard
• Characteristics include Authenticity, Simplicty, Emotion

Delivery Costs

• Includes various costs for Machine Learning (ML) and Large Language Models (LLMs)
• ML data requires:
  • Data Acquisition = $10,500 - $70,000
  • Data Annotation = up to $70,000
  • Infrastructure = $100 - $30K/month
  • Compute - varies
  • Model = varies
  • Training - $10k+
  • Fine Tuning = varies
  • Personnel Vary
• Large Language data model costs:
• Data Acquisition $100,000+ and includes costs for data annotation
• Infrastructure costs $200k+
• Training is millions of dollars.

Discovery Costs

• Iterate/feedback driven cost curve validates the right thing to build with minimal cost from a Pretotype to a Protype.