5 Unpack AI Assumptions and Unknowns from Slides

• The presentation discusses the importance of data in AI solutions, AI considerations, and shaping winning solutions.
• The Virtuous Cycle of Data drives continuous growth and competitive advantage.

Agenda topics:

• Stay Calm and Focus on the Problem Space
• Feel their Pains, Learn their Gains
• Stake Out an AI Playing Field
• Stand on the Shoulders of Giants
• Unpack Assumptions & Unknowns
• Build on Tiny Bets for Big Wins
• Socialize an Outcomes Blueprint

The Nature of Data in AI:

• AI's reliance on data is a key concept.
• Amazon's product recommendations are an AI data success story; algorithms provide relevant product suggestions by leveraging vast amounts of structured and semi-structured data including user browsing history, purchase records, product ratings, and reviews.
• IBM Watson Health's oncology algorithm is an AI data failure story; the algorithm was trained on a limited and biased dataset, primarily consisting of synthetic cancer cases and treatment guidelines from a small number of US hospitals, leading to questionable treatment recommendations.

Creating a Virtuous Cycle of Data:

• More Data should be collected from additional user/system data.
• Better Algorithms which improve AI Performance are important.
• Better Service enhance personalization/performance.
• More Usage increases customer/system engagement.

Data Monetization:

• Direct Monetization involves selling AI-driven data insights such as Healthcare organizations selling anonymized data for public health research.
• Indirect Monetization uses AI to improve training programs and personalize content such as Coursera using learner data to offer tailored training recommendations.
• Data-Driven Products involves building AI tools for disaster response and training, IBM Watson Health offers predictive disaster models to aid organizations like the Red Cross.

Data, AI, & the Product Manager focus on:

• Keeping data fresh
• Training the model
• Avoiding bias
• Monitoring everything
• Supporting scalability
• Avoiding fines & penalties

Data Coverage & Freshness:

• The key questions to ask are if data satisfies customer problems, if outdated data is hurting AI effectiveness, and if more data sources or licenses are needed.

Data Completeness & Quality measures:

• Volume: Is there enough data?
• Variety: Is the data diverse?
• Velocity: How fast do we get data?
• Veracity: Can we trust our data?
• Value: Is the data relevant?
• Cleanliness: Is the data clean & organized?
• Labeling: Is our data properly marked for learning?
• Ethical Use: Are we using data responsibly?

Training Data Decisions:

• Questions to consider: Do we need to featurize data, and if so, how extensively? Should we use synthetic data to fill gaps? Is it worth licensing external data for better training? How do we integrate disparate data sources without creating silos?

Bias & Ethics:

• Examples:
• IBM builds diverse datasets to minimize bias in facial recognition.
• Microsoft conducts bias audits in its AI systems to ensure fairness.
• Amazon scrapped its AI hiring tool due to outdated gender bias in historical data.

Model Performance:

• The goal is to monitor AI model effectiveness to ensure customer needs are continuously met.
• Lemonade's AI chatbot settling an insurance in 3 seconds and Microsoft's Tay Chatbot which created customer issues by generating offensive content are examples of bad models.
• Google Flu Trends which overestimated flue outbreaks because of model drift are examples of bad usage of models.

Scalability & Data Pipelines:

• Uber's AI ride-matching system handles millions of real-time updates supporting global scalability.
• LinkedIn ensures reliable data flow by integrating multiple sources and fueling AI recommendations.
• Facebook eliminated data analysis bottlenecks with a tool that speeds up AI-driven insights.

Data Pipelines:

• Understanding the role of data science and data engineering is important.
• Data pipelines involve: Source -> Extract -> Transform -> Load -> Analyze
• They also involve presentation data and warehousing/storage

Data Privacy & Compliance:

• The goal is to monitor compliance and minimize risks related to customer data usage in AI.
• Key Questions: Is customer data compliant with regulations, are we legally using customer data, are there privacy concerns, and is customer data being misused in AI?

Data Integrity & Security Standards:

• ISO/IEC 25012: Data quality management.
• SOC 2: Data security and privacy.
• GDPR: EU data privacy regulation.
• HIPAA/PCI DSS: Healthcare & payment data protection.
• ISO/IEC 27001: Information security management.
• ISO/IEC 20000: IT service management.

AI Model Performance & Fairness Standards:

• ISO/IEC 23053: AI training guidelines.
• IEEE 7010: Fairness in AI decision-making.
• AI Fairness 360: Bias mitigation toolkit.
• ISO/IEC 23894: AI performance guidelines.
• ISO/IEC 25010: Software quality and performance.
• TPC: Database and pipeline performance benchmarks.

Before building, ask:

• Is this valuable for the customer?
• Is this a viable business for the company?
• Is this effort feasible with our current organization?
• The product market fit where viable and feasible is the value brought to a company by the product managers.

Valuable Considerations:

• How does it solve a customer problem?
• How does it add significant value?
• Do our customers want it?
• Can it add competitive differentiation?
• Can we acquire and retain customers?

Viable Considerations:

• What are our potential business models?
• Will it be profitable?
• Costs: Setup, Compute Resources, & Energy Consumption
• How big is the market opportunity?

Feasible Considerations:

• Data Sets, Data Quality, and Training Bias
• Access to Skills and IP
• Platform/API dependencies
• Evolving industry Standards
• How do we ensure Data Privacy?
• How do we manage Ethical Considerations including Bias, Transparency, Safety, and Keeping Humans in the Loop, Equity
• Are we covering Social & Cultural issues?
• Are we equipped for Accountability of Errors (hallucinations/fictions) and Discrimination
• What are we doing to incorporate Legal & Policy
• Examples of questions involve valuable, viable, and feasible questions.

Compliance & Ethics:

• The goal is to address compliance from the start, meeting standards without complicating or delaying the solution.
• Apple integrates privacy by design, Microsoft created an AI ethics committee (AETHER), and IBM opened sourced AI Fairness 360.

Complexity & Team Expertise:

• OpenAI's GPT-3 requires diverse skills leading to new roles like prompt engineers.
• Uber's shift to microservices added initial debt but provided long-term scalability.
• Amazon's A/B testing accelerates experiments but increases complexity.

External Factors Impacting AI:

• PESTel framework provides a structured approach to categorization and responding to the risks external factors might present.
• Market conditions don't just change; they can potentially change us, our business, and our products.
• In each category you must consider what do I act on? what do I watch?
• The 6 hypothetical events can effect your AI such as EU demanding AI Transparency and Funding Disappearing.