4%20MLOps%20Canvas.pdf

Human Centric ML Ops Canvas User Name: Use Case Name Team Name: Canvas Date: Case Description MLOps: MLOps unifies machine learning development and operations, emphasizing seamless integration and deployment of ML models in production environments. Human-Centric MLOps: Human-Centric MLOps emph...

Human Centric ML Ops Canvas User Name: Use Case Name Team Name: Canvas Date: Case Description MLOps: MLOps unifies machine learning development and operations, emphasizing seamless integration and deployment of ML models in production environments. Human-Centric MLOps: Human-Centric MLOps emphasizes how AI fits into human workflows and behaviors, ensuring models align with organizational needs and are adoptable by users. Comment on the Main Difference: While MLOps focuses on efficient ML deployment, Human-Centric MLOps ensures AI integrates well with human needs and organizational processes. Scoping Data Modelling Deployment Usage Monitoring This is the phase where we clearly define what problem we want to solve using AI. Think of it like choosing the destination for a journey. We need to ensure that the goal is achievable, beneficial to the organization, and well-understood by everyone involved. In the world of AI, data is like the fuel for our car. Before we start our journey (or build our AI model), we need to ensure we have the right kind of fuel and enough of it. This phase involves collecting relevant data and preparing it for use. This is where the magic happens. We take our prepared data and use it to train an AI model. Think of it like choosing the best car for our journey, based on the terrain and destination. Once our AI model is ready, it's time to put it into action in the real world. This phase is like driving our car out of the garage and onto the roads. We integrate the AI system into our existing operations, ensuring it's accessible to users. Now that our AI system is in operation, we need to ensure people are using it effectively. It's like ensuring drivers and passengers know how to use the car's features and follow the rules of the road. This phase emphasizes training and adaptation. Like regularly servicing our car, we need to keep an eye on our AI system to ensure it runs smoothly. This phase involves checking its performance, ensuring it's still solving the problem effectively, and making necessary adjustments. Why it's important: Why it's important: Just as you'd choose a rugged SUV for a mountain trip and not a sports car, selecting or building the right AI model ensures we effectively address the problem we set out to solve. Why it's important: Even the best AI system can't add value if people don't use it correctly or understand its benefits. Without a clear destination, we might end up building an AI system that doesn't address any real-world problem or need. Problem Definition: Just as a car can't run without fuel, AI models can't function without data. The better the quality of our data, the more efficient and accurate our AI system will be. Acquiring Data: What specific challenge or opportunity are we aiming to address with AI? What kind of data do we need to address our defined problem or opportunity? How does this problem or opportunity align with our broader organizational goals or needs? Which data sources are currently available internally within our organization? Who are the stakeholders or departments responsible for this data? What existing solutions or processes are in place, and how does AI offer a unique or improved approach? If internal data is insufficient, what external data sources can we consider? Are there third-party providers or public datasets that could be relevant? Why it's important: Requirements & Expectations: An AI model that remains unused is like a car that's never driven—it's a wasted resource. Deployment ensures our AI-driven solutions reach the people who need them. Integration & Interaction: What specific requirements do we have for our model? For instance, do we need it to be highly explainable for regulatory or user trust reasons? How will the AI model be integrated into our existing systems or platforms? Are there APIs or other interfaces that need to be developed or adapted? How accurate does our model need to be? Is there a minimum performance threshold that it should meet? With which systems, databases, or services will the deployed model interact? Are there any specific compatibility considerations or technical constraints to address? Are there specific considerations regarding false positives or false negatives? How critical would it be if the model makes a mistake, and what could be the potential repercussions? Why it's important: People What specific objectives are we aiming to achieve? Are there any ethical or regulatory considerations we should be aware of when acquiring external data? How can we ensure that data acquisition respects privacy and has proper consent? Ownership: Who are the decision-makers? What is the vision for this AI solution in the organization? What is the allocated budget for the deployment, including hidden costs? The world changes, and so does data. Continuous monitoring ensures our AI system remains relevant, accurate, and beneficial, just as servicing ensures our car remains roadworthy. Responsibility and Management Decision-Makers: Before Deployment: Why it's important: Which team or department will own the responsibility for managing and monitoring the AI product post-deployment? During Deployment: What metrics will be used to measure ROI during the implementation phase? What are the major milestones in the deployment roadmap? After Deployment: Are there specific roles, like AI product managers or MLOps engineers, within the team to oversee this? Is the solution achieving its intended objectives? How does the maintenance cost compare to the forecasted budget? Communication: How will the responsible team communicate findings and updates related to the AI product to other stakeholders in the organization? What is the process for escalating any critical issues that arise during monitoring? Objective Setting: User Experience & Accessibility: What are the specific, measurable outcomes we aim to achieve with our AI solution? How will end-users interact with the deployed model? Is it through a web application, mobile app, or some other interface? How will we measure the success or impact of the AI implementation? What will the user experience look like? How can we ensure that it's intuitive, efficient, and satisfying for the user? Are these objectives both short-term and long-term, and how might they evolve over time? Users: Who are the users? Before Deployment: During Deployment: After Deployment: What does the current workflow look like? What kind of training will facilitate smoother transition and adoption? How intuitive is the AI solution for end-users? Which aspects of the current system are pain points? How will we ensure minimal disruption during transition? Are there any emerging challenges or pain points post-deployment? Internal Capabilities: Do we have access to the necessary data to train and validate an AI model for this objective? Does our current technical infrastructure support the development, training, and deployment of an AI solution? How will changes in data distribution (data drift) be detected and managed? If not, do we need to consider external contractors or consultants? How will we evaluate and choose the right external partners? Does our organization have the necessary infrastructure and tools to support model development, training, and testing? User Feedback: Once we've identified our data sources, how will we collate and consolidate this data into a usable format? Feasibility Analysis: Data-Driven Feedback: Is there a mechanism in place to automatically retrain or flag the model if it starts performing below a certain threshold? Do we have internal teams or departments with the expertise to build and train this model? Preparing Data: Feedback Collection How will user feedback be captured post-deployment? People Affected: Is there a structured way for users to report any anomalies, errors, or challenges they face while interacting with the AI system? Who are the people affected? Are there any immediate quality issues with the data, such as missing values, duplicates, or inconsistencies, that need addressing? Technical Skills & Resources: Do we have a clear understanding of what each data field or feature represents? If not, who in the organization can provide clarity or context? How will we divide the data to evaluate our AI solution's performance effectively, considering we might want a portion of the data as a reference for its success? What technical skills are needed for deployment? Which roles will be most impacted by the introduction of this AI solution? Are there specific programming languages or platforms that our team needs to be proficient in? Are there any concerns or apprehensions among the affected parties? Do we have the necessary expertise in-house? If not, do we need to hire new talent or collaborate with external partners? What are the anticipated benefits of pursuing an AI approach compared to other potential solutions? Before Deployment: During Deployment: After Deployment: How will we maintain transparent communication with affected parties? Have roles been positively or negatively impacted post-deployment? What mechanisms will be in place to capture feedback? Were initial concerns effectively addressed? Process Impact: How will the system's impact on existing workflows and processes be assessed continuously? Are there any KPIs (Key Performance Indicators) set up to measure the tangible benefits of the AI system on organizational processes? Research & Existing Solutions: Are there existing models or solutions available, either within our organization or externally, that address similar problems? Can these be adapted or fine-tuned for our use case? Processes Has someone documented building a similar model in research papers, case studies, or online platforms? Can we leverage insights or findings from these existing works to expedite our model development or to set performance benchmarks? Stakeholder Engagement: Before Deployment: Hardware & Software Considerations: Who are the critical stakeholders for this AI project, both internal and external? Is the deployment purely software-based, or is there a hardware component involved (e.g., IoT devices, sensors, edge devices)? How can we ensure consistent communication and collaboration among these stakeholders throughout the AI lifecycle? If there's a hardware aspect, how will it be sourced, installed, and maintained? Are there any specific environmental or logistical considerations to account for? What concerns or input might these stakeholders have, and how can we address or integrate their feedback? General Data Availability and Management: How will the AI solution integrate into our existing workflows? Which specific processes stand to benefit the most from the AI's capabilities? During Deployment: Are there any processes that need to be temporarily adjusted or halted during implementation? How will we ensure a seamless integration of the AI solution into ongoing processes? System Monitoring After Deployment: How have workflows evolved with the AI solution in place? Are there any unforeseen process bottlenecks that have arisen post-deployment? Do we have a centralized system or platform within our organization where data is stored and managed? If so, is this system accessible for our AI project? Who are the gatekeepers or stakeholders responsible for data management within our organization? Do we need to get permissions or collaborate with them for our AI initiative? Are there established data management practices or protocols within our organization that we should be aware of or align with? Expectation Management: How are we communicating the potential outcomes and limitations of the AI project to all stakeholders? What is our estimated timeline for each phase of the AI lifecycle, from scoping to monitoring? If external data is being considered, how will it be integrated with our internal data? Are there compatibility or format issues we should be mindful of? Objectives of System Monitoring: How will we measure and ensure the reliability of the AI system over time? What safeguards are in place to maintain the security of the AI solution, especially concerning user data and proprietary information? What benchmarks are set up to ensure the AI system is performing optimally? Key Areas to Monitor: Technical Glitches: How will we monitor for hardware and software glitches that could impair the AI system's functioning? Is there an alert system in place for immediate notification of any technical issues? Model Selection & Evaluation: Based on our problem definition and requirements, what types of machine learning models might be appropriate? (e.g., regression, classification, clustering) Which metrics will we use to evaluate our model's performance? (e.g., accuracy, precision, recall, F1 score) How will we handle trade-offs? For instance, if achieving higher accuracy compromises model explainability, how will we prioritize? User Behavior: What metrics and tools will we use to understand how users are interacting with the AI system? Roll-out Value & Impact: What is the expected value or impact of the deployed model? How does it align with our broader business or organizational objectives? How will we measure the success of the deployment? What metrics or KPIs will be used? Test Pilots: Who are the ideal candidates to be our first test pilots for the AI solution? What criteria make them suitable for this initial phase? Roll-out Strategy: How will user feedback be used to enhance and refine the AI system over time? Feedback Mechanism: How will the roll-out be structured in phases? How will we capture feedback during the initial roll-out? What are the benchmarks for moving from one phase to the next? What mechanisms are in place to quickly act upon the feedback received? Security: How will potential security breaches be detected? What mechanisms are in place to ensure data integrity and prevent unauthorized access? How will we manage and address potential shifts in expectations or project objectives as we progress? Continuous Improvement: Based on the findings from monitoring, how often will the AI system be updated or refined? Is there a pipeline in place for continuous integration and continuous deployment (CI/CD) for the AI solution? © Tristan Post (2023), [email protected]