SEAS 8500 Week 6: Model Deployment

• The presentation welcomes students to SEAS Online at George Washington University.

Audio Settings

• Students are instructed to mute their audio to eliminate background noise.
• To speak, students should click the hand icon and unmute the microphone when called upon.
• Students should remember to mute themselves after speaking.

Chat and Recordings

• Students can type questions in the chat.
• Recordings of each class session are provided for registered students' private use only.
• Releasing the class recordings is strictly prohibited.

Fundamentals of AI-Enabled Systems

• SEAS 8500 is a course at George Washington University
• Week 6 focuses on model deployment and prediction services.
• John Fossaceca is the instructor.
• The slides are adapted from material in Designing Machine Learning Systems by Chip Huyen.

Agenda

• Topics include deploying your model, ML deployment myths, batch vs. online prediction, model compression methods (low-rank factorization, knowledge distillation, pruning, quantization), and ML on the cloud and on the edge.

Deploying Your Model

• Deployment involves moving a model from development to production.
• "Production" means making the model accessible to end-users.
• Key steps include containerizing the model and its dependencies, deploying the container to a cloud platform, and exposing a prediction API endpoint.

Challenges in Deployment

• Challenges include monitoring performance, updating models, scaling to handle traffic, ensuring reliability and uptime, managing costs, securing access, compliance and regulations, and understanding metrics.

Deploying at Scale

• Scaling involves exposing an API endpoint for model predictions and having downstream apps send requests.
• Getting basic deployment to work is straightforward, but challenges arise with millions of users, millisecond latency requirements, and 99% uptime expectations.

Operating at Scale

• Operation at scale includes monitoring, alerting for problems, debugging root causes, and updating models seamlessly.
• Responsibilities can fall on model developers or a separate deployment team, involving high communication overhead, slower model updating, and harder debugging.

ML Model Deployment

• Understanding deployment gives insight into model constraints.
• Models should be tailored based on intended use.
• Predictions can be served online in real-time or in batch.
• Deployment location, whether device (edge) or cloud, impacts design.

Deployment Considerations

• Key consideration include common deployment myths, online vs. batch prediction, edge vs. cloud deployment, infrastructure considerations, user experience factors, and background knowledge gaps.

Machine Learning Deployment Myths

• Common myths are that deployment is easy, models perform the same as in development, and models only need to be deployed once.
• The reality is that deployment is more involved, performance degrades, and models require continuous monitoring and updating.
• Debunking these myths helps set realistic expectations.

Myth 1: Deploying One or Two Models

• Academia focuses on single models, but real applications use many.
• Real-world applications require different features, separate models per country/region, and other segmentation.
• Ridesharing apps use many models for demand forecasting, ETA, pricing, fraud, churn, and country-specific models.
• This can add up to hundreds or thousands of models.

Reality: Many Models

• Uber has thousands of models in production.
• Google trains thousands of models concurrently with billions of parameters.
• Booking.com has over 150 models.
• 41% of large companies have over 100 models in production.
• Infrastructure must support many models in parallel.
• Deploying only one model in isolation is no longer viable.

Myth 2: Consistent Performance

• Software performance degrades over time ("bit rot").
• ML models suffer from data distribution shift.
• Differences exist between training and production data.
• Model accuracy declines after deployment.
• Ongoing monitoring and updating is essential.
• "Set and forget" models are not viable in production.

Myth 3: Infrequent Updates

• Models require frequent updates because performance degrades over time.
• Models should be updated as fast as possible.
• DevOps best practices should be followed for frequent updates.
• Etsy deployed 50x per day, Netflix 1000s per day, and AWS every 11 seconds in 2015.

Reality: Continuous Updates

• Many companies update monthly or quarterly, but leaders update faster.
• Weibo updates certain models every 10 minutes.
• Alibaba, ByteDance (TikTok) iterate rapidly.
• There is a trend towards continuous deployment for Machine Learning.

Myth 4: Limited Scale

• Scale varies, often meaning hundreds of queries per second or millions of users per month.
• The misconception is that scale is only a concern for large companies.
• Most ML jobs are at large companies of 100+ employees, so ML roles are likely similar.

Preparing for Scale

• Industry ML jobs are likely at 100+ person companies.
• ML systems need scalability.
• Scale is no longer an exception.
• ML engineers should care about scale and apply scalable solutions upfront, as retrofitting later is difficult.

Batch vs. Online

• Key decision is how the model serves predictions.
• Batch prediction generates predictions asynchronously and is not latency-critical, using batch features.
• Online, real-time prediction requires the user to wait, is latency-sensitive, and uses streaming features.
• Factors include user experience needs, infrastructure constraints, throughput required, and data dependencies.

Batch Prediction Details

• Batch Prediction generates predictions periodically or on a trigger
• It is stored and retrieved on demand
• Also known as asynchronous or offline prediction, it's not latency-critical
• Useful for internal analytics
• Recommendations and segmentation computed nightly are examples of batch processing.

Online Prediction Details

• Online prediction generates predictions immediately on request.
• It is also called on-demand, real-time, or synchronous prediction.
• The User must waits for prediction, and it is latency sensitive.
• Requests are sent via REST API (HTTP requests) and common for user-facing apps

Key Differences

• Batch prediction happens periodically, such as every four hours.
• Online prediction occurs as soon as requests come.
• Batch processing is good for accumulating data when you don't need immediate results, like recommender systems.
• Online Prediction works where predictions are needed as soon as a data sample is generated (such as fraud detection), and this comes at a lower latency

Batch vs. Streaming Data

• Batch features are computed from historical data.
• Streaming features are computed from real-time data.
• Batch prediction uses only batch features.
• Online prediction can use both batch and streaming features.
• Delivery time estimation is an example, in batch is the restaurant's past prep time is used, and the streaming feature are current orders, and the availability of delivery people.

Streaming Prediction Architecture

• Streaming Prediction Architecture can combine both batch and streaming features.
• Another term is Streaming Prediction
• Retrieves batch features from databases and data warehouses.
• Streaming characteristics are computed from real-time data.
• Batch Prediction suits popular queries while online predictions suit long tail of a query

Online vs. Batch Prediction - Use Cases

• Practical application, Food Delivery Platforms: Curates restaurant suggestions (batch) while Recommendations dishes once a specific restaurant is selected (online prediction).
• Common Perception: Online prediction lags in cost & is less performant however Reality: Efficiency varies.
• Out of 31 million Grubhub users, only 622,000 placed daily orders. Predicting for all would squander 98% of computational resources

From Batch to Online Prediction

• The appropriate type of Prediction depends on the decisions and how users experience those decisions
• Batch prediction and Online prediction are described, their pos and cons
• Their purpose is to fit into the larger picture of Machine Learning

Online vs. Batch Prediction - Trade Offs

• Online Prediction Pros - Instantaneous Insight: Intuitive for academicians and researchers. Fast Prototyping
• Online Prediction Trade offs: Cloud solutions like Amazon SageMaker and Google App Engine do readily expose endpoints for real-time predictions.
• Batch Prediction Pros - Predict Now, Use Later:. Efficient at Scale. and low Latency.
• Batch Prediction Trade offs: Predictions are calculated beforehand, Efficiency at Scale, retrieval is often quicker than real-time

Constraints of Batch Prediction

• Adaptability Hurdles: Batch Struggle with swift adaptability to evolving user behaviors, if one's recent viewings shift genres, recommendations may remain static.
• Challenges w/ Dynamic Queries: for instance, real-time translation services can't predict every conceivable sentence or phrase.
• There needs to be Instantaneous reactions in high-frequency trading and instant fraud detection

Towards Enhanced Online Prediction

• Hardware Innovations & Algorithmic: These two are making online predictions faster and more cost-effective while the Journey to become the standard.
• Strategic Corporate Investments - Many are pivoting towards online prediction to enhance user experience and decision accuracy. and so
• Must counteract Latency in real time by: Real-time Processing Pipeline & Rapid-response Models.

Unifying Batch Pipeline & Streaming Pipeline

• Batch is from the older days of computing, as historically dominant tools that enabled efficient periodic processing of large datasets through batch systems for Machine Learning.
• Now, their a need for real time responsiveness with streaming, its crucial for seperate pipelines now and the architecture is indispensible.
• Navigation is a Concrete Example from Google Maps to predict route timings for batch and real time by adjusting in real time for new changes
• Major Challenge - Running separate pipelines can lead to maintenance overhead and inconsistencies

Dual nature of processing in real time

• Dynamic model is good for a platforms arrival and so continual corrections can adjust along the way with real time data
• There is feature capture to consider with leveraging historical data and model training.
• The dual nature of speeds while driving are helpful too, as batch helps with the historical data but real time is critical for up to the minute traffic
• The implementation is to be aware of code bugs and its helpful to share code, collaborate more and to do more testing

Integrating Stream & Batch Processing

• ML is about the cohesive integration of stream and batch processing through transformations to bridge the gap.
• Avoid discrepancies, and do feature stores for uniform features.
• Benefit - allows businesses to scale their ML models and remain adaptive

Data piplines

• Data pipelines are designed to stream data to preform research and are also used for inference.

Model Compression

• The goal is to create Real-machine learning in applications for speed, but can be slow at speeds. Therefore the trade off is to create better accurancy and speeds.
• Stategies to enhance - Enhance Speed, hardware, Model Compression and reduceing the accuracy

Model compression (techniques)

• low rank optimization (simplifies weight matrices)
• knowledge distillation ('student' model learns from 'teacher' model)
• pruning (removes unneeded weights/neurons), quantization (reduces numerical precision & requirements)

low rank factorization

• with high-dimensional tensors, important to filter redundancy. and is done to simplify tensors with lower dimensions.
• its helps with speed and memory.

Factorization

• Its about avoiding parameterization by having filters. To transition from smaller areas to larger areas. the drastic reduction is in reducing the number of parameters.

Case studies for models to have fewer parameters

• squeezenets with alexnet with much smaller parameters
• Mobilenets which breaks down convolutions

Knowledge Distillation

• lighter a student model is trained to replicate the behavior of a more complex teacher model.
• These are trained quickly at the same time.
• For example, DistilBert retains 97% the the BERT models data

Benefits and negatives

• Distillation is flexible in that a teacher can pass what its teaches to a random place, but can preforn badly with a specific model design. which is why its not widely used.

ٍSparsity

• is helped through parameter and notewide pruning. so in summary - Pruning can create computational efficiency in model structures

Pruning Effectiveness

• Is effective although it can induce biases. also weights and Architecture is debatable with what to prune
• Can also retrain to capture losses through various testing and methods

Quantization

• A method to compress ML models by representing model parameters with fewer bits to represent data well. This all reduces the number of memory required
• Default systems store data in certain ways . And types of quantization. All this saves power and increase efficiency

Challenges in Quantization

• The problems is that bit representation can be lost with numbers, as well as with training . However NVIDIA and TPU have special sauce to alleviate those concerns.
• Therefore industry tools must provide with what needs to take advantage of minimal code with mobile devices

Edge verses cloud

• Its about what hardware and code has access to versus local machines, its the difference between performance as well are internet connectivivty issues

Cloud vs. edge advantages

• Scalability, Resource Pooling and Maintenance
• Data locality is essential

Hardware at the Edge

• Tailored Hardware Design is essential and manufacturers are looking forward to ML operations.
• Tensor Processing Units, The Neural Engine , and startups are all focused with energy in mind

Challenges

• The challenges across diverse hardware is due to the differnence and what to accelorate. The way forward is to have the unioforms be on optimized versions

Hardware and Software Mapping

• Software synergies are paramount. Its analogous of an application needing operating support and should work in harmony

Hardware and Software Mapping

• The challenge is for different code components all work together in an efficient environment

Process Bridging

• Wrong with Direct Mapping?: Multiple frameworks and many hardware backends make direct compatibility to impossible,
• The solution is to have intermediaries across all components and backends and simplifies new integrations to be more feasible

Process Bridging

• The way to have it low from high, so code works for harward natives after sequence operations. From computation to the actual blueprints for usage

Graphs and Visuals

• The components are visual with arrows to help under stand flow and gradient

Model Optimization

• Its about speed verses accurecy. and achieving great power through the right performance is critical.

How to make code components

• Code components must bridge together and have network adpatation.

ٍScalability

• To save power, its essential to vectorize and parallelize for speed

Scaling Efficiency

• There must have a loop and operator set together is great for optimization

Scaling Efficiency challenges

• Local verses scaling is needed to reduce restriction due to bad architeture

Optimization challenges

• Its about optimizing is a complex task with AI that may need to perform real time processing

Optimization

• Its about the human expertise
• The issue with machine code is that its not well known with frameworks such as cuDNN and autoTVM. This code only optimizes parts and not entire pieces.

autoTVM

• Is only in use if things are working with a broader scope as to adapating strategies

autoTVM

• It works with each subgraph with various processes to make the coding work

Code Efficiency

• There tradeoffs to account for in compilation is more important that it be optimized and coded more efficient.

Is it in the browser

• Its great to have the code in the browser for ease of access as to be device independent.
• Its aslo great to use JavaScript for code, TensorFlow or synapses

Code limitation - JavaScript

• JavaScript although popular, code does not run effectively

Its all about WASM now

• Its made so everyone can access as a quick stack method

Benefit - WASM

• performance of running code faster and supports many components like javascript

Cons WASM

• still has growing constraints and some features are being developed.

All together

• Be strategic and prepare to check that it may work