AI/ML Systems and Data Engineering Fundamentals

Lecture 1: AI/ML Systems Fundamentals and Data Engineering 101

Introductions

• Course attendees introduce themselves by stating their name, background/profession, what they hope to get out of the course, and any fun thing to share.

ML in Production: Expectation vs Reality

• Ideally, deploying ML models should be straightforward, beginning with collecting data, training a model via ML algorithms, and deploying for optimal performance
• In reality, it is an iterative process with continuous adjustments and feedback loops.
• Optimizing ML models begins with choosing performance metrics aligning with project objectives
• Choosing the wrong metric can lead to misleading results
• New data must be continuously collected, processed, and integrated to maintain model accuracy
• Training reveals data quality issues that may require relabeling, cleaning, or starting over.
• Iterations aim to improve the model, but new issues can arise, such as poor performance on specific classes or overfitting.
• User feedback and new data may expose biases or poor performance on edge cases after deployment
• It may necessitate a rollback to previous versions or extensive retraining.
• Scaling the model to handle larger workloads may reveal new bottlenecks, so continuous monitoring is essential
• Continuous monitoring is essential to detect drifts in model accuracy, as well as require frequent updates or a complete retraining pipeline
• Deployment rarely goes as planned, which mean drops in performance, revenue impacts, or operational issues

Project Considerations

• Effective project framing sets the foundation and addresses the challenges of deploying ML models in real-world environments.
• Focus on optimizing performance, handling data challenges, and ensuring operational stability.
• It is essential to ask why a project is essential where efforts are aimed at identifying and mitigating post-deployment issues, improving model robustness and reliability.
• Objectives should include enhancing model accuracy across datasets, reduced resources for retraining, and building continuous feedback loops
• Common constraints for ML deployment include data, resources, and operations
• Limited labeled data, data privacy, storage, computational power, time constraints, and user expectations are typical constraints
• Structuring the project into phases includes the phases Data Collection & Preparation, Model Development & Training, Evaluation & Testing
• Evaluation & Testing validates the model and ensures performance benchmarks are met
• Setting-up monitoring systems and schedules comes last

Framing the Problem

• Task types include regression, classification, binary, multiclass and multilabel
• For multilabel problems, a label can belong to multiple classes, unlike multiclass problems
• Multilabel tasks can be handled using a set of multiple binary problems instead of a multiclass problem
• It is harder to create ground truth labels and decide decision boundaries for multilabel tasks

Project Objectives

• ML objectives target performance and latency
• Business objectives target to reduce cost, improve return on investment (ROI), and ensure regulatory compliance.
• ML projects should aim to increase profits directly through sales, conversion rates, or indirectly through improved client satisfaction
• It is important to establish baselines, set usefulness thresholds, and evaluate false negatives vs false positives
• It is also impotant to measure and consider interpretability and confidence in prediction

Data Engineering 101

• Involves data sources, formats, models, storage engines, and processing
• Data sources include those user and system generated
• Internal databases and third-party data also act as data sources

Data Sources

• User-generated data includes user inputs.
• System-generated data includes logs, metadata, and predictions.
• Third party data has social media data, income data, and demographic groups
• Important questions include the use cases for the data, storage considerations, and accessibility requirements

Data Storage

• Data serialization is critical for storing and accessing data
• Data formats are serialized if they can be reconstructed later

Data Formats

• Questions to consider are how frequently data is accessed and hardware to be used to run data
• Human-readable formats are JSON and CSV
• Binary formats include Parquet, Avro, Protobuf, and Pickle
• Binary (Parquet) format in Amazon S3 can unload 2x faster from Amazon Redshift and consume up to 6x less storage versus text.
• Column-major storage retrieves column-by-column for feature access
• Row-major storage retrieves row-by-row for accessing samples

Row-Major vs Column-Major

• Pandas DataFrames use a column-major format to access a column and NumPy
• NumPy ndarrays use a row-major format by default, with the option to specify column-based storage.

Data Models

• Data models describes how data is represented
• Two main paradigms exist relational and NoSQL

Relational Model

• It is similar to SQL model and have formats such as CSV and Parquet
• Normalization involves structuring the database to reduce redundancy and dependency

Relational Model(Normalization)

• Pros includes minimize mistakes, Easier to update, Easier localization
• Cons includes slow to join across multiple large tables
• Differences exist between SQL Model and Relational Model
• SQL is a declarative language that specifies what data is wanted and relies on optimization to figure out how to execute the query

SQL

• Understanding and using SQL is essential for data scientists
• Additional challenges exist if there is a need to add a column or change a column type.

NoSQL

• NoSQL (Not Only SQL) includes document and graph models.
• There was negative responses to relational database technology when NoSQL launched
• The main pain point when adopting NoSQL adoption is schema management

NoSQL (Document and Graph Models)

• Document model has documents as a central concept.
• In Document models, relationships between documents are usually rare
• Graph models emphasize the relationships between nodes and edges

Structured vs. Unstructured Data

• Structured data has defined schemas and is easy to search / analyze
• Unstructured data is flexible to changes and handles any source.
• Structural data is assumed to write when storing data
• Unstructural data is assumed to read when storing data

Choosing the Right Data Storage Engine

• Online transaction processing(OLTP) such as requesting a ride when using Lyft
• Online analytical processing(OLAP) is performed such as getting aggregations from data when using services like Lyft
• Newer paradigms support decoupling storage from processing
• Data can be stored in the same place as the processing layer on top which can be optimized for different query types

Differences between OLTP and OLAP

OLTP:

• Low latency
• High availability
• Atomicity, Consistency, Isolation, Durability (ACID) not necessary
• Typically row-major OLAP:
• Complex queries on large volumes of data
• Okay response time (seconds, minutes, even hours)
• Column-major

ETL (Extract, Transform, Load)

• Extract, Transform and Load. ETL is everywhere.
• Extract, Transform, Load and is commonly done in batches.
• Transform: the meaty part involves cleaning, validating, transposing, deriving values, joining from multiple sources, deduplicating, splitting, aggregating, etc
• It typically works by extracting standardized data to unstructured data, then changing it back through tools and infrastructure