AWS Data Engineering: Data Pipeline Design

• The AWS Academy Data Engineering course covers design principles and patterns for data pipelines.

Module Objectives

• Use the AWS Well-Architected Framework to design analytics workloads.
• Account for key milestones in data store and architecture evolution.
• Describe the components of modern data architectures on AWS.
• Cite AWS design considerations and key services for a streaming analytics pipeline.

AWS Well-Architected Framework and Lenses

• The Well-Architected Framework provides best practices and design guidance across six pillars.
• Well-Architected Lenses extend guidance to specific domains.
• The Data Analytics Lens helps with design decisions related to data elements like volume, velocity, variety, veracity, and value.
• Data Analytics Lens indentifies cloud best practices to build data pipelines

Well-Architected Framework Lenses

• Well-Architected Lenses extend the AWS Well-Architected Framework guidance to specific domains and contain insights from real-world case studies.
• The Data Analytics Lens provides key design elements for analytics workloads and reference architectures for common scenarios.
• The ML Lens addresses differences between application and machine learning workloads and provides a recommended ML lifecycle.

Evolution of Data Architectures

• Application architecture has evolved into more distributed systems over time.
• 1970s: Mainframe
• 1980s: Client-Server
• 1990s: Internet 3-tier
• 2020s: Cloud-based microservices
• Data stores have evolved to handle a greater variety of data.

Data Storage Evolution

• 1970s: Relational databases, where hierarchical databases were too rigid for complex data relationships
• 1990s: Non-relational databases, became popular since the internet's data variety didn't perform well in relational schemas
• 2010s: Data lakes, needed because big data and AI/ML required storage for huge volumes of unstructured and semi-structured data
• 2020s: Purpose-built cloud data stores, which increased demand because Cloud microservices are best matched for data type and function
• Data architectures have evolved to handle volume and velocity.

Data Architecture Evolution

• 1980s: Data warehouses and OLTP vs. OLAP databases, where application databases were overburdened
• 2000s: Big data systems, which was an improvement from relational databases that could not scale effectively for analytics and AI/ML
• 2010s: Lambda architecture and streaming solutions, which was needed because big data systems could not keep up with demands for real-time analysis
• Modern data architectures unify distributed solutions.
• Data stores and architectures adapt to demands of data volume, variety, and velocity.
• Modern data architectures use different types of data stores to suit different use cases.
• The goal of modern architecture is to unify disparate sources to maintain a single source of truth.

Modern Data Architecture on AWS

• Key design considerations include a scalable data lake, performant and cost-effective components, seamless data movement, and unified governance.
• AWS provides purpose-built data stores and analytics tools.
• AWS services manage data movement and governance.
• A centralized data lake provides data access for all consumers.
• Purpose-built data stores and processing tools integrate with the lake to read and write data.
• The architecture supports three types of data movement: outside in, inside out, and around the perimeter.
• Key AWS services for seamless access to a centralized lake are Amazon S3, Lake Formation, and AWS Glue.

Modern Data Architecture Pipeline: Ingestion and Storage

• Ingestion matches AWS services to data source characteristics and integrates with storage.
• Storage provides durable, scalable storage and includes a metadata catalog for governance and discoverability of data.
• The AWS modern data architecture uses purpose-built tools to ingest data based on characteristics of the data.
• The storage layer uses Amazon Redshift as its data warehouse and Amazon S3 for its data lake.
• The Amazon S3 data lake uses prefixes or individual buckets as zones to organize data in different states, from landing to curated.
• AWS Glue and Lake Formation are used in a catalog layer to store metadata.
• With the catalog, Amazon Redshift Spectrum can query data in Amazon S3 directly.

Modern Data Architecture Pipeline: Processing and Consumption

• Processing transforms data into a consumable state and uses purpose-built components.
• Analysis and Visualization (Consumption) democratizes consumption across the organization and provides unified access to stored data and metadata.
• Components in the processing layer are responsible to transform data into a consumable state.
• The processing layer supports three types of processing: SQL-based ELT, big data processing, and near real-time ETL.
• The consumption layer provides unified interfaces to access all the data and metadata in the storage layer.
• Supports three analysis methods: interactive SQL queries, BI dashboards, and ML.

Streaming Analytics Pipeline

• Streaming analytics include producers and consumers.
• A stream provides temporary storage to process incoming data in real time.
• The results of streaming analytics might also be saved to downstream destinations.