AWS Data Engineering: Design Principles

• Design Principles and Patterns for Data Pipelines with AWS Academy Data Engineering by Amazon Web Services
• Module objectives include using the AWS Well-Architected Framework, recounting milestones in data evolution, describing modern data architectures on AWS, and citing AWS design considerations for streaming analytics.

Well-Architected Framework

• Informs the design of analytics workloads.
• The pillars include Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimization and Sustainabilty
• Includes lenses that extend guidance to specific domains and contain insights from real-world case studies.

Well-Architected Framework Lenses

• Well-Architected Lenses extend the AWS guidance to domains.
• Data Analytics Lens provides key design elements and reference architectures for analytics workloads.
• ML Lens addresses application differences and provides a recommended ML lifecycle.

Data Architecture Evolution

• Application architecture evolved into more distributed systems, from mainframes in 1970 to client-server in 1980, Internet 3-tier in 1990, and cloud-based microservices in 2020.
• Data stores evolved to handle a greater variety of data.
  • 1970: Relational databases arose as hierarchical databases were found to be too rigid.
  • 1990: Nonrelational databases came about because the internet's variety of data did not perform well in relational schemas.
  • 2010; Data lakes became necessary as Big data and AI/ML needed to store huge volumes of unstructured and semistructured data.
  • 2020: Purpose-built cloud data stores were created to match data type and function as cloud microservices increased in demand.
• Data architectures evolved to handle volume and velocity.
  • 1970: Relational databases
  • 1980: Data warehouses and OLTP vs OLAP databases were needed as application databases were overburdened.
  • 1990: Non relational databases
  • 2000: Big data systems were needed as relational databases could not scale effectively for analyitics and AI/ML
  • 2010: Data lakes were created
  • 2020: Lambda architecture and streaming solutions were developed as big data systems could not keep up with demands for real-time analysis.
• Modern data architectures unify distributed solutions.

Modern Data Architecture

• Unifies disparate sources to maintain a single source of truth.
• Key design considerations include a scalable data lake, performant and cost-effective components, seamless data movement, and unified governance.
• Components include relational and nonrelational databases, a data lake, big data processing, log analytics, data warehousing, and machine learning.
• AWS has purpose-built data stores and analytics tools, such as Amazon EMR, Athena, DynamoDB, Amazon S3, SageMaker, and Amazon Redshift.

Key AWS Services

• Centralized data lakes provide data available to all consumers.
• Key to seamless access and include Amazon S3, Lake Formation, and AWS Glue.
• Purpose-built data stores and processing tools integrate to read and write data.
• Architecture supports three types of data movement: outside in, inside out, and around the perimeter.

Data Pipeline: Ingestion and Storage

• Ingestion matches AWS services to data source characteristics and integrates with storage.
• Storage provides durable, scalable storage and a metadata catalog for governance, and discoverability.
• Ingestion services include Amazon AppFlow, AWS DMS, DataSync, Kinesis Data Streams, and Firehose.
• The storage layer includes AWS Glue Data Catalog, Lake Formation, Amazon Redshift, and Amazon S3.
• Varying data is loaded into;
  • Traditional Schemas
  • Staging Tables
  • Objects
• Amazon S3 data lake uses prefixes or individual buckets as zones to organize data in different states, from landing to curated.

Data Pipeline: Processing and Consumption

• Processing transforms data into a consumable state by using purpose-built components.
• Consumption enables unified access to stored data and metadata across the organization.
• The processing layer supports SQL-based ELT, big data processing, and near real-time ETL.
• The consumption layer supports interactive SQL queries, BI dashboards, and ML.

Streaming Analytics

• Streaming analytics includes producers and consumers.
• A stream provides temporary storage to process incoming data in real time.
• The results of streaming analytics might also be saved to downstream destinations.
• Stream processing pipeline includes CloudWatch Events, Kinesis Data Streams, and Amazon Managed Service for Apache Flink,OpenSearch Service, Amazon S3, and Amazon Redshift.