AWS Analytics Workload Design

• The module aims to use the AWS Well-Architected Framework to guide analytics workload design
• The module aims to recount the evolution of data stores and architectures
• The module aims to describe components of modern data architectures on AWS
• The module aims to identify AWS design considerations and services for streaming analytics pipelines

AWS Well-Architected Framework

• The Well-Architected Framework provides best practices and design guidance across six pillars
• The Well-Architected Framework Lenses extend guidance to focus on specific domains
• The Data Analytics Lens provides guidance for design decisions related to data volume, velocity, variety, veracity, and value
• The Well-Architected Framework extends AWS architectural guidance to specific domains and incorporates real-world case studies
• The Data Analytics Lens provides key design elements for analytics workloads with reference architectures
• The ML Lens addresses the difference between machine learning workloads and application workloads
• The ML Lens provides a recommended ML lifecycle

The Evolution of Data Architectures

• Application architecture evolved into more distributed systems
• 1970 saw adoption of Mainframes
• 1980 saw adoption of Client-Server architecture
• 1990 saw adoption of Internet 3-tier architecture
• 2010 saw adoption of Cloud-based microservices
• Data stores and architectures evolved to adapt to increasing demands of data volume, variety, and velocity
• Modern data architectures continue to use different types of data stores to suit different use cases
• The goal of the modern architecture is to unify disparate sources to maintain a single source of truth
• In 1970, Hierarchical databases were used but were too rigid for complex relations, and replaced by Relational databases
• In 1990, the Internet introduced data that didn't perform well in relational schemas, and Nonrelational databases were adopted
• In 2010, Big data and AI/ML needed huge volumes of unstructured/semi-structured data, and Data Lakes were adopted
• In 2020, Cloud microservices increased demand for data stores that matched to data type and function, and Purpose-built cloud data stores were adopted
• In 1980, data warehouses and OLTP vs OLAP databases were introduced, as application databases became overburdened
• In 2000, Big data systems emerged because Relational Databases do not scale for analytics and AI/ML
• In 2010, Lambda architecture and streaming solutions emerged as Big Data systems couldn't keep up with demands for real-time analysis

Modern Data Architecture on AWS

• Key design considerations include a scalable data lake, performant and cost-effective components, seamless data movement, and unified governance
• A centralized data lake provides data that can be available to all consumers
• Purpose-built data stores and processing tools integrate with the lake to read and write data
• The architecture supports three types of data movement: outside in, inside out, and around the perimeter
• AWS services that are key to seamless access to a centralized lake include Amazon S3, Lake Formation, and AWS Glue
• Relational databases, Nonrelational databases, Data lakes, and data warehouses are all components
• Services like Big data processing, Machine Learning and Log analytics are integrated and unified
• AWS offers purpose-built data stores like Amazon EMR, Amazon Athena, Amazon DynamoDB, Amazon Redshift, Amazon SageMaker, Amazon OpenSearch Service
• AWS also provides services like Lake Formation, and AWS Glue to manage data movement and governance

Modern Data Architecture Pipeline: Ingestion and Storage

• Ingestion matches AWS services to data source characteristics and integrates with storage
• Storage provides durable, scalable storage and includes a metadata catalog for governance and discoverability of data
• The AWS modern data architecture uses purpose-built tools to ingest data based on characteristics of the data
• The storage layer uses Amazon Redshift as its data warehouse and Amazon S3 for its data lake
• Amazon S3 stores highly structured data that is loaded into traditional schemas
• Amazon S3 stores semistructured data that is loaded into staging tables
• Amazon S3 stores Unstructured, semistructured, and structured data as objects
• The Amazon S3 data lake uses prefixes or individual buckets as zones to organize data in different states, from landing to curated
• AWS Glue and Lake Formation are used in a catalog layer to store metadata
• Amazon S3 stores raw, landing and trusted data while Amazon Redshift is used for more complex querying
• AWS Glue crawlers provide Schema info with Amazon Redshift providing Amazon Redshift Spectrum for wider cataloging

Modern Data Architecture Pipeline: Processing and Consumption

• The processing layer transforms data into a consumable state and uses purpose-built components
• The consumption layer democratizes consumption across the organization and provides unified access to stored data and metadata
• The architecture provides SQL-based ETL, big data processing, and near real-time ETL.
• Components in the processing layer are responsible to transform data into a consumable state
• The consumption layer supports three analysis methods: interactive SQL queries, BI dashboards, and machine learning
• Analysis and visualization are achieved through Athena, Amazon Redshift and QuickSight
• For interactive SQL, Athena and Amazon Redshift pull from Storage using AWS Glue Data Catalog and Lake Formation
• AWS Glue Data Catalog, Lake Formation and Amazon Redshift are used when Consuming Data For Business Intelligence and ML

Streaming Analytics Pipeline

• Streaming analytics includes producers and consumers
• A stream provides temporary storage to process incoming data in real time
• The results of streaming analytics might also be saved to downstream destinations
• Services used in streaming analytics include CloudWatch Events, Kinesis Data Streams, Amazon Managed Service for Apache Flink and OpenSearch Service
• Downstream destinations include AWS S3 and Amazon Redshift