AWS Analytics Workload Design

Module Objectives

• This module aims to prepare individuals to utilize the AWS Well-Architected Framework for analytics workload design
• Recount key milestones in the evolution of data stores and data architectures
• Describe the components of modern data architectures on AWS
• Cite AWS design considerations and key services for streaming analytics pipelines

Well-Architected Framework Lenses

• The AWS Well-Architected Framework provides guidance across six pillars
• Well-Architected Lenses extend guidance to focus on specific domains
• The Data Analytics Lens provides guidance for design decisions related to data elements like volume, velocity, variety, veracity, and value
• Well-Architected Lenses extend the AWS Well-Architected Framework guidance to specific domains
• They also provide insights from real-world case studies
• The Data Analytics Lens provides key design elements of analytics workloads and includes reference architectures for common scenarios
• The ML Lens addresses differences between the application and machine learning workloads and provides an ML lifecycle

Evolution of Data Architectures

• Application architecture has evolved into more distributed systems
• This has moved across forms like Mainframe in 1970, Client-Server in 1980, Internet 3-tier in 1990, and Cloud-based microservices in 2020
• Data stores have also evolved to handle a greater variety of data
• This has transitioned through Relational databases in 1970, Nonrelational databases in 1990, Data lakes in 2000, and Purpose-built cloud data stores in 2020
• Data architectures have evolved to handle volume and velocity
• Data warehouses and OLTP vs. OLAP databases occurred in 1980 but application databases were overburdened
• Big data systems emerged in 2000 although relational databases could not scale effectively for analytics and AI/ML
• Lambda architecture and streaming solutions came about in 2010 but big data systems couldn't keep up with demands for real-time analysis
• Data stores and architectures evolved to adapt to increasing demands of data volume, variety, and velocity
• Modern data architectures continue to use different types of data stores to suit different use cases
• The goal of modern architecture is to unify disparate sources to maintain a single source of truth

Modern Data Architecture

• Modern data architecture seeks to unify distributed solutions on AWS
• Key design considerations include a scalable data lake, performant and cost-effective components, seamless data movement, and unified governance
• AWS purpose-built data stores and analytics tools include Amazon EMR, Aurora, Relational databases, Nonrelational databases, Athena, Log analytics, Amazon S3, Machine learning, Data warehousing, DynamoDB, SageMaker, and Amazon Redshift
• AWS services which help manage data movement and governance are Lake Formation and AWS Glue
• A centralized data lake provides data that can be available to all consumers
• Purpose-built data stores and processing tools integrate with the lake to read and write data
• The architecture supports three types of data movement like outside in, inside out, and around the perimeter
• AWS services that are key to seamless access to a centralized lake include Amazon S3, Lake Formation, and AWS Glue

Ingestion and Storage

• Key aspects of modern data architecture pipelines are Ingestion and Storage
• Ingestion matches AWS services to data source characteristics and integrates with storage
• Storage provides durable, scalable storage and includes a metadata catalog for governance and discoverability of data
• Ingestion services are matched to variety, volume, and velocity depending on their respective data sources
• Examples for SaaS apps are Amazon AppFlow, OLTP/ERP/CRM/LOB is AWS DMS, File shares is DataSync, and for Web/Devices/Sensors/Social media is Kinesis Data Streams/Firehose with Ingest
• The storage layer uses AWS Glue Data Catalog and Lake Formation for the Catalog as well as Amazon Redshift and Amazon S3 for Storage
• High structured data is loaded into traditional schemas, semistructured data is loaded into staging tables, and unstructured, semistructured, and structured data is stored as objects
• Data zones in Amazon S3 include curated for enrich and validate, trusted for structure, raw, and landing for clean
• The AWS modern data architecture uses purpose-built tools to ingest data based on the characteristics of the data
• The storage layer uses Amazon Redshift as its data warehouse and Amazon S3 for its data lake
• The Amazon S3 data lake uses prefixes or individual buckets as zones to organize data in different states, from landing to curated
• AWS Glue and Lake Formation are used in a catalog layer to store metadata
• Employing the catalog, Amazon Redshift Spectrum can query data in Amazon S3 directly

Processing and Consumption

• Key parts of modern data architecture pipeline involve processing and consumption
• Processing the transforms data into a consumable state and uses purpose-built components
• Analysis and visualization (consumption) democratizes consumption across the organization and provides unified access to stored data and metadata
• The processing layer encompasses SQL-based ELT with Amazon Redshift, Big data processing with Amazon EMR and AWS Glue, and Near real-time ETL with Amazon Managed Service for Apache Flink as well spark streaming on Amazon EMR or AWS Glue
• Consumption includes Athena and Amazon Redshift for Interactive SQL, QuickSight for Business intelligence, and SageMaker for Machine learning
• The processing layer is responsible to transform data into a consumable state
• The processing layer supports three types of processing, like SQL-based ELT, big data processing, and near real-time ETL
• The consumption layer provides unified interfaces to access all the data and metadata in the storage layer
• The consumption layer supports three analysis methods including interactive SQL queries, BI dashboards, and ML

Streaming Analytics

• Streaming Analytics is a valuable pipeline to use
• Data sources for continuous streams are fed through Ingestion and producers -> Stream storage -> Stream processing and consumers -> Analysis and Visualization
• An example stream processing pipeline is where AWS activities that emit CloudWatch Events are processed via CloudWatch, then Kinesis Data Streams, followed again by Amazon Managed Service for Apache Flink
• These are then sent to OpenSearch Service or downstream destinations to Amazon S3 or Amazon Redshift where streaming analytics includes producers and consumers
• A stream provides temporary storage to process incoming data in real time
• Processing the results of streaming analytics may be saved to downstream destinations