AWS Well-Architected Framework for Analytics

• This module prepares you to use the AWS Well-Architected Framework to inform the design of analytics workloads.
• Key milestones in the evolution of data stores and data architectures are recounted through this module.
• The components of modern data architectures on AWS are described.
• AWS design considerations and key services for a streaming analytics pipeline are cited.

AWS Well-Architected Framework

• Provides best practices and design guidance across six pillars.
• The AWS Well-Architected Framework Lenses extend guidance to focus on specific domains.
• The Data Analytics Lens provides guidance that helps with design decisions related to the elements of data (volume, velocity, variety, veracity, and value).

Well-Architected Lenses

• Extend AWS Well-Architected Framework guidance to specific domains.
• They contain insights from real-world case studies.

Data Analytics Lens

• Provides key design elements of analytics workloads.
• Reference architectures are included for common scenarios.

ML (Machine Learning) Lens

• Addresses the differences between application and ML workloads.
• Offers a recommended ML lifecycle.

Activity

• Use the Data Analytics Lens from the Well-Architected Framework.
• Identify cloud best practices when building data pipelines

Evolution of Data Architectures

• Data stores and architectures evolved to adapt to increasing demands of data volume, variety, and velocity.
• Modern data architectures continue to use different types of data stores to suit different use cases.
• The goal of modern architecture is to unify disparate sources to maintain a single source of truth.
• Application architecture evolved into more distributed systems.
• In 1970, Mainframes were the standard
• Client-Server architecture became common in 1980
• Internet 3-tier architecture was standard during the year 2000
• Cloud-based microservices are standard in 2020
• Relational databases were standard during the year 1970
• Nonrelational databases became common in 1990
• Data Lakes became common in 2010
• Purpose-built cloud data stores are standard in 2020

Modern Data Architecture:

• Should have a scalable data lake
• Should have performant and cost-effective components
• Must have seamless data movement
• Should have Unified governance.
• A centralized data lake makes data available to all consumers.
• Purpose-built data stores and processing tools integrate with the data lake.
• The architecture supports three types of data movement: outside in, inside out, and around the perimeter.
• Key AWS services for seamless access to a centralized lake: Amazon S3, Lake Formation, and AWS Glue.

Ingestion and Storage Layers

• Ingestion matches AWS services to data source characteristics and integrates with storage.
• Storage provides durable, scalable storage and includes a metadata catalog for governance and discoverability of data.
• Storage services include AWS Glue Data Catalog and Lake Formation.
• Services that store date include Amazon Redshift and Amazon S3
• The modern data architecture uses purpose-built tools to ingest data based on characteristics of the data.
• The storage layer uses Amazon Redshift as its data warehouse and Amazon S3 for its data lake.
• The Amazon S3 data lake uses prefixes or individual buckets as zones to organize data in different states, from landing to curated.
• AWS Glue and Lake Formation are used in a catalog layer to store metadata.
• With the catalog, Amazon Redshift Spectrum can query data in Amazon S3 directly.
• Highly structured data is loaded into traditional schemas for Fast BI dashboards
• Semistructured data is loaded into staging tables within Amazon Redshift
• Unstructured, semistructured, and structured data is stored as objects within Amazon S3 for usage for Big data AI/ML
• Data zones in Amazon S3 include curated, trusted, raw, and landing.
• The curated Amazon S3 data zone enriches and validates the data
• The trusted Amazon S3 data zone applies structure to the data
• The raw Amazon S3 data zone is the landing zone and cleans the data

Processing and Consumption Layers

• Processing transforms data into a consumable state and is purpose-built.
• Consumption democratizes consumption across the organization and provides unified access to stored data and metadata.
• Key AWS service: Amazon Redshift.

Processing

• Transforms data into a consumable state.
• Uses purpose-built components.
• The processing layer supports three types of processing: SQL-based ELT, big data processing, and near real-time ETL.

Consumption

• Democratizes consumption across the organization.
• Provides unified access to stored data and metadata.
• The consumption layer supports three analysis methods: interactive SQL queries, BI dashboards, and ML.

Consumption Analysis Methods

• Interactive SQL queries can be done with Athena or Amazon Redshift
• Business intelligence is done with Amazon Redshift and Quicksight
• Machine learning is done with Sagemaker

Streaming analytics pipeline

• Streaming analytics includes producers and consumers.
• A stream provides temporary storage to process incoming data in real time.
• The results of streaming analytics might also be saved to downstream destinations.