AWS Well-Architected Framework for Analytics

• This module helps you use the AWS Well-Architected Framework for analytics workloads and describe modern data architectures on AWS.
• It also facilitates recounting milestones in data store evolution and citing AWS design considerations for streaming analytics.

AWS Well-Architected Framework

• The AWS Well-Architected Framework provides best practices and design guidance for building robust, secure, efficient, and cost-effective systems in the cloud.
• The framework includes six pillars: operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability.

Well-Architected Lenses

• Well-Architected Lenses extend the AWS Well-Architected Framework guidance to specific domains with real-world case studies.
• The Data Analytics Lens provides key design elements and reference architectures for analytics workloads.
• The ML Lens addresses differences between application and machine learning (ML) workloads, also giving a recommended ML lifecycle.
• An activity will have you using the Data Analytics Lens from the Well-Architected Framework to identify cloud best practices when building data pipelines.

Data architecture evolution

• Application architecture evolved from mainframe to client-server, then to internet 3-tier, and finally to cloud-based microservices.
• Data stores evolved from hierarchical to relational and non-relational databases, then to data lakes, and finally to purpose-built cloud data stores.
• Data architectures evolved to handle volume and velocity with data warehouses for OLTP vs. OLAP databases and big data systems. Application databases became overburdened.
• Relational databases cannot effectively scale for analytics and AI/ML, and big data systems cannot keep up with demands for real-time analysis.
• Modern data architectures unify distributed solutions for a single source of truth.

Modern Data Architecture on AWS

• Modern data architecture on AWS requires consideration for a scalable data lake, performant and cost-effective components, seamless data movement, and unified governance.
• AWS has purpose-built data stores and analytics tools, including Relational databases such as Amazon Aurora and DynamoDB.
• AWS also uses analytics tools, like EMR (Elastic MapReduce), SageMaker, and Redshift, among others.
• AWS services which help to manage data movement and governance feature Lake Formation and AWS Glue.
• A centralized data lake makes data available to all consumers
• Purpose-built data stores and tools integrate with the lake for reading and writing data.
• The S3 data lake uses prefixes or buckets as zones to organize data in different states, from landing to curated
• AWS Glue and Lake Formation are used in a catalog layer to store metadata, while Redshift Spectrum can query data in Amazon S3 directly.

Modern data architecture pipeline

• Ingestion matches AWS services to data source characteristics and integrates with storage.
• Storage provides durable, scalable storage and a metadata catalog for governance and discoverability.
• Common data source types are SaaS apps, OLTP, ERP, CRM, LOB, File shares, Web, Devices, Sensors, and Social media.
• AWS has dedicated ingestion services based on the data source type like Amazon AppFlow, AWS DMS, DataSync, Kinesis Data Streams, Firehose, etc.
• Amazon S3 is used to natively integrate semi-structured and unstructure data along with structured data as objects for data storage and use cases like Big data AI/ML.
• Amazon Redshift is used for highly structured data which is loaded into schemas and use cases like Fast BI Dashboards.
• Data zones in Amazon S3 have curated, trusted, raw, and landing zones to enrich, validate, structure, and clean the data.
• AWS Glue Data Catalog and Lake Formation make up the catalog layer in data storage. AWS Glue crawlers and Amazon Redshift use Spectrum to identify schemas.
• The processing layer transforms data into a consumable state using purpose-built components.
• The analysis and visualization (consumption) layer democratizes consumption across the organization and provides unified access to stored data and metadata.
• Processing can be SQL-based ELT, big data processing, or near real-time ETL.
• Consumption can be through interactive SQL queries, BI dashboards, or ML.
• Amazon Athena, Amazon Redshift, and QuickSight can be used to consume data for interactive SQL and BI dashboards.

Streaming Analytics Pipeline

• Streaming analytics has producers and consumers.
• A stream provides temporary storage to process incoming data in real time.
• Results of streaming analytics might also be saved to downstream destinations such as Amazon S3 or Redshift.