AWS Well-Architected Framework for Data analytics

• Module objectives include using the AWS Well-Architected Framework to inform the design of analytics workloads.
• Module objectives include recounting key milestones in the evolution of data stores and data architectures.
• Module objectives include describing the components of modern data architectures on AWS.
• Module objectives include citing AWS design considerations and key services for a streaming analytics pipeline.

AWS Well-Architected Framework

• The Well-Architected Framework provides best practices and design guidance across six pillars.
• The Framework Lenses extend guidance to focus on specific domains.
• The Data Analytics Lens provides guidance that helps with design decisions related to the elements of data: volume, velocity, variety, veracity, and value.

Well-Architected Framework Lenses

• Well-Architected Lenses extend the AWS Well-Architected Framework guidance to specific domains.
• Well-Architected Lenses contain insights from real-world case studies.
• Data Analytics Lens provides key design elements of analytics workloads.
• Data Analytics Lens includes reference architectures for common scenarios.
• ML Lens addresses differences between application and machine learning (ML) workloads.
• ML Lens provides a recommended ML lifecycle.

Evolution of Data Architectures

• Data stores and architectures evolved to adapt to increasing demands of data volume, variety, and velocity.
• Modern data architectures continue to use different types of data stores to suit different use cases.
• The goal of modern data architecture is to unify disparate sources to maintain a single source of truth.
• Application architecture evolved into more distributed systems from 1970 to 2020, starting with Mainframe, then Client-Server, Internet 3-tier, and now Cloud-based microservices.
• The evolution of data stores to handle a greater variety of data moved from Relational databases to Nonrelational databases, then Data lakes, and finally to Purpose-built cloud data stores.
• Application databases are overburdened, leading to data warehouses and Online Transaction Processing (OLTP) vs. Online Analytical Processing (OLAP) databases.
• Big data systems scale effectively for analytics and AI/ML, whereas Relational databases cannot.
• Big data systems cannot keep up with demands for real-time analysis, resulting in Lambda architecture and streaming solutions.
• Modern data architecture on AWS unifies distributed solutions.

Modern Data Architecture on AWS Design Considerations

• Key design considerations include: scalable data lake, performant and cost-effective components, seamless data movement, and unified governance.
• AWS purpose-built data stores and analytics tools address scalability and cost-effectiveness.
• AWS services manage data movement and governance, facilitating seamless data movement and unified governance with services like AWS Glue and Lake Formation.
• A centralized data lake provides data accessible to all consumers.
• Purpose-built data stores and processing tools integrate with it for reading and writing data.
• This architecture supports outside in, inside out, and around the perimeter types of data movement.
• Key AWS services for seamless lake access include Amazon S3, Lake Formation, and AWS Glue.

Modern Data Architecture Pipeline: Ingestion and Storage

• Matches AWS services to data source characteristics.
• Integrates with storage.
• Provides durable, scalable storage.
• Includes a metadata catalog for governance and data discoverability.
• Highly structured data is loaded into traditional schemas and used for Fast BI dashboards.
• Semi-structured data is loaded into staging tables using Amazon Redshift.
• Unstructured, semi-structured, and structured data is stored as objects and is used for Big data AI/ML.
• The Amazon S3 data lake uses prefixes or individual buckets as zones to data in different states, from landing to curated, which can be used for complex querying by Amazon Redshift.
• AWS Glue and Lake Formation are used in a catalog layer to store metadata.
• Amazon Redshift Spectrum can query data in Amazon S3 directly with the catalog.

Modern Data Architecture Pipeline: Processing and Consumption

• Components in the processing layer transform data into a consumable state.
• The processing layer supports three types of processing: SQL-based ELT, big data processing, and near real-time ETL.
• The consumption layer provides unified interfaces to access all the data and metadata in the storage layer.
• The consumption layer supports three analysis methods: interactive SQL queries, Business Intelligence (BI) dashboards, and Machine Learning (ML).

Streaming Analytics Pipeline

• Streaming analytics includes producers and consumers.
• A stream provides temporary storage to process incoming data in real-time.
• The results of streaming analytics might also be saved to downstream destinations.