AWS Well-Architected Framework for Analytics

• The module prepares you to use the AWS Well-Architected Framework to design analytics workloads.
• Key milestones in the evolution of data stores and data architectures will be reviewed.
• The components of modern data architectures on AWS will be described.
• AWS design considerations and key services for a streaming analytics pipeline will be cited.

AWS Well-Architected Framework

• The Well-Architected Framework provides best practices and design guidance across six pillars.
• The Well-Architected Framework Lenses extend guidance to focus on specific domains.
• The Data Analytics Lens provides guidance to help with design decisions related to the elements of data (volume, velocity, variety, veracity, and value).
• The AWS Well-Architected Framework informs the design of analytics workloads.
• The Well-Architected Framework has lenses for specific domains
• The Well-Architected Lenses contain insights from real-world case studies.
• The Data Analytics Lens provides key design elements of analytics workloads.
• Common scenarios include reference architectures.
• The ML Lens addresses the differences between application and machine learning (ML) workloads.
• The ML Lens provides a recommended ML lifecycle.

Application Architecture Evolution

• Application architecture evolved into more distributed systems.
• In the 1970s, the application architecture was mainframe.
• In the 1980s, the application architecture was client-server.
• In the 1990s, the application architecture was internet 3-tier.
• In the 2010s and beyond, the application architecture is cloud-based microservices.
• Data stores and architectures evolved to adapt to increasing demands of data volume, variety, and velocity.
• Modern data architectures continue to use different types of data stores to suit different use cases.
• The goal of modern architecture is to unify disparate sources to maintain a single source of truth.

Data store Evolution

• Data stores evolved to handle a greater variety of data.
• In the 1970s, hierarchical databases were used, which were too rigid for complex data relationships, so relational databases were introduced.
• In the 1990s, the internet's data variety did not perform well in relational schemas, so nonrelational databases were introduced.
• In the 2010s, big data and AI/ML needed to store huge volumes of unstructured and semistructured data, so data lakes were introduced.
• Cloud microservices increased the demand for data stores that matched data type and function, so purpose-built cloud data stores were introduced.
• Data architectures evolved to handle volume and velocity.
• In the 1980s, application databases were overburdened, leading to data warehouses and OLTP vs. OLAP databases.
• In the 2000s, relational databases could not scale effectively for analytics and AI/ML, so big data systems were introduced.
• In the 2010s, big data systems could not keep up with the demands for real-time analysis, so Lambda architecture and streaming solutions were introduced.
• Modern data architectures unify distributed solutions.
• Data stores and architectures evolved to adapt to increasing demands of data volume, variety, and velocity.

Modern Data Architecture

• The modern data architecture on AWS unifies distributed solutions.
• Key design considerations should include a scalable data lake, performant and cost-effective components, seamless data movement, and unified governance.
• The modern data architecture includes relational and nonrelational databases, a data lake, and data warehousing.
• It also includes big data processing, log analytics, and machine learning (ML).
• AWS services manage data movement and governance.

AWS Purpose-Built Data Stores

• AWS includes purpose-built data stores and analytics tools.
• Key design considerations should include a scalable data lake and performant and cost-effective components.
• AWS services that are key to seamless access to a centralized lake include Amazon S3, Lake Formation, and AWS Glue.
• A centralized data lake provides data that can be available to all consumers.
• Purpose-built data stores and processing tools integrate with the lake to read and write data.
• The architecture supports three types of data movement: outside in, inside out, and around the perimeter.
• The AWS modern data architecture uses purpose-built tools to ingest data based on the characteristics of the data.

Data Pipeline: Ingestion and Storage

• Ingestion matches AWS services to data source characteristics.
• Ingestion integrates with Storage.
• Storage provides durable, scalable storage.
• Storage includes a metadata catalog for governance and discoverability of data.
• Data is stored based on variety, volume, and velocity.
• Highly structured data is loaded into traditional schemas for fast BI dashboards.
• Semistructured data is loaded into staging tables in Amazon Redshift.
• Unstructured, semistructured, and structured data is stored as objects in Amazon S3 for big data AI/ML.
• AWS Glue and Lake Formation are used in a catalog layer to store metadata.
• With the catalog, Amazon Redshift Spectrum can query data in Amazon S3 directly.
• The storage layer uses Amazon Redshift as its data warehouse and Amazon S3 for its data lake.
• The Amazon S3 data lake uses prefixes or individual buckets as zones to organize data in different states, from landing to curated.

Data Pipeline: Ingestion Services

• Amazon AppFlow is used to ingest data from SaaS apps.
• AWS DMS is used to ingest data from OLTP, ERP, CRM, and LOB.
• DataSync is used to ingest data from file shares.
• Kinesis Data Streams is used to ingest data from web, devices, sensors, and social media.
• Firehose is used to ingest data from web, devices, sensors, and social media.

Data Pipeline: Storage Zones

• Data zones in Amazon S3 are used to organize data in different states.
• The states are landing, raw, trusted, and curated.
• landing is for clean data.
• Raw is for structured data.
• Trusted is for structured data.
• Curated is for data enrichment and validation.

Data Pipeline: Processing and Consumption

• The components in the processing layer are responsible for transforming the data into a consumable state.
• It transforms data into a consumable state and uses purpose-built components.
• The processing layer supports three types of processing: SQL-based ELT, big data processing, and near real-time ETL.
• The consumption layer democratizes consumption across the organization and provides unified access to stored data and metadata.
• The consumption layer provides unified interfaces to access all the data and metadata in the storage layer.
• The consumption layer supports three analysis methods: interactive SQL queries, BI dashboards, and ML.

Streaming Analytics

• Streaming analytics includes producers and consumers.
• A stream provides temporary storage to process incoming data in real time.
• The results of streaming analytics might also be saved to downstream destinations.