AWS Modern Data Architecture: Storage Types

Module Objectives

• Define storage types in modern data architecture.
• Distinguish between data storage types.
• Select data storage options matching storage needs.
• Implement secure storage practices for cloud-based data.

Simplified Iterative Data Pipeline

• The stages in a data pipeline are Ingestion, Storage, Processing, and Analysis & Visualization.

Storage in the AWS Modern Data Architecture

• Amazon EMR is used for big data processing.
• Amazon S3 can be used for data warehousing, machine learning, and log analytics.
• Amazon Redshift is for data warehousing.
• Amazon SageMaker can be used for machine learning.
• Amazon OpenSearch Service assists with log analytics.
• DynamoDB is for non-relational databases.
• Aurora is for relational databases.

Types of Cloud Storage

Block Storage

• Offers dedicated, low-latency storage.
• It is scalable with high performance.
• Similar to local direct attached storage or a SAN.
• Amazon Elastic Block Storage (EBS) is an example.

File Storage

• Stores data as files.
• Highly scalable.
• Suited for content repositories and media stores.
• Amazon Elastic File System (EFS) provides an example.

Object Storage

• Stores unstructured, semistructured, or structured data.
• It is highly scalable.
• Each object has a unique identifier.
• Lower cost than traditional storage.
• Amazon Simple Storage Service (Amazon S3) provides an example.

Comparing Data Lakes and Data Warehouses

Data

• Data warehouses use relational data from transactional systems, operational databases, and line of business applications.
• Data lakes use nonrelational and relational data from IoT devices, websites, mobile apps, social media, and corporate applications.

Schema

• Data warehouses use "schema on write," designed prior to data warehouse implementation.
• Data lakes use "schema on read," written at the time of analysis.

Price and Performance

• Data warehouses provide the fastest query results using higher cost storage.
• Data lakes have query results that are faster using low-cost storage.

Data Quality

• Data warehouses use highly curated data that serves as the central version of the truth.
• Data lakes can use any data, curated (e.g., raw data) or not.

Users

• Data warehouses are for business analysts.
• Data lakes are for data scientists, data developers, and business analysts, using curated data.

Analytics

• Data warehouses support batch reporting, business intelligence (BI), and visualizations.
• Data lakes support ML, predictive analytics, and data discovery and profiling.

Data Lakes

• A centralized repository is provided.
• Both structured and unstructured data is stored.
• Data is cataloged and indexed for analysis without movement.
• Data is stored, secured, and protected at unlimited scale.
• There is in-place transformation and querying of data assets.
• Data lakes are built using Amazon S3.

Amazon Simple Storage Service (Amazon S3)

• It is secure, scalable, and durable.
• A low-cost storage solution is provided.
• Structured and unstructured data is stored.
• In-place transformation and querying is supported.
• Object storage classes are utilized.
• There is a strong data consistency model.
• Multipart upload is supported.
• It is the basis of data lake creation.

AWS Lake Formation

• Fully managed service.
• The ability to build, secure, and manage data lakes is provided.
• Automates elements of data lake creation.
• Augments the AWS Identity and Access Management (IAM) permissions model.
• Supports atomic, consistent, isolated, and durable (ACID) transactions using governed tables.
• Integrates with AWS analytics and ML services.

Key Takeaways: Data Lake Storage

• Data lakes store data as-is, without needing structured data to begin running analytics.
• Amazon S3 promotes data integrity through strong data consistency and multipart uploads.
• Using Lake Formation, governed tables can enable concurrent data inserts and edits across tables.

Data Warehouses

• A centralized repository is provided.
• Structured and semistructured data is stored
• Data is stored in one of two ways: frequently accessed data in fast storage and infrequently accessed data in cheap storage.
• Might contain multiple databases organized into tables and columns.
• Analytics processing is separate from transactional databases.
• Amazon Redshift is an example.

Amazon Redshift

• Provides a cloud-based data warehouse solution.
• Fully managed service.
• Supports near real-time data analysis.
• Uses columnar storage.
• Offers multiple node types for tailored solutions, including DC2, DS2, and RA3.

Key Takeaways: Data Warehouse Storage

• Data warehouses consist of three tiers and can store structured, curated, or transformed data.
• Amazon Redshift is a fully-managed data warehouse service, using computing resources called nodes.
• Redshift Spectrum can write SQL queries combining data from both your data lake and your data warehouse.

Choosing Your Purpose-Built Database

• Choosing the right database is the key to supporting your application architecture.
• The database will affect what the application can handle, how it will perform, and the operation you are responsible for.
• Consider: Application workload, data shape, performance requirements, and operations burden.

Factors in Choosing a Purpose-Built Database

Application Workload

• Consider if the workload is transactional or if it should be used for analytics purposes.
• Consider if your workload needs caching to improve response times.

Data shape

• How will the data be accessed?
• How often will the data be updated?

Performance

• How fast does the data access need to be?
• What is the average size of the records that are being used?
• How will end users use the service?

Operations burden

• How will you prepare for instance failures?
• How will you configure backups?
• What future upgrades might be needed?

Common Database Use Cases

Relational

• Traditional applications, enterprise resource planning (ERP), customer relationship management (CRM), and e-commerce.
• AWS services include Aurora, Amazon RDS, and Amazon Redshift.

Key-Value

• High-traffic web applications, e-commerce systems, and gaming applications.
• AWS services include DynamoDB.

Document

• Content management, catalogs, and user profiles.
• AWS services include Amazon DocumentDB.

Graph

• Fraud detection, social networking, and recommendation engines.
• AWS services include Neptune.

Key Takeaways: Purpose-Built Databases

• The chosen database will affect application handling, performance, and responsible operations.
• When choosing a database, consider application workload, data shape, performance, and operations burden.

Secure, Protect, and Manage Your AWS Data Lake

• Benefit from the built-in security of Amazon S3.
• Manage access through resource-based and user policies.
• Choose from multiple encryption options.
• Use tags to categorize and manage data, and to manage access permissions.
• Use Lake Formation for centralized governance and access control.

Security for a Data Warehouse in Amazon Redshift

• Amazon Redshift database security is separate from that of the service itself.
• Amazon Redshift has extra features to manage database security.
• Due to third-party auditing, Amazon Redshift can help support applications required to meet international compliance standards.
• Amazon Redshift integrates with Amazon CloudWatch, AWS CloudTrail, and AWS Security Hub for monitoring and alerting.

Key Takeaways: Securing Storage

• Security for data lake storage relies on the intrinsic security features of Amazon S3.
• Access policies are highly customizable to provide access to resources in a data lake.
• Data lakes on AWS rely on server-side and client-side encryption.
• Amazon Redshift handles service and database security separately.

Sample Exam Question

Key Words and Phrases

• Query in data directly from files in the company's data lake built on Amazon S3.

Correct Answer

• Amazon Redshift Spectrum.