Cloud Storage and Modern Data Architecture

• A modern data architecture allows the definition of storage types.
• Data storage options that match specific storage needs can be selected.
• Secure storage practices for cloud-based data should be implemented.

Simplified Iterative Data Pipeline

• Consists of data sources, ingestion, storage, processing, and analysis/visualization.

AWS Modern Storage Architecture

• Includes Amazon EMR for big data processing.
• Aurora, a relational database.
• DynamoDB, a non-relational database.
• Amazon S3 for data warehousing.
• SageMaker for machine learning.
• OpenSearch Service for log analytics.
• Amazon Redshift.

Types of Cloud Storage

• Block storage offers dedicated, low-latency storage and high performance, is similar to local direct attached storage or SAN, and is exemplified by Amazon EBS.
• File Storage stores data as files, is highly scalable and ideal medium for content repositories and media stores, is exemplified by Amazon EFS.
• Object storage stores unstructured, semistructured, or structured data, is highly scalable, uses unique identifiers for each object, and has a lower cost than traditional storage, is exemplified by Amazon S3.

Comparing Data Lakes and Data Warehouses

• Data warehouses use relational data from transactional systems, operational databases, and line of business applications.
• Data lakes use nonrelational and relational data from IoT devices, websites, mobile apps, social media, and corporate applications.
• Data warehouse schema is designed prior to implementation (schema on write).
• Data lake schema is written at the time of analysis (schema on read).
• Data warehouses provide query results using higher cost storage.
• Data lakes provide query results using low-cost storage.
• Data warehouses use highly curated data as the central version of the truth.
• Data lakes use any data, which might or might not be curated.
• Data warehouse users are business analysts.
• Data lake users are data scientists, data developers, and business analysts.
• Data warehouses use batch reporting, business intelligence (BI), and visualizations.
• Data lakes use ML, predictive analytics, and data discovery and profiling.

Data Lakes

• It provides a centralized repository for structured and unstructured data.
• Catalogs and indexes data for analysis without data movement.
• Securely stores and protects data at unlimited scale.
• Offers in-place transformation and querying of data assets.
• It is built using Amazon S3.

Amazon S3

• A secure, scalable, and durable, low-cost storage solution that stores structured and unstructured data.
• Offers in-place transformation and querying; uses object storage classes, has a strong data consistency model, supports multipart upload, and is the basis of data lake creation.

AWS Lake Formation

• It provides the ability to build, secure, and manage data lakes and automates elements of data lake creation.
• It augments the AWS Identity and Access Management (IAM) permissions model.
• It supports atomic, consistent, isolated, and durable (ACID) transactions using governed tables.
• Integrates with AWS analytics and ML services.
• Data lakes can store data as-is, without needing to structure it before running analytics.
• Amazon S3 promotes data integrity through strong data consistency and multipart uploads.
• Governed tables can be used to enable concurrent data inserts and edits across tables via Lake Formation.

Data Warehouses

• Provide a centralized repository for structured and semistructured data.
• Data is stored with frequently accessed data in fast storage and infrequently accessed data in cheap storage.
• Might contain multiple databases that are organized into tables and columns.
• Separate analytics processing from transactional databases.
• Amazon Redshift is one example.

Amazon Redshift

• A fully managed service that provides a cloud-based data warehouse solution and supports near real-time data analysis.
• It uses columnar storage and offers multiple node types for tailored solutions, including DC2, DS2, and RA3.

Data Warehouses Key Takeaways

• They consists of three tiers and can store structured, curated, or transformed data.
• Amazon Redshift is a fully-managed data warehouse service that uses computing resources called nodes.
• Redshift Spectrum can combine data from both your data lake and your data warehouse by writing SQL queries.

Purpose Built Databases

• Choosing the right database is key to supporting your application architecture.
• A database will affect what your application can handle, how it will perform, and the operation that you are responsible for.
• Factors to consider are application workload, data shape, performance requirements, and operations burden.

Factors in Choosing a Purpose-Built Database

• Considerations for application workload should includes transactional usage, analytics purposes, and caching to improve response times.
• Considerations for data shape include how the data will be accessed and how often the data will be updated.
• Performance factors include how fast the data access needs to be, the average size of the records, and how end users will use the service.
• Operation burden consideration includes preparation for instance failures, configuration of backups, and future upgrades.

Common Database Use Cases

• Relational databases are suitable for traditional applications, enterprise resource planning (ERP), customer relationship management (CRM), and e-commerce, with AWS services like Aurora, Amazon RDS, and Amazon Redshift.
• Key-value databases are suitable for high-traffic web applications, e-commerce systems, and gaming applications.
• Document databases are suitable for content management, catalogs, and user profiles with AWS services like Amazon DocumentDB.
• Graph databases are suitable for fraud detection, social networking, and recommendation engines with AWS services like Neptune.
• Choice of database will impact your application, its performance, and the operations that you are responsible for.
• In choosing your database, consider application workload, data shape, performance, and operations burden.

Securing Data

• Security for data lake storage is built upon the intrinsic security features of Amazon S3.
• Access policies through resource-based and user policies provide a highly customizable way to provide access to resources in your data lake.
• Encryption options are server-side or client side.
• Tags are used to categorize and manage data and to manage access permissions.
• AWS lake formation is used for centralized governance and access control.
• Data lakes that are built on AWS rely on server-side and client-side encryption.
• Amazon Redshift handles service security and database security as distinct functions.
• Amazon Redshift integrates with Amazon CloudWatch, AWS CloudTrail, and AWS Security Hub for monitoring and alerting.

Exam Question - Key Takeaway

• Redshift Spectrum enables the use of query data directly from files in the company’s data lake built on Amazon S3.