Modern Data Architecture and Cloud Storage

Module Objectives

• You will learn about modern data architecture
• It will define storage types
• It will distinguish between data storage types
• Matching data storage options to storage needs is a lesson objective
• Secure storage practices specifically for cloud-based data will be covered

Simplified Data Pipeline

• The iterative data pipeline includes ingestion, storage, processing, analysis, and visualization

AWS Modern Data Architecture

• AWS services for data storage and analysis include:
  • Amazon EMR, Aurora, DynamoDB, SageMaker, Amazon Redshift, Amazon S3, and OpenSearch Service.

Cloud Storage Types

Block Storage

• Offers a dedicated, low-latency storage solution
• High performance and scalability is a feature
• It is similar to local direct attached storage or a storage area network (SAN)
• An example is Amazon Elastic Block Storage (Amazon EBS)

File Storage

• It stores data as files
• It is highly scalable
• Suited for content repositories and media stores
• An example is Amazon Elastic File System (Amazon EFS)

Object Storage

• It stores unstructured, semistructured, or structured data
• Highly scalable
• Uses a unique identifier for each object
• Cheaper than traditional storage
• An example is Amazon Simple Storage Service (Amazon S3)

Data Lakes vs Data Warehouses

Data

• Data warehouses use relational data from transactional systems and operational databases
• data lakes use both relational and nonrelational data:
  • Internet of Things (IoT) devices, websites, mobile apps, social media, and corporate applications

Schema

• Data warehouses use a schema on write:
  • It is designed prior to implementation
• Data lakes use a schema on read:
  • It is written at the time of analysis

Price and Performance

• Data warehouses have faster query results but use higher cost storage
• Data lakes have query results that get faster using low-cost storage

Data Quality

• Data warehouses use highly curated data and serves as a central version of the truth
• Data lakes may or may not use curated data, such as raw data

Users

• Data warehouses are used by business analysts
• Data lakes are used by data scientists, data developers, and business analysts using curated data

Analytics

• Data warehouses use batch reporting, business intelligence (BI), and visualizations
• Data lakes use machine learning (ML), predictive analytics, data discovery, and profiling

Data Lakes

• Provide a centralized repository
• Store both structured and unstructured data
• Catalogs and indexes data for analysis without data movement
• Stores, secures, and protects data at unlimited scale
• Offer in-place transformation and querying of data assets
• Built using Amazon S3

Amazon S3

• It is secure, scalable, and durable
• Provides a low-cost storage solution
• Stores structured and unstructured data
• Offers in-place transformation and querying
• Uses object storage classes
• Has a strong data consistency model
• Supports multipart upload
• Is the basis of data lake creation

AWS Lake Formation

• It is a fully managed service
• Provides the ability to build, secure, and manage data lakes
• Automates elements of data lake creation
• Augments the AWS Identity and Access Management (IAM) permissions model
• Supports atomic, consistent, isolated, and durable (ACID) transactions
  • This achieved by using governed tables
• Integrates with AWS analytics and ML services

Data Lake Storage: Key Takeaways

• Data lakes store data "as-is"
• No need to structure data before running analytics
• Amazon S3 promotes data integrity through strong data consistency and multipart uploads
• Lake Formation enables concurrent data inserts and edits across tables using governed tables

Data Warehouses

• Provide a centralized repository
• Stores structured and semistructured data
• Stores data in two forms:
  • Frequently accessed data in fast storage
  • Infrequently accessed data in cheap storage
• Might contain multiple databases organized into tables and columns
• Separate analytics processing from transactional databases
• Amazon Redshift is an example

Amazon Redshift

• It provides a cloud-based data warehouse solution
• A fully managed service
• Supports near real-time data analysis
• Uses columnar storage
• Offers node types for tailored solutions:
  • DC2
  • DS2
  • RA3

Data Warehouse Storage: Key Takeaways

• Data warehouses consist of three tiers
• They can store structured, curated, or transformed data
• Amazon Redshift is a fully-managed data warehouse service
  • It uses computing resources called nodes
• Redshift Spectrum writes SQL queries that combine data from both data lakes and data warehouses

Purpose-Built Databases

• Choosing the right database is key for supporting your application architecture
• A database affects what your application can handle and how it will perform; and it affects the operation that you are responsible for
• Factors to consider:
  • Application workload
  • Data shape
  • Performance requirements
  • Operations burden

Factors for choosing

• Aspects of application workload include transactional needs, analytics purposes and caching needs
• Aspects of data shape include data access methods and data update frequency
• Aspects of performance include data access speed, record size to be used, and how it is used by end user
• Aspects of operations burden include preparing for instance failures, configuring backups and future upgrades

Database Use Cases

• Relational databases are useful for traditional applications, enterprise resource planning (ERP), customer relationship management (CRM), and e-commerce. AWS services include Aurora, Amazon RDS, and Amazon Redshift
• Key-value databases are useful for high-traffic web applications, e-commerce systems, and gaming applications. AWS service includes DynamoDB
• Document databases are useful for content management, catalogs, and user profiles. AWS service includes Amazon DocumentDB
• Graph databases are useful for fraud detection, social networking, and recommendation engines. AWS service includes Neptune

Purpose-Built Databases: Key Takeaways

• A database choice affects what the application can handle, how it will perform, and the operations to be responsible for
• When choosing a database, consider several factors:
  • Application workload
  • Data shape
  • Performance
  • Operations burden

Securing AWS Data Lakes

• Benefit from the built-in Amazon S3 security
• Manage access through resource-based policies and user policies
• Choose from multiple encryption options
• Use tags to categorize and manage data, and manage access permissions
• Use Lake Formation for centralized governance and access control

Security for Data Warehouses in Amazon Redshift

• Amazon Redshift database security is district from the service itself
• Amazon Redshift provides additional features to manage database security
• Due to third-party auditing, Amazon Redshift can help to support applications that need to meet international compliance standards
• Amazon Redshift integrates with Amazon CloudWatch, AWS CloudTrail, and AWS Security Hub for monitoring and alerting

Securing Storage: Key Takeaways

• Security for data lake storage is built upon Amazon S3's intrinsic security features
• Access policies provide a highly customizable way to provide access to resources in your data lake
• Data lakes built on AWS rely on server-side and client-side encryption
• Amazon Redshift handles service security and database security as two distinct functions