CM2606: Data Engineering Lecture 3 - Data Storage & PySpark

Podcast

Play an AI-generated podcast conversation about this lesson

Questions and Answers

What is a key use case for Amazon EBS?

Relational database storage (correct)

Long-term data archiving

Big data analytics

Media processing

Which Amazon service provides scalable, shared file storage for EC2 instances?

Amazon EFS (correct)

Amazon RDS

Amazon EBS

Amazon Glacier

What is a key use case for Amazon EFS?

Relational database storage

Container storage

EC2 boot volumes

Shared file systems (correct)

What is Amazon Glacier used for?

Long-term data archiving Signup and view all the answers

What is a key benefit of using Amazon EBS?

Low-latency storage Signup and view all the answers

Which service is used for primary storage for hosting the root file systems of EC2 instances?

Amazon EBS Signup and view all the answers

What is the primary characteristic of Storage Area Network (SAN) storage systems?

Block-level data access and high performance Signup and view all the answers

What type of storage is designed for massive scalability, high availability, and cost-effectiveness in AWS?

Amazon S3 Object storage Signup and view all the answers

What is the primary use case for Amazon S3 as a data lake?

Store vast amounts of raw, unstructured data for analysis Signup and view all the answers

What is the key characteristic of Big Data Distributed Storage systems?

Distributed, fault-tolerant, and geographically distributed Signup and view all the answers

What is the primary use case for Amazon S3 as a log storage solution?

Efficiently collect and store application and system logs Signup and view all the answers

What is the primary advantage of using Traditional Disk Storage in AWS?

Cost-effectiveness and local access Signup and view all the answers

What type of archive is suitable for storing policy documents, medical records, and emails that require long-term preservation?

Document archives Signup and view all the answers

Which data storage type is suitable for storing sensor data from IoT devices for later analysis?

S3 Signup and view all the answers

What type of storage is suitable for powering a live chat application with fast data access?

S3 Signup and view all the answers

Which data storage type is suitable for sharing application log files across multiple servers?

Glacier Signup and view all the answers

What type of storage is suitable for creating a new EC2 instance?

EBS Signup and view all the answers

What type of archive is suitable for long-term storage of backups and datasets that are infrequently accessed?

Data archives Signup and view all the answers

Study Notes

Amazon EBS and Its Use Cases

Amazon Elastic Block Store (EBS) is commonly used for providing block storage to EC2 instances, especially for databases and applications that require consistent performance.
A major benefit of Amazon EBS is that it offers automatic snapshot capabilities for data backup and recovery.
EBS serves as the primary storage for hosting the root file systems of EC2 instances.

Shared File Storage Solutions

Amazon Elastic File System (EFS) provides scalable, shared file storage that can be accessed concurrently by multiple EC2 instances, ideal for applications that need file-based storage.

Amazon EFS Use Case

EFS is particularly useful for workloads that require file sharing and need to scale up or down based on usage.

Amazon Glacier Attributes

Amazon Glacier is designed for long-term data archiving and provides secure, low-cost storage for data that is infrequently accessed but must be retained for compliance or regulatory purposes.

Storage Area Network (SAN) Characteristics

SAN storage systems are characterized by high performance, low latency, and the ability to manage large volumes of data across multiple servers, typically used in enterprise environments.

AWS Storage Solutions

Amazon Simple Storage Service (S3) is designed for massive scalability, high availability, and cost-effectiveness, making it suitable for various types of data storage.

Amazon S3 Use Cases

As a data lake, Amazon S3 is used for storing large volumes of diverse datasets, enabling analytics and data processing.
When used for log storage, S3 can store extensive logs generated by applications, allowing for archival and easier access for analysis.

Traditional Disk Storage Advantage

Traditional Disk Storage in AWS offers predictability in performance and can be beneficial for workloads that require stable, consistent I/O.

Archiving Solutions

Amazon S3 Glacier is suitable for archiving policy documents, medical records, and emails, particularly for those needing long-term preservation.
For infrequently accessed backups and datasets, S3 provides an efficient solution due to its lower storage costs.

IoT Data Storage

Amazon S3 is appropriate for storing sensor data from IoT devices as it can handle large quantities of data and provide durability for later analysis.

Live Application Support

Fast data access suitable for powering applications like live chat can be achieved using Amazon ElastiCache or DynamoDB, offering low latency and high throughput performance.

For sharing application log files across multiple servers, Amazon EFS is suitable due to its file system capabilities, allowing concurrent access across different instances.

EC2 Instance Creation

Amazon EBS provides the storage necessary for creating a new EC2 instance, ensuring it has the required storage capacity for operating systems and applications.

Studying That Suits You

Use AI to generate personalized quizzes and flashcards to suit your learning preferences.

Description

Test your understanding of data storage concepts, types of storage systems, and PySpark introduction, as covered in Lecture 3 of CM2606 Data Engineering. Explore traditional disk storage, network attached storage, and more. Get hands-on experience with PySpark and data storage in AWS Cloud.