AWS Data Ingestion

• This module prepares one to perform key tasks when building an ingestion layer.
• The module also covers how purpose-built AWS services support ingestion tasks.
• The features of AWS Glue work together to automate batch ingestion.
• You can describe AWS streaming services and features to simplify streaming ingestion.
• Identify configuration options in AWS Glue and Amazon Kinesis Data Streams that help you scale your ingestion processing.
• The module also covers distinct characteristics of ingesting Internet of Things (IoT) data by using AWS IoT Core.

Batch and Streaming Data Flow

• Batch ingestion processes a batch of records as a dataset, running on demand, on a schedule, or based on an event.
• Streaming ingestion continually ingests records and processes sets of records as they arrive on the stream.
• Key drivers for data ingestion are data volume and velocity.

Batch Ingestion

• Sales transaction data is sent periodically to a central location for overnight analysis and reports.

Streaming Ingestion

• Clickstream data has a large volume of small bits of data sent at a continuous pace and must be analyzed immediately for recommendations.
• Batch jobs query the source, transform the data, and load it into the pipeline.
• Traditional ETL uses batch processing.
• With stream processing, producers put records on a stream where consumers get and process them.
• Streams are designed to handle high-velocity data and real-time processing.
• Batch ingestion involves writing scripts and jobs to perform ETL or ELT processes.
• Key characteristics for pipeline design include ease of use, data volume and variety, orchestration and monitoring, scaling, and cost management.

Building a Batch Processing Pipeline

• Start by extracting data from sources, then transform/load, and finally load/transform.

Data Volume and Variety

• Handling large volumes of data is required.
• Support disparate source and target systems.
• Must support different data formats seamlessly.

Orchestration and monitoring parameters

• You have to support workflow creation.
• Key to Provide dependency management on the workflow.
• Supporting bookmarking is key.
• Enable logging.
• Alerts on job failure

Match AWS Purpose-Built Tools to Data Sources

• Amazon AppFlow ingests data from software as a service (SaaS) applications, by creating connectors with filters.
• Amazon AppFlow can map fields and perform transformations, perform validation, and securely transfer to Amazon S3 or Amazon Redshift.
• AWS DMS ingests data from relational databases; connecting to source data and formatting it for a target.
• With AWS DMS one can Use source filters and table mappings, perform data validation, and write to many AWS data stores or create a continuous replication task.
• AWS DataSync is used for ingesting data from file systems, by applying filters to transfer a subset of files.
• With DataSync one can use a variety of file systems as sources and targets, including Amazon S3 as a target.
• DataSync can also securely transfer data between self-managed storage systems and AWS storage services.
• AWS Data Exchange helps integrate third-party datasets into pipelines.
• With AWS Data Exchange you can find and subscribe to sources, preview before subscribing, copy subscribed datasets to Amazon S3, and receive notifications of updates.

Key Takeaways

• Choose purpose-built tools that match the type of data to be ingested and simplify ingestion tasks.
• Amazon AppFlow, AWS DMS, and DataSync each simplify the ingestion of specific data types.
• AWS Data Exchange provides a simplified way to find and subscribe to third-party datasets.
• AWS Glue simplifies batch ingestion tasks through schema identification, data cataloging, job authoring and monitoring.
• AWS Glue offers serverless ETL processing and ETL orchestration.
• AWS Glue crawlers derive schemas from data stores and provide them to the centralized AWS Glue Data Catalog.
• AWS Glue Studio provides visual authoring and job management tools.
• AWS Glue Spark runtime engine processes jobs in a serverless environment.
• AWS Glue workflows provide ETL orchestration.
• CloudWatch provides integrated monitoring and logging for AWS Glue, including job run insights.

Horizontal Scaling

• Increase the number of workers allocated to the job.
• Horizontal Scaling is used when working with large, splittable datasets.

Vertical Scaling

• Choose a worker type with larger CPU, memory, and disk space.
• Vertical Scaling should be used when Working with memory-intensive or disk-intensive applications or executing Machine learning (ML) transformations.

Scaling Considerations

• Performance goals should focus on optimizing batch processing.
• Scale AWS Glue jobs horizontally by adding more workers, or vertically by choosing a larger worker in the job configuration.
• Large, splittable files let the AWS Glue Spark runtime engine run many jobs in parallel with less overhead than processing many smaller files.
• Key characteristics for stream ingestion and processing are throughput, loose coupling, parallel consumers, checkpointing, and replay.
• Throughput means planning for a resilient, scalable stream that can adapt to velocity and volume.
• Loose coupling involves building independent ingestion, processing, and consumer components.
• Parallel consumers allow multiple consumers on a stream to process records in parallel and independently.
• Checkpointing and replay maintains record order and allow replay, and supports the ability to mark the farthest record processed on failure.
• Amazon Data Firehose performs no-code or low-code streaming ETL by ingesting from many AWS services, applying built-in and custom transformations.
• With Amazon Data Firehose you can deliver directly to data stores, data lakes, and analytics services.
• Amazon Managed Service for Apache Flink queries and analyzes streaming data by ingesting data from other services, enriching and augmenting data across time windows.
• It Builds applications in Apache Flink and uses SQL, Java, Python, or Scala.
• The stream is a buffer between the producers and the consumers of the stream.
• Scaling Considerations for stream processing Kinesis Data Streams provides for the management of scaling options, throughput of data, and scaling of data written on the stream.
• AWS IoT Core provides the ability to securely connect, process, and act on IoT device data; including features to filter and transform.
• AWS IoT Core can also route data to other AWS services, including streaming and storage services.
• With AWS IoT services, one can use MQTT and a pub/sub model to communicate with IoT devices.
• You can also use AWS IoT Core to securely connect, process, and act upon device data.
• The AWS IoT Core rules engine transforms and routes incoming messages to AWS services.