Data Ingestion: batch and streaming

• This module goes over the primary tasks a data engineer must perform when building an ingestion layer
• It describes how AWS services support ingestion tasks and automated batch ingestion
• It also identifies streaming services and features that simplify streaming ingestion
• It identifies configuration options in AWS Glue and Amazon Kinesis Data Streams
• It details the scaling of ingestion processing, and the characteristics of ingesting IoT data when using AWS IoT Core

Batch vs Stream Ingestion

• Batch ingestion processes a batch of records as a dataset
  • Runs on demand, on a schedule, or based on an event
• Streaming ingestion ingests records continually and processes sets of records as they arrive
• Data volume and velocity are primary drivers for deciding on which ingestion method to use
• The method should fit both the amount of data ingested and the frequency of ingestion
• Batch jobs query the source, transform the data, and load it into the pipeline
• Traditional ETL uses batch processing
• With stream processing, producers put records on a stream where consumers get and process them
• Streams are designed to handle high-velocity data and real-time processing

Batch Ingestion Processing Pipeline

• To build a batch processing pipeline:
  • Extract: Connect to sources and select data
  • Transform/Load: Identify the source and target schemas, transfer and store data securely, and transform the dataset
  • Load/Transform: Load the dataset to durable storage
• Key characteristics for pipeline design include ease of use, data volume and variety, orchestration and monitoring, and scaling and cost management
• Batch ingestion involves writing scripts and jobs to perform the ETL or ELT process
• Workflow orchestration helps handle interdependencies between jobs, and manage failures within a set of jobs

Purpose-Built AWS tools for Batch Ingestion

• AWS provides purpose-built tools that match the type of data to be ingested and simplify the tasks involved
• These tools also provide: secure connections, data store integration, automated updates, Amazon CloudWatch monitoring, and selection and transformation
• For SaaS applications, use Amazon AppFlow to ingest data
  • This involves creating a connector with filters, and mapping fields, then performing transformations, and validating
  • Finally, securely transfer data to Amazon S3 or Amazon Redshift
• For relational databases, use AWS DMS to ingest data
  • Connect to source data and format it for a target
  • Use source filters and table mappings
  • Perform data validation and writes to many AWS data stores
  • You can also, create a continuous replication task
• For file systems, use DataSync to ingest data
  • Apply filters to transfer a subset of files
  • It can use a variety of file systems as sources and a target, including Amazon S3
  • Securely transfer data between self-managed storage systems and AWS storage services
• For third-party datasets in your pipeline, use AWS Data Exchange
  • Find and subscribe to sources and preview before subscribing
  • Copy subscribed datasets to Amazon S3
  • Receive notifications of updates

AWS Glue for Batch Ingestion

• AWS Glue simplifies batch ingestion tasks.
• AWS Glue’s functions encompass Schema identification, Data cataloging, Job authoring and monitoring, Serverless ETL processing, and ETL orchestration
• AWS Glue is a fully managed data integration service that simplifies ETL tasks.
• AWS Glue crawlers derive schemas from data stores and provide them to the centralized AWS Glue Data Catalog.
• AWS Glue Studio provides visual authoring and job management tools.
• The AWS Glue Spark runtime engine processes jobs in a serverless environment.
• AWS Glue workflows provide ETL orchestration.
• CloudWatch provides integrated monitoring and logging for AWS Glue, including job run insights.
• Based on Apache Spark, it is, fully managed and serverless, optimized for fast queries across large datasets
• It also supports complex, multi-job, multi-crawler ETL processing, is trackable as one entity, and runs on schedule or on demand

Scaling AWS Glue

• Performance goals should focus on on what factors are most important for your batch processing
• Scale AWS Glue jobs horizontally, by adding more workers
• Scale AWS Glue jobs vertically, by choosing a larger type of worker in the job configuration
• Large, splittable files let the AWS Glue Spark runtime engine run many jobs in parallel with less overhead than processing many smaller files
• When using AWS Glue workers, you can increase the number of workers allocated to the job for horizontal scaling
• You can also choose a worker type with larger CPU, memory, and disk space for vertical scaling

Stream Processing

• Tasks to build a real-time stream processing pipeline include:
  • Extract: Put records on the stream (Producers) and provide secure, durable storage
  • Transform/Load: Get records off the stream (Consumers) and Transform records (Consumers)
  • Load/Transform: Analyze or store processed data
• Data moves through the pipeline continuously
• Key characteristics for stream ingestion and processing:
  • Throughput: Plan for a resilient, scalable stream that can adapt to changing velocity and volume
  • Loose coupling: Build independent ingestion, processing, and consumer components
  • Parallel consumers: Allow multiple consumers on a stream to process records in parallel and independently
  • Checkpointing and replay: Maintain record order and allow replay, and support the ability to mark the farthest record processed on failure

Stream Processing Services from AWS

• Kinesis Data Streams - ingests the data stream
• Amazon Data Firehose - transforms and loads the data for future analyses
• Amazon S3 - stores the transformed data
• Amazon Managed Service for Apache Flink - processes and analyzes data in real time
• OpenSearch Service - also can process and analyze data in real time, depending on how it's configured

Kinesis Data Streams

• The stream is a buffer between the producers and the consumers of the stream
• The KPL simplifies the work of writing producers for Kinesis Data Streams
• Data is written to shards on the stream as a sequence of data records
• Data records include a sequence number, partition key, and data blob
• A shard is a uniquely identified sequence of data records
• A partition key determines which shard to use

Amazon Data Firehose

• You can perform no-code or low-code stream ETL
• Ingest data from many AWS services, including Kinesis Data Streams
• Apply built-in and custom transformations
• Deliver data directly to data stores, data lakes, and analytics services

AWS IoT

• By using IoT services you can use MQTT and pub/sub model to communicate with IoT devices
• You can use AWS IoT Core to securely connect, process, and act upon device data
• The AWS IoT Core rules engine transforms and routes incoming messages to AWS services
• The Rules engine routes and transforms data utilizing AWS IoT Core, Amazon Data Firehose and Amazon S3

Scaling Kinesis Data Streams

• Scaling options help manage the throughput of data on the stream
• The amount of writable data, the length of time that that data is stored on a stream, and the throughput each consumer gets can all be scaled CloudWatch provides metrics to help monitor what the stream handles