AWS Data Ingestion

• The module prepares you to:
  • List data engineer tasks for building an ingestion layer
  • Describe how AWS services support ingestion tasks
  • Illustrate how AWS Glue features automate batch ingestion
  • Describe how AWS streaming services simplify streaming ingestion
  • Identify configuration options in AWS Glue and Amazon Kinesis Data Streams
  • Describe ingesting Internet of Things (IoT) data using AWS IoT Core

Batch vs. Streaming Ingestion

• Batch ingestion processes records as a dataset on demand, schedule, or event.

• Streaming ingestion continually ingests records and processes them as they arrive.

• Important factors when choosing an ingestion method are data volume and velocity

• Batch ingestion involves sales data sent periodically, analyzed overnight, and reported in the morning.

• Streaming ingestion involves processing clickstream data immediately to provide product recommendations.

• Batch jobs query the source, transform the data, and load it into the pipeline.

• Traditional ETL uses batch processing.

• With stream processing, producers put records on a stream for consumers to process.

• Streams handle high-velocity data and real-time processing.

Batch Processing Pipeline

• Tasks for building batch pipeline:
  • Extract by connecting to sources and selecting data
  • Transform/Load by identifying source and target schemas and securely transferring data
  • Load/Transform by transforming the dataset and loading it to durable storage
  • Orchestrate workflows
• Key characteristics for batch processing design choices:
  • Ease of use: flexible, low-code/no-code options, serverless options
  • Data volume and variety: handle large volumes, support disparate systems/formats
  • Orchestration and monitoring: support workflow creation, dependency management, bookmarking, alerting, and logging
  • Scaling and cost management: automatic scaling, pay-as-you-go options
• Batch ingestion involves writing scripts and jobs to perform the ETL or ELT process.
• Workflow orchestration is helpful to handle interdependencies between jobs and manage failures.
• Pipeline design should include ease of use, data volume, variety, orchestration, monitoring, scaling, and cost management.

AWS Purpose-Built Tools

• AWS offers purpose-built tools that match data types and simplify ingestion tasks.
• SaaS apps use Amazon AppFlow
• Creates a connector with filters
• Maps fields and performs transformations
• Performs validation
• Securely transfers to Amazon S3 or Amazon Redshift
• Suitable for use cases like ingesting customer support ticket data from Zendesk.
• Relational databases use AWS DMS
  • Connects to source data
  • Formats the data for a target
  • Uses source filters and table mappings
  • Performs data validation
  • Writes to many AWS data stores
  • Creates a continuous replication task
  • Suitable for use cases like ingesting line of business transactions from an Oracle database
• File shares use DataSync
  • Applies filters to transfer a subset of files
  • Uses a variety of file systems as sources and target, including Amazon S3 as a target
  • Securely transfers data between self-managed storage systems and AWS storage services
  • Suitable for use cases such as ingesting on-premises genome sequencing data to Amazon S3
• Third-party datasets use AWS Data Exchange
  • Finds and subscribes to sources
  • Previews before subscribing
  • Copies subscribed datasets to Amazon S3
  • Receives notifications of updates
  • Suitable for use cases such as ingesting de-identified clinical data from a third party
• Amazon AppFlow, AWS DMS, and DataSync simplify specific data type ingestion.
• AWS Data Exchange simplifies subscription to third-party datasets.
• These tools support secure connections, data store integration, automated updates, CloudWatch monitoring, selection, and transformation.

AWS Glue

• AWS Glue simplifies batch ingestion with schema identification, data cataloging, job authoring/monitoring, serverless ETL processing, and ETL orchestration.
• AWS Glue crawlers derive schemas from data stores for the AWS Glue Data Catalog.
• Job authoring enables low-code job creation for ETL management
• AWS Glue Studio provides visual authoring and job management tools.
• The AWS Glue Spark runtime engine processes jobs in a serverless environment.
• AWS Glue workflows provide ETL orchestration.
• CloudWatch provides integrated monitoring and logging for AWS Glue, including job run insights.

Scaling Considerations for Batch Processing

• To scale AWS Glue jobs horizontally, more workers can be added
  • Suitable for working with large, splittable datasets as in processing a large .csv file
• To scale AWS Glue jobs vertically, a type of worker in the job configuration with larger CPU, memory and disk space can be chosen
  • Suitable for memory-intensive or disk-intensive applications, i.e. Machine Learning transformations
• Performance goals should focus on important factors for batch processing.
• Large, splittable files let the AWS Glue Spark runtime engine run many jobs in parallel with less overhead than processing many smaller files.

Real-Time Stream Processing Pipeline

• For real-time stream processing pipeline:
  • Extract puts records on the stream (Producers)
  • Transform/Load provides secure, durable storage and get records off the stream(Consumers)
  • Load/Transform transforms records(Consumers) and analyze or store processed data
  • Data moves through the pipeline continuously
• Key characteristics of stream ingestion and processing:
  • Throughput: Plan for a resilient, scalable stream that can adapt to changing velocity and volume
  • Loose coupling: Build independent ingestion, processing, and consumer components
  • Parallel consumers: Allow multiple consumers on a stream to process records in parallel and independently
  • Checkpointing and replay: Maintain record order and allows replay; support the ability to mark the farthest record processed on failure

Services that Simplify Stream Ingestion

• Kinesis Data Streams ingest and store data from various sources
  • Web, Sensors, Devices, Social Media, etc
• Amazon Data Firehose transforms and loads data for future analysis
  • Amazon S3
• Amazon Managed Service for Apache Flink processes and analyzes data in real-time
  • OpenSearch Service
• For Kinesis Data Streams:
  • A shard uniquely identifies a sequence of data records.
  • A partition key determines which shard to use.
  • A data record contains a sequence number, partition key, and data blob
• Amazon Data Firehose performs no-code or low-code streaming ETL
  • It can ingest from many AWS services, including Kinesis Data Streams
  • Apply built-in and custom transformations
  • Deliver directly to data stores, data lakes, and analytics services.
• Amazon Managed Service for Apache Flink can query and analyze streaming data
  • It can ingest from other services, including Kinesis Data Streams
  • Enrich and augment data across time windows
  • Build Applications in Apache Flink
  • Use SQL, Java, Python, or Scala.
• Monitoring a Kinesis Data Stream
  • CloudTrail tracks API actions, including changes to stream configuration and new consumers
  • CloudWatch tracks record age, throttling, and write and read failures

Stream Scaling Considerations

• Three scaling configurations for Kinesis Data Streams
  • Duration of data availability: can set how long stream records are avaiable
  • Write capacity: Choose the stream capacity mode: on-demand or provisioned
  • Read capacity: Choose consumer types: shared fan-out or enhanced fan-out
• Components:
  • Producers
  • Data stream
  • Consumers
• The stream is a buffer between producers and consumers.
• KPL simplifies the work of writing Kinesis Data Streams producers.
• Data is written to shards on the stream as a sequence of data records.
• Records include a sequence number, partition key, and data blob.
• Amazon Data Firehose delivers streaming data directly to storage, including Amazon S3 and Amazon Redshift.
• Amazon Managed Service for Apache Flink performs real-time analytics on data as it passes through the stream.
• Kinesis Data Streams provides scaling options to manage throughput and storage.
• CloudWatch provides metrics to monitor data handling in the stream.

Ingesting IoT Data

• AWS IoT services use MQTT and a pub/sub model for IoT device communication.
• AWS IoT Core securely connects, processes, and acts upon device data.
• The AWS IoT Core rules engine transforms and routes incoming messages to AWS services.