Data Ingestion with AWS Services

• This module details the tasks to be performed by a data engineer when building an ingestion layer.
• It will describe which AWS services support ingestion tasks.
• It demonstrates how the features in AWS Glue work together to support and automate batch ingestion.
• This module will describe the AWS streaming services that simplify streaming ingestion.
• It will allow the student to identify configuration options in AWS Glue and Amazon Kinesis Data Streams to help scale ingestion processing.
• Finally, this module will describe the characteristics of ingesting Internet of Things (IoT) data by using AWS IoT Core.

Batch & Stream Ingestion Data Flow

• Batch ingestion processes a batch of records as a dataset, on demand, on a schedule, or based on an event.
• Streaming ingestion continually ingests records and processes sets of records as they arrive on the stream.

Data Volume & Velocity

• Data volume and velocity are primary drivers when selecting an ingestion method.
• Batch ingestion applies to sales transaction data sent periodically to a central location, then analyzed overnight to send reports to branches in the morning.
• Streaming ingestion applies to clickstream data from a retailer's website, sending a large volume of small bits of data at a continuous pace to provide product recommendations.

Primary Takeaways

• Batch jobs query the source, transform data, and load it into the pipeline.
• Traditional ETL uses batch processing.
• With stream processing, producers put records on a stream where consumers get and process them.
• Streams handle high-velocity data and real-time processing.

Batch Pipeline Tasks

• Extract - Connect to sources and select data.
• Transform/Load - Identify the source and target schemas, as well as transfer and store data securely and transform the dataset.
• Load/Transform - Load the dataset to durable storage and orchestrate workflows.

Batch Processing Design Characteristics

• Ease of use - Make it flexible, offer low-code options, and offer serverless options.
• Data volume and variety - Handle large data volumes, support different source and target systems, and support different data formats seamlessly.
• Orchestration and monitoring - Support workflow creation, dependency management, bookmarking, job failure alerts, and logging.
• Scaling and cost management - Enable automatic scaling and offer pay-as-you-go options.

Purpose Built Tools

• Batch ingestion involves writing scripts and jobs to perform the ETL or ELT process.
• Workflow orchestration handles interdependencies between jobs and manages failures.
• Characteristics for pipeline design include ease of use, data volume/variety, orchestration/monitoring, scaling, and cost management.

AWS Purpose-Built Tools

• Choose purpose-built tools that match the data type to be ingested and simplify ingestion tasks.
• Amazon AppFlow, AWS DMS, and DataSync each simplify the ingestion of specific data types.
• AWS Data Exchange provides a simplified way to find and subscribe to third-party datasets.
• Amazon AppFlow allows student to ingest data from a software as a service (SaaS) application.

Amazon Appflow

• Creates a connector with filters and map fields
• It helps perform data validation and transformations
• Transfers securely to Amazon S3 or Redshift
• Example use case, ingest customer support tickets from Zendesk

AWS DMS

• Allows student to ingest data from a relational database
• Creates continuous replication task
• Performs data transformation and validation
• Allows to connect to source data and format to a target
• Use source filters and mappings
• Write to many AWS datastores
• Example Use case, ingest line of business transactions from an Oracle database

AWS DataSync

• Facilitates ingest of data from file systems
• Apply filters to transfer subset of files
• Uses a variety of file systems as sources, including Amazon S3
• Transfers securely between self-managed storage systems and AWS storage services
• Example Use Case, ingest on-premises genome sequencing data to Amazon S3

AWS Data Exchange

• Integrates third party datasets into your pipeline
• Find and subscribe to sources
• Allows to preview before subscribing
• Copy subscribed datasets to Amazon S3
• Receives notifications of updates
• Example Use Case, Ingest de-identified clinical data from a third party

AWS Glue

• Simplifies batch ingestion tasks with schema identification, data cataloging, job authoring and monitoring, serverless ETL processing, and ETL orchestration.
• Glue crawlers derive schemas from data stores and provide them to the centralized AWS Glue Data Catalog.
• Glue Studio provides visual authoring and job management tools.
• The Glue Spark runtime engine processes jobs in a serverless environment.
• Glue workflows provide ETL orchestration.
• CloudWatch provides integrated monitoring and logging.

AWS Glue Horizontal Scaling

• Scaling option that increases the number of workers allocated to the job.
• It is best suited for large spitable datasets.
• An example use case is processing a large .csv file

AWS Glue Vertical Scaling

• Scaling option that increases the worker type with larger CPU, Memory, and disk space.
• It is best suited for memory-intensive applications.
• An example use case is machine learning

Kinesis Data Streams

• Kinesis Data Streams provide scaling options to manage throughput on the stream.
• Scale how much data is written to the stream, how long data is stored, and how much throughput each consumer gets.
• CloudWatch provides metrics to monitor how the stream handles data being written to and read from it.
• The stream is a buffer between producers and consumers.
• Key information, the KPL simplifies the work of writing producers for Kinesis Data Streams, data is written to shards, and data records include a sequence number, partition key, and data blob.
• Amazon Data Firehose delivers streaming data directly to storage, like Amazon S3 and Redshift.
• Amazon Managed Service for Apache Flink performs real-time analytics.
• Plan for a resilient, scalable stream to adapt to changing velocity/volume.
• Build independent ingestion, processing, and consumer components and allow multiple consumers to process records in parallel+independently.
• Maintain record order and allow replay, and mark the farthest record processed on failure.

AWS IoT core for data analytics

• Designed to connect securely, process, and act on device data
• Has feature to filter and transform data
• Routes data to other AWS services, including streaming storage services.
• Allows you to use MQTT and a pub/sub model to communicate with loT devices.
• AWS IoT Core rules engines transforms and routes incoming messages to AWS services