Processing Big Data with Amazon EMR

• Processing Big Data

Module Objectives

• Compare and select the best big data processing framework for workloads.
• Explain the principles of Apache Hadoop and Amazon EMR and how they support data processing in AWS.
• Launch, configure, and manage an Amazon EMR cluster to support big data processing.

Big Data Processing Concepts

• Big data processing uses an iterative data pipeline that ingests data from various sources.
• The data is then stored and processed before being analyzed and visualized.
• In modern architecture, this might involve SQL-based ELT, big data processing tools like Amazon EMR and AWS Glue, and near real-time ETL with Kinesis Data Analytics.
• Spark streaming can be used on Amazon EMR or AWS Glue, and data can be transformed for further processing or consumption.

Types of Data Processing

• Batch data processing queries infrequently accessed (cold) data.
• Batch processing works on input data in batches at varying intervals.
• Batch processing tolerates both structured and unstructured data.
• Batch processing is capable of deep analysis of big datasets.
• Examples of batch data processing include Amazon EMR and Apache Hadoop.
• Streaming data processing queries frequently accessed (hot) data.
• Streaming processing works sequentially and incrementally in near-real-time.
• Streaming processing can handle less predictable data on a massive scale.
• Streaming processing enables analysis of continually generated data.
• Examples of streaming data processing include Amazon Kinesis Data Streams and Apache Spark Streaming.

Frameworks for Batch & Stream Processing

• Apache Spark supports both Batch and Stream Processing
• Amazon Kinesis and Apache Spark Streaming support Stream Processing
• Apache Hadoop MapReduce supports Batch Processing
• AWS Lambda and Apache Flink support Stream Processing
• Apache Hive and Apache Pig support both Batch Processing and Stream Processing
• Apache Storm supports Stream Processing

Key Takeaways: Big Data Processing

• Big data processing is usually divided into batch and streaming.
• Batch data typically involves "cold" data and analytics workloads with longer processing times.
• Streaming data involves many data sources that must be processed sequentially and incrementally.
• Batch processing and stream processing benefit from specialized big data processing frameworks.

Apache Hadoop Characteristics

• Apache Hadoop is an open-source, distributed processing framework.
• Hadoop enables distributed storage and processing for large amounts of data.
• Hadoop maps tasks to nodes within clusters of servers.
• Hadoop's components include HDFS (Hadoop Distributed File System), YARN, MapReduce, and Hadoop Common.
• Clusters consist of main nodes and worker nodes.

Hadoop Distributed File System (HDFS)

• HDFS is a distributed file system designed to store and process large datasets across clusters of commodity hardware.
• A client sends a data request, which is managed through metadata retrieved by the NameNode.
• The system stores data in blocks and supports read/write operations.
• HDFS provides data replication to ensure data reliability and availability.

Yet Another Resource Negotiator (YARN)

• YARN is a resource management framework used in Apache Hadoop to allocate system resources to various applications running in a Hadoop cluster.
• The key components of YARN include:
  • Resource Manager, which manages the allocation of resources across the cluster.
  • Application Manager.
  • Node Manager, which manages resources on individual nodes.
• The process includes clients submitting jobs, resource negotiation, and containers for application execution.

Hadoop MapReduce

• Hadoop MapReduce is a programming model and software framework for distributed processing of large datasets on computer clusters.
• There are 4 steps to MapReduce; Input, Map Tasks, Reduce Tasks and Output.

Processing Data with Hadoop MapReduce

• A single file from Amazon S3 is split into four parallel HTTP requests.
• The Hadoop default split results in four files, using four of the eight available mappers.

Common Hadoop Frameworks to Process Big Data

• Apache Flink:
  • Streaming data flow engine.
  • Uses APIs optimized for distributed streaming and batch processing.
  • Provides the ability to perform transformations on data sources.
  • API is categorized into DataSets and DataStreams.
• Apache Hive:
  • Open-source, SQL-like data warehouse solution.
  • Helps avoid writing complex MapReduce programs in lower-level computing languages.
  • Integrates with AWS services like Amazon S3 and Amazon DynamoDB.
• Presto:
  • Open-source, in-memory SQL query engine.
  • Emphasizes faster querying for interactive queries.
  • Operates in-memory, based on its own engine.
• Apache Pig:
  • Textual data flow language.
  • Performs analysis of large datasets using parallel processing.
  • Supports automatic install when an Amazon EMR cluster is launched.
  • Supports interactive development.

Key Takeaways: Apache Hadoop

• Hadoop includes a distributed storage system (HDFS).
• Data within HDFS is split into smaller data blocks.
• MapReduce processes large datasets with a parallel, distributed algorithm on a cluster.

Apache Spark Characteristics

• Apache Spark is an open-source, distributed processing framework.
• It uses in-memory caching and optimized query processing.
• Spark supports code reuse across multiple workloads.
• Spark clusters have leader and worker nodes.

Spark Clusters

• Spark clusters consist of a Driver Program and Worker Nodes, managed by a Cluster Manager.
• The Driver Program contains a SparkContext, which coordinates the execution of tasks on the Worker Nodes.
• Worker Nodes have Executors, which perform the actual tasks and use in-memory caching.
• The tasks are distributed and processed in parallel across the cluster.

Spark Components

• Spark SQL
• Spark GraphX
• Spark Streaming
• Spark MLlib
• Spark Core (R, Python, Scala, Java)

Key Takeaways: Apache Spark

• Apache Spark performs processing in-memory, reduces the number of steps in a job, and reuses data across multiple parallel operations.
• Spark reuses data by using an in-memory cache to speed up ML algorithms.

Amazon EMR Characteristics

• Amazon EMR is a managed cluster platform.
• EMR is a big data solution for petabyte-scale data processing, interactive analytics, and machine learning.
• Amazon EMR processes data for analytics and BI workloads using big data frameworks.
• It can transform and move large amounts of data into and out of AWS data stores.

Clusters and Nodes

• The central component of Amazon EMR is the cluster.
• Each instance is a node.
• The role that each node serves is the node type.
• EMR uses three node types: main, core, and task.

Amazon EMR Service Architecture

• Storage: HDFS, EMR File System (EMRFS), local file system
• Cluster Resource Management: YARN
• Data Processing Frameworks: Hadoop MapReduce, Apache Spark
• Applications and Programs: Apache Spark, Apache Hive, Apache Flink, Apache Hadoop

Processing Data in Amazon EMR

• Data flows from the AWS Cloud to the main node and core nodes in the EMR cluster.
• Steps include:
  • Submitting input dataset
  • Processing output
  • Processing second input
  • Writing output dataset

Key Takeaways: Amazon EMR

• Amazon EMR can be used to perform automated installations of common big data projects.
• The Amazon EMR service architecture consists of four layers:
  • Storage
  • Cluster resource management
  • Data processing frameworks
  • Applications and programs

Launching and Configuring Amazon EMR Clusters

• Three methods are available to launch EMR clusters: interactive mode, command-line mode, and API mode.

Cluster Characteristics

• Long-running clusters:
  • Persistent clusters
  • Interactive job submission
  • Persistent data until shutdown
  • Large dataset processing
• Transient clusters:
  • Shut down after data is processed and stored
  • Typically read code and data from Amazon S3 at startup
  • Do not persist HDFS data after termination

Connecting to your Cluster

• External connections are made to the main node Amazon EC2 instance.
• The main node exposes the public DNS name for connections.
• Amazon EMR creates security group rules for the main node, core and task nodes.
• Clusters must be running for connections to be made.

Scaling Your Cluster Resources

• Scaling is accomplished automatically or manually.
• Two options are available for automatic scaling:
  • Amazon EMR managed scaling
  • Custom automatic scaling policy

Key Takeaways: Managing Your Amazon EMR Clusters

• EMR clusters can be launched using interactive, command-line, and API methods.
• EMR clusters are characterized as long-running or transient, based on their usage.
• External connections to EMR clusters can only be made through the main node.

Apache Hudi Characteristics

• Apache Hudi is an open-source data management framework.
• It provides record-level insert, update, upsert, and delete capabilities.
• Hudi integrates with Apache Spark, Apache Hive, and Presto.
• The Hudi DeltaStreamer utility can create or update Hudi datasets.
• It organizes a dataset into a partitioned directory structure under a base path, similar to a Hive table.

Key Hudi Concepts

• Hudi dataset storage types:
  • Copy on Write (CoW) - Data is stored in columnar format (Parquet), each update creates a new version of the files, also the default storage type.
  • Merge on Read (MoR) - Data is stored in combination of columnar format (Parquet) and row-based (Avro) formats with updates logged to row-based delta files and compacted as needed.
• Hudi view options:
  • Read-optimized view
  • Incremental view
  • Real-time view

Key Takeaways: Apache Hudi

• Apache Hudi provides the ability to ingest and update data in near real-time.
• Hudi maintains metadata of the actions performed to ensure atomicity and consistency.

Module summary

• Compare and select the big data processing framework that best supports your workloads.
• Explain the principles of Apache Hadoop and Amazon EMR, and how they support data processing in AWS.
• Launch, configure, and manage an Amazon EMR cluster to support big data processing.

Additional Notes

• The knowledge check is delivered online within your course.
• The knowledge check includes 10 questions based on material presented on the slides and in the slide notes.
• You can retake the knowledge check as many times as you like.

Sample Exam Question

• A data engineer has deployed an Amazon EMR cluster to support their ML workload but SSH connections to the active EC2 instance are failing.
• Select the correct response to the correct action.
• The ElasticMapReduce-main security group needs an inbound rule that allows SSH access.