Big Data Processing Frameworks

Module objectives

• Compare and select the big data processing framework best suited for specific workloads
• Hadoop and Amazon EMR principles and how they facilitate data processing in AWS
• Launch, configure, and manage an Amazon EMR to support big data processing

Big Data Processing Concepts

• Storage
• Ingestion
• Processing
• Analysis & Visualization
• Big data processing can use storage with Amazon S3 and Lake Formation Data Catalog
• SQL-based ELT allows querying with Amazon Redshift
• Big data processing can use Amazon EMR and AWS Glue
• Near real-time ETL can use Kinesis Data Analytics
• Spark streaming can be used on Amazon EMR or AWS Glue

Types of Data Processing

• Batch data processing involves infrequently (cold) accessed data querying and input data is processed in batches at varying intervals
• Batch data processing tolerates structured and unstructured data and allows the deep analysis of big datasets
• Amazon EMR and Apache Hadoop are examples of batch data processing
• Streaming data processing involves frequently accessed (hot) data querying and data is processed sequentially and incrementally in near real-time
• Streaming data processing is capable of processing less predictable data at scale and enables analysis of continually generated data
• Examples of streaming data processing include Amazon Kinesis Data Streams and Apache Spark Streaming.

Frameworks Supporting Batch and Stream Processing

• Batch processing frameworks include Apache Spark, Apache Hadoop MapReduce, Apache Hive, and Apache Pig
• Stream processing frameworks include Amazon Kinesis, Apache Spark Streaming, AWS Lambda, Apache Flink, and Apache Storm
• Apache Spark and Apache Hive can support both stream and batch processing

Key Takeaways: Big Data Processing Concepts

• Big data processing is categorized into batch and streaming.
• Batch data generally involves cold data and analytics workloads with longer processing times
• Streaming data comes from multiple sources and is processed sequentially and incrementally.
• Specialized frameworks that benefit big data processing are available for batch and stream processing

Apache Hadoop Characteristics

• Hadoop is an open-source, distributed processing framework.
• Large amounts of data can be processed and stored in a distributed fashion
• Tasks are mapped to nodes within clusters.
• Hadoop is composed of Hadoop Distributed File System (HDFS), YARN, MapReduce, and Hadoop Common

Hadoop Distributed File System (HDFS)

• The HDFS client sends a data request to the NameNode
• The NameNode retrieves the data's metadata
• The client then reads from or writes to a DataNode
• Data replication is performed across DataNodes in the Hadoop cluster

Yet Another Resource Negotiator (YARN)

• YARN manages resources in a Hadoop cluster
• Clients submit jobs, which are managed by the Resource Manager, Application Manager and Scheduler.
• Node Managers manage containers, which may contain Application Leaders that execute functions

Hadoop MapReduce

• Hadoop MapReduce processes data in parallel
• The input data is split into chunks for the Map Tasks
• Map tasks apply MapCode to each chunk
• Reduce Tasks applies ReduceCode and then the final output is generated

Processing Data with Hadoop MapReduce

• A single 500MB file from Amazon S3 is split into four parallel HTTP requests.
• The Hadoop default split results in four files, using four of the eight available mappers

Common Hadoop Frameworks

• Apache Flink is a streaming data flow engine and utilizes APIs optimized for both distributed streaming and batch processing, performing data source transformations and is categorized into DataSets and DataStreams
• Apache Hive is an open-source, SQL-like data warehouse solution which helps to avoid complex MapReduce programs in lower-level computing languages and integrates with AWS services like Amazon S3 and DynamoDB
• Presto is an open-source, in-memory SQL query engine emphasizing faster querying for interactive queries and is based on its own engine performing in-memory operations
• Apache Pig is a textual data flow language and allows analysis of large datasets using parallel processing with automatic install when an Amazon EMR cluster is launched and supports interactive development.

Key Takeaways: Apache Hadoop

• Hadoop includes a distributed storage system, HDFS.
• Hadoop splits data into smaller data blocks when storing in HDFS.
• MapReduce processes large datasets with a parallel, distributed algorithm on a cluster.

Apache Spark Characteristics

• Is an open-source, distributed processing framework.
• Utilizes in-memory caching and optimized query processing
• Supports code reuse across multiple workloads.
• Clusters consist of leader and worker nodes.

Spark Clusters

• SparkContext connects to a cluster manager, such as YARN or Kubernetes and coordinates the execution of tasks on worker nodes.
• Worker nodes have Executors that run tasks and Cache

Spark Components

• Spark SQL: Used for working with structured data, including querying data using SQL.
• Spark GraphX: A library for manipulating and analyzing graph data.
• Spark Streaming: Used for building real-time streaming applications, processing data from sources like Kafka or Kinesis.
• Spark MLlib: A machine learning library that provides a variety of algorithms for tasks like classification, regression, and clustering.
• Spark Core: The underlying engine for parallel computing in Apache Spark.

Key Takeaways: Apache Spark

• Apache Spark has in-memory processing, reduces the number of steps in a job, and reuses data across multiple parallel operations.
• Data is reused with an in-memory cache to speed up ML algorithms.

Amazon EMR Characteristics

• Managed cluster platform.
• Big data solution for petabyte-scale data processing, interactive analytics, and machine learning.
• Processes data for analytics and BI workloads using big data frameworks.
• Transforms and moves large amounts of data into and out of AWS data stores.

Clusters and Nodes

• The central component of Amazon EMR is the cluster.
• Each instance in the cluster is a node
• A node type is the role that each node serves.
• Node types consist of main, core, and task.

Amazon EMR Service Architecture

• Storage: HDFS, EMR File System (EMRFS), local file system
• Cluster resource management: YARN
• Data processing frameworks: Hadoop MapReduce, Apache Spark
• Applications and programs: Apache Spark, Apache Hive, Apache Flink, Apache Hadoop
• The Amazon EMR architecture consists of multiple layers.

Processing Data in Amazon EMR

• Source data is in AWS Cloud, in a service like S3
• Data is processed in the Amazon EMR cluster environment
• Main nodes divide tasks to core nodes, and outputs are generated
• The dataset is submitted, output is processed and second input processed before the data can be written to the intended output dataset

Key Takeaways: Amazon EMR

• Amazon EMR allows automated installations of common big data projects.
• The Amazon EMR service architecture has four layers: storage, cluster resource management, data processing frameworks, and applications and programs

Launching and Configuring Amazon EMR Clusters

• EMR clusters can be created through interactive, command line, and API methods

Cluster Characteristics

• Long-running clusters include persistent clusters, interactive jobs submission, persistent data (until shutdown), and large dataset processing.
• Transient clusters shut down after data is processed and stored, and typically read code and data from Amazon S3 at startup
• Transient clusters do not persist HDFS data after termination

Connecting to Your Cluster

• External connections connect to the main node Amazon EC2 instance.
• The public DNS name for connections is exposed by the main node.
• The main node has security group rules created by Amazon EMR.
• Core and task nodes have separate security group rules created by Amazon EMR.
• The cluster must be running for connections to be made.

Scaling Your Cluster Resources

• Scaling is accomplished automatically or manually.
• Automatic scaling has two options: Amazon EMR managed scaling and custom automatic scaling policy.

Key Takeaways: Managing Amazon EMR Clusters

• EMR clusters can be launched through interactive, command line, and API methods.
• EMR clusters are categorized as long-running or transient based on their usage.
• External connections to EMR clusters are made through the main node.

Apache Hudi Characteristics

• Apache Hudi is an open-source data management framework.
• This allows record-level insert, update, upsert, and delete capabilities
• It integrates with Apache Spark, Apache Hive, and Presto
• The Hudi DeltaStreamer utility has the ability to create or update Hudi datasets
• Datasets are organized into a partitioned directory structure under a base path, similar to a Hive table

Key Hudi Concepts

• Copy on Write (CoW) stores data in columnar format (Parquet), which updates a new version of files during write processes and is the default storage type
• Merge on Read (MoR) stores data in a combination of columnar format (Parquet) and row-based (Avro) formats so updates are logged to row-based delta files and compacted as needed and provides read-optimized, incremental, and real-time views

Key Takeaways: Apache Hudi

• Apache Hudi can ingest and update data in near real time.
• Hudi maintains metadata of actions to ensure that they are atomic and consistent.

Module Summary

• Big data processing frameworks that best supports your workloads can be compared and selected
• The principles of Apache Hadoop and Amazon EMR, and how they support data processing in AWS can be explained
• Amazon EMR can be launched, configured, and managed to support big data processing

Sample Exam Question

• Connectivity issues to the main node of an Amazon EMR cluster via SSH might be caused by The ElasticMapReduce-main security group needing an inbound rule that allows SSH access.