Hadoop Ecosystem Quiz

Big Data Analytics - Chapter III: Software Layers

• Hadoop foundations are crucial for understanding the system.
• The HDFS file system is key; understanding and applying the Map/Reduce paradigm is vital.
• Servers, racks, and network architectures are parts of the Hadoop ecosystem.
• Layers within Hadoop's architecture are interconnected.

RDBMS vs. Hadoop Properties

• Traditional RDBMS typically handles gigabytes of data whereas Hadoop manages petabytes.
• RDBMS supports interactive and batch access while Hadoop mainly supports batch processing.
• RDBMS allows for frequent read and write operations, while Hadoop favors write-once, read-many times.
• RDBMS works with static schemas, whereas Hadoop uses dynamic schemas.
• RDBMS ensures high integrity, whereas Hadoop's integrity is relatively lower.
• RDBMS scaling is nonlinear, while Hadoop scaling is linear.

Advantages of Hadoop

• Hadoop handles vast amounts of data effectively.
• It is an economical solution.
• Hadoop's architecture allows for efficient processing.
• Hadoop is scalable to manage growing data volumes.
• Hadoop is robust and reliable

Applications Not for Hadoop

• Low-latency data access is not Hadoop's forte.
• HBase is a better choice for low-latency needs.
• Processing large numbers of small files is not ideal on Hadoop.
• File system metadata stored in memory limits the number of files Hadoop can process effectively.
• Multiple writers and arbitrary file modifications are not supported by Hadoop.

Hadoop Cluster - Servers, Racks, and Networks

• Hadoop Clusters usually have a two-level arrangement.
• Nodes in the cluster are typically standard/commodity computers.
• The typical number of nodes per rack is 30-40.
• Uplink connections from a rack are typically 3-4 Gigabit.
• Rack-internal connections often use 1 Gigabit connections.
• An aggregation switch connects racks to each other.
• An 8-Gigabit connection will connect to the aggregated switch.
• A 1-Gigabit connection connects to the racks.
• Standard computer components are used for the network.

The Core Apache Hadoop Project

• Hadoop Common provides Java libraries required by other Hadoop modules.
• HDFS (Hadoop Distributed File System) handles data storage.
• Hadoop YARN (Yet Another Resource Negotiator) manages scheduling and cluster resource management.
• Hadoop MapReduce is a programming model for large-scale data processing.

Hadoop Layers

• Hadoop MapReduce (data processing)
• YARN (cluster resource management)
• HDFS (storage)
• Spark (data processing)
• Flink (data processing)
• Others

Hadoop Related Subprojects

• Pig is a high-level language for data analysis.
• HBase is table storage for semi-structured data.
• ZooKeeper coordinates distributed applications.
• Hive is an SQL-like query language.
• Mahout is a machine learning library.

Hadoop Distributed File System (HDFS)

• HDFS is a distributed file system built for very large files.
• The design features of HDFS include a very large distributed file system with 10,000+ nodes, 100 million files, and 10 PB of data.
• Data in HDFS is replicated for fault tolerance. Replication is critical for availability and to recover from hardware failure.
• The design also includes optimization for batch processing to match computing to where data resides.
• HDFS supports heterogeneous operating systems (OS).

HDFS Design

• HDFS uses a single namespace for the entire cluster.
• The system is coherent and data is accessible using a write-once-read-many approach.
• Clients only append to existing files.
• Files are broken into blocks (typically 64-128 MB).
• Blocks are replicated on multiple DataNodes.
• Clients can find block locations from the NameNode.
• Data access is direct from DataNodes.

HDFS Architecture

• Namenode manages the metadata of files and blocks.
• DataNodes store the data blocks.
• Metadata includes information about files, and their replication scheme.
• Clients interact with the Namenode to locate blocks and with DataNodes to retrieve data.
• Replication is critical to ensure high availability in case of failure of a particular DataNode.

NameNode Functions

• The NameNode manages the file system namespace.
• The NameNode maps blocks to DataNodes and file names to sets of blocks.
• The NameNode manages cluster configuration.
• The NameNode manages replication of blocks.
• To ensure high availability, both active and standby NameNodes are necessary, operating as dedicated master nodes.

How Files Are Stored in HDFS

• Data files are divided into blocks and distributed to DataNodes.
• Each block is replicated for redundancy (default is 3 times).
• The Namenode stores metadata about files and blocks, including block locations.

NameNode Metadata

• Namenode metadata is stored in main memory.
• There is no demand paging for metadata.
• Metadata types include lists of files, blocks, DataNodes, and file attributes such as creation time.
• A transaction log records file creations and deletions.

HDFS NameNode Availability

• The NameNode must be running for the cluster to be accessible.
• High availability mode (in CDH4 and later) features two NameNodes (active and standby).
• Classic mode uses one NameNode with a secondary helper node for bookkeeping but no backup.

Secondary NameNode

• The Secondary NameNode copies the FsImage and the transaction log from the Namenode to a temporary directory.
• The Secondary NameNode merges the copied FsImage and Transaction Log into a new FsImage in a directory.
• The secondary Namenode ensures a checkpoint in HDFS for recovery if the primary Namenode fails.
• Edits regarding data files are copied to the Secondary node for safety from primary.

Standby Name Node-QJM

• Hadoop 2 introduced high availability with two NameNodes.
• One is active, handling client requests, while the other is standby, synchronized to take over if the active one fails.
• The Quorum Journal Manager (QJM) runs on each NameNode facilitating communication with Journal Nodes using RPC, handling namespace modifications, and maintaining data synchronization.

ZooKeeper

• ZooKeeper coordinates distributed applications.
• It monitors the health of the NameNode and other components.
• It handles block reports and ensures data synchronization.

DataNode

• DataNodes are block servers.
• They store data blocks on local file systems with checksums (e.g., CRC).
• DataNodes are responsible for serving data and metadata to clients.
• They periodically report block status to the Namenode.
• Datanodes facilitate pipelining of data by forwarding data to other specified datanodes, improving processing efficiency and minimizing network overhead.

Block Placement

• Hadoop places data replications on local nodes, remote racks, and other remote racks for fault tolerance and better performance.
• Placement follows a rack awareness algorithm, replicating data across multiple racks.
• Clients access the nearest replica for optimized read performance.

Heartbeats

• DataNodes periodically send heartbeats to the Namenode.
• Frequency is often once every 3 seconds.
• Heartbeats enable Namenode to detect DataNode failures.

NameNode as Replication Engine

• After detecting DataNode failures, the NameNode selects new DataNodes to host those blocks (replicas).
• The NameNode balances disk usage across all DataNodes.
• The NameNode balances communication traffic to DataNodes.

Data Pipelining (i)

• Clients request a list of DataNodes to store a block's replicas.
• The data is written to the first Datanode in the sequence of placement on the cluster.
• Pipelining occurs where the first DataNode delivers the block data to the next DataNode in the sequence (pipeline).
• The process continues until the appropriate number of replicas is stored as requested by the client.

Data Pipelining (ii)

• The writing procedure uses a client JVM to signal a write request to HDFS, including the IP addresses of the target DataNodes.
• The request is processed by the NameNode and sent to the appropriate Core Swtich for networking across the data nodes.
• The DataNodes that are ready store the block.

Rebalancer

• The goal of the rebalancer is to ensure similar disk space utilization across all the DataNodes within the cluster by rebalancing data distribution.
• Rebalancing occurs frequently, especially after the addition of a new DataNode.

User Interface

• Commands for HDFS users (e.g., creating directories, reading/writing files).
• Commands for HDFS administrators (e.g., monitoring, de-commissioning DataNodes).
• A web interface for monitoring and administration.

Introduction to Hadoop YARN

• YARN is a resource manager crucial for enterprise Hadoop.
• It provides centralized resource management, security, and data governance tools across Hadoop clusters.

Applications that run on YARN

•  A variety of applications/programs can run on top of YARN.

Before/After 2012: Hadoop Versions

• Pre-2012 Hadoop relied primarily on the MapReduce programming model for its processing tasks.
• Post-2012, Hadoop 2.7 and later versions greatly broadened the capabilities to support other processing techniques beyond MapReduce.

YARN Cluster Basics

• The ResourceManager (RM) is the master daemon that directs resource allocation, tracks cluster resources, and schedules work.
• NodeManagers are worker daemons on the worker nodes to handle tasks.

YARN Resource Monitoring (i) & (ii)

• YARN uses v-cores and memory as primary resources.
• Node Managers track their own resources and report to the RM.
• The RM manages the total resources in the cluster.

Yarn Container

• A container in YARN is a request for resources.
• Containers manage the resources allocated (vcores and memory) to run a program.
• Containers are run as processes.

YARN Application and ApplicationMaster

• YARN applications comprise tasks (Map/Reduce).
• ApplicationMaster manages running tasks and coordinates the application execution.

Interactions among YARN Components (i), (ii), (iii), (iv), & (v)

• Steps outlining how applications interact with YARN components: submission, container request, ApplicationMaster launch processes, task assignment, task execution, and exit.

How Applications run on YARN - Steps 1, 2, 3, 4, 5, 6, 7

• Step-by-step details on how applications using YARN operate within the Hadoop distributed computing system. 

Schedulers

• YARN's scheduler manages cluster resources, following a defined policy and allowing constraints like capacity, fairness, and SLA.

FIFO, Capacity, and Fair Schedulers

•  Algorithms/protocols for managing jobs (first-come, first-served, capacity allocation, balanced scheduling).

### MapReduce - Overview

• MapReduce is a programming model for executing parallel computations over large datasets.
• It consists of map and reduce phases (map & reduce functions).

MapReduce: Terminology

• Explains job execution in MapReduce.
• Defining 'job' within MapReduce (a program).
• Understanding a 'task' as a part of execution.
• Clarifying 'task attempts' to address failures within distributed tasks.

Hadoop Components: MapReduce

• Mappers operate on one HDFS block at a time; local data processing when possible.
• Mappers generate results as intermediate value/key pairs and send to reducers.
• Reducers aggregate data in the data processing phase by combining similar value/key pairs.

Mappers Run in Parallel

• In parallel execution, Mappers run on multiple nodes, processing/gathering information locally, to concurrently process/improve resource utilization and minimize network overhead.

MapReduce: The Mapper

• The Mapper function reads data, typically as key value pairs (in text formatting).
• Mappers process/transform data according to the program's requirements.
• Mappers produce key-value pairs as intermediate results, then the reducer collects those.

MapReduce: The Reducer

• The Reducer function combines intermediate results by processing/combining key-related value pairs.
• Reducer tasks receive sorted data from Mappers, combining value pairs for that key, creating a final output.
• Reducers output final results as (key, value) pairs.

Features of MapReduce

• Automated parallelization and data distribution.
• Built-in fault tolerance (processes failures).
• Clean abstraction that hides underlying cluster management.
• Tools for monitoring execution status.

Word Count Example

• A simple MapReduce example demonstrating word counting.
• The application logic for (key, value) pairs in the execution process.

Example

• Steps of distributed execution demonstration.
• Shows the generation of multiple tasks and their execution on different cluster components.

SORT and SHUFFLE

• Demonstrates how Hadoop sorts and rearranges intermediate data before reducing it.

MapReduce - Word Count Example Flow

•  Visual representation of MapReduce word count processing from input to output with various intermediate results.

MAPREDUCE-Steps

• Detail steps within MapReduce algorithm/process.
• Includes input/splitting, mapping, combining, shuffling/sorting, reducing, and output generation.

Input and Output Formats

• Formats for data input and output within MapReduce and how to specify them.
• Standard options/formats such as TextInputFormat, TextOutputFormat.

INTRODUCTION TO YARN AND MAPREDUCE INTERACTION

•  Introduction to interactions between YARN and MapReduce.

MapReduce on Yarn

• Description of the mapping of MapReduce tasks onto YARN tasks, showing how efficient the allocation can be.

Putting it Together: MapReduce and YARN

• Visualization of how MapReduce tasks operate within a YARN container environment on the worker nodes.

Scheduling in YARN

• Describes the Resource Manager's role in tracking resources in the cluster, including the scheduler process responsible for managing allocations.