HDFS: Hadoop Distributed File System

• HDFS stands for Hadoop Distributed File System and is used for storage.
• NameNode is the master node in HDFS.
• DataNode is the slave node in HDFS.

HDFS Features

• Files stored in HDFS are easy to access
• HDFS provides high availability and fault tolerance
• Nodes can be scaled up or down as per requirements
• Data is stored in a distributed fashion, with various DataNodes responsible for storing the data
• HDFS has replication to prevent data loss
• HDFS provides high reliability and can store data in a petabyte range
• NameNode and DataNode servers are built-in
• NameNode and DataNode facilitate easy retrieval of cluster information
• HDFS provides high throughput

Hadoop Architecture

• Hadoop = HDFS + MapReduce
• Hadoop is similar to the kernel of an operating system, also known as Hadoop Core
• HBase, Hive, Pig, Oozie, Flume, and Sqoop are components often deployed with Hadoop
• These components form a "Hadoop Stack"
• Not all components must be deployed

HDFS Characteristics

• HDFS is a distributed file system providing high-throughput access to application data
• HDFS uses a master/slave architecture
• In the master/slave setup a NameNode (master) controls one or more DataNodes (slaves).
• Data/Files are broken into 128 MB blocks and stored on DataNode.
• Each block is replicated on other nodes for fault tolerance
• The NameNode keeps track of blocks written to the DataNode

Job Scheduling

• Job Scheduling includes JOB TRACKER & TASK TRACKER
• Distributed Data Processing includes Map Reduce
• Distributed Data Storage consists of HDFS

JobTracker

• JobTracker is a daemon service for submitting and tracking MapReduce jobs in Hadoop.
• JobTracker accepts MapReduce jobs from client applications.
• JobTracker communicates with NameNode to determine data location.
• JobTracker locates available TaskTracker Nodes.
• JobTracker submits work to the chosen TaskTracker Node

TaskTracker

• TaskTracker node accepts map, reduce, or shuffle operations from a JobTracker
• TaskTracker is configured with a set of slots that indicate the number of tasks it can accept
• JobTracker seeks the free slot to assign a job
• TaskTracker notifies the JobTracker about job success status
• TaskTracker sends heartbeat signals to the job tracker to ensure its availability
• TaskTracker reports the number of available free slots

Data Replication

• Data Replication is needed because HDFS is designed to handle large-scale data in a distributed environment
• Data Replication is needed to mitigate hardware or software failure, or network partition
• Replication is required for fault tolerance

Handling Data Node Failure

• If a data node fails, the NameNode identifies the blocks the Data Node contained, creates the same replications to other alive nodes, and unregisters the dead node.

Data Integrity

• Corruption may occur in network transfer or hardware failure
• Checksum checking is applied on the content of files on HDFS.
• Checksums are stored in the HDFS namespace
• If the checksum is incorrect after fetching, the block is dropped, and another replication is fetched from other machines

HDFS vs RDBMS

• RDBMS is used for storing structured data while HDFS can store both structured and unstructured data
• RDBMS can effectively handle a few thousand records, while HDFS can handle millions and billions of records.
• RDBMS is best suited for transaction management while HDFS is not recommended for transaction management
• Processing time depends on the server machine's configuration in RDBMS while processing time depends on the number of cluster machines in HDFS
• Consistency is preferred over availability in RDBMS while availability is preferred over consistency in HDFS