HDFS: Hadoop Distributed File System

HDFS (Hadoop Distributed File System)

• HDFS is used for storage in Hadoop.
• Utilizes a master/slave architecture.
  • NameNode: Master node.
  • DataNode: Slave node.
• Breaks data/files into small blocks, each 128 MB.
• Stores blocks on DataNode.
• Replicates each block on other nodes for fault tolerance.
• Provides high-throughput access to application data.
• NameNode tracks blocks written to the DataNode.

NameNode (Master Node)

• Manages all services and operations.
• Running the NameNode process coordinates Hadoop storage operations.
• Part of the Master node and responsible for coordinating HDFS functions.
• Informs the location of a file block when requested.
• Having a secondary NameNode increases scalability and high availability.

DataNode (Slave Node, Worker Node)

• Stores data in a Hadoop cluster.
• Provides infrastructure such as CPU, memory, and local disk for storing and processing data.
• Runs the DataNode process.
• Handles actual reading and writing of data blocks from/to storage.

Features of HDFS

• Easy to access stored files.
• Provides high availability and fault tolerance.
• Offers scalability to scale-up or scale-down nodes based on requirements.
• Data is stored in a distributed manner; Datanodes are responsible for storing the data.
• Provides replication to prevent data loss.
• Offers high reliability and can store data in petabytes.
• Has in-built servers in NameNode and DataNode for easy retrieval of cluster information.
• Provides high throughput.

Hadoop Architecture

• Consists of, HDFS and MapReduce.
• Components can form a "Hadoop Stack".
• Not all components must be deployed.

JobTracker

• Daemon service for submitting and tracking MapReduce jobs in Hadoop.
• Accepts MapReduce jobs from client applications.
• Communicates with NameNode to determine data location.
• Locates available TaskTracker Node.
• Submits work to the chosen TaskTracker Node.

TaskTracker

• Accepts map, reduce, or shuffle operations from a JobTracker.
• Configured with a set of slots indicating the number of tasks it can accept.
• Notifies the JobTracker about job success status.
• Sends heartbeat signals to the JobTracker to ensure availability.
• Reports the number of available free slots.

Data Replication

• Needed because HDFS is designed to handle large-scale data in a distributed environment.
• Addresses hardware or software failures, or network partitions exist.
• Provides fault tolerance.

Data Node Failure

• If a data node fails, the NameNode identifies contained blocks.
• Replications are created on other alive nodes.
• The dead node is unregistered.

Data Integrity

• Corruption can occur in network transfer or due to hardware failure.
• Checksum checking is applied to file contents on HDFS and stored in the HDFS namespace.
• If the checksum is incorrect after fetching, that retrieval is dropped, and another replication is fetched from other machines.

HDFS vs RDBMS

• HDFS stores structured and unstructured data, while RDBMS stores structured data.
• HDFS handles millions and billions of records whereas RDBMS handles a few thousand records.
• HDFS is not advised or transaction management but RDBMS is best suited for transaction management.
• HDFS processing time depends on the number of cluster machines, while RDBMS processing time depends on the configuration of the server machine.
• HDFS availability is preferred over consistency while RDBMS consistency is preferred over availability.

File Blocks in Hadoop

• Data/Files are broken into small blocks(128 MB each block) and stored on DataNodes.