Hadoop Main Components and Functions

Hadoop is designed for small-scale data processing.

HDFS stands for Hadoop Distributed File System and provides reliable storage.

The master node in Hadoop manages the file system namespace and block information.

Hadoop follows a distributed computing environment with a master-slave architecture.

Each file in Hadoop is stored as a single block on the cluster.

The NameNode in Hadoop is responsible for maintaining and managing the file system namespace.

NameNode manages and maintains the DataNodes in Hadoop HDFS.

DataNode is responsible for serving the client read/write requests in Hadoop HDFS.

MapReduce works by breaking the data processing into three phases: Map phase, Shuffle phase, and Reduce phase.

Yarn provides resource management for Hadoop through the ResourceManager running on the slave machines only.

Hadoop includes additional modules such as Hive, which is a relational, distributed database.

Hadoop is commonly used in big data scenarios such as data warehousing, business intelligence, and machine learning.

The Edit log in Hadoop contains all the recent changes performed to the file system namespace up to the most recent Fsimage.

DataNodes send block reports to NameNode to report the health of HDFS.

NameNode takes care of the replication factor of all the blocks in Hadoop HDFS.

The Reduce phase in MapReduce is where light-weight processing like aggregation/summation is specified.