Introduction to Hadoop

• Hadoop is a tool to store and process huge amounts of data quickly.
• This is done using a distributed computing model that allows scaling by adding computing nodes.

Hadoop vs. Relational Databases

• Hadoop: supports all data types (structured, semi-structured, unstructured), manages large amounts of data (terabytes/petabytes), uses HQL, has a dynamic schema, and is free, using Key-Value Pairs
• Relational Databases: supports structured data only, manages small to medium data (few GBs), uses SQL, has a static schema, incurs license costs, and uses Relational Tables

Hadoop Architecture

• The Hadoop Architecture consists of four components:
• MapReduce
• HDFS (Hadoop Distributed File System)
• YARN (Yet Another Resource Negotiator)
• Common Utilities

MapReduce

• MapReduce is an algorithm or data structure based on the YARN framework.
• MapReduce performs distributed processing in parallel in a Hadoop cluster.
• This parallel processing makes Hadoop function rapidly.
• Main tasks: Map, Reduce

Map and Reduce Stages

• Map Stage: processes the input data.
• Input data (file or directory) is stored in HDFS
• The mapper processes the data and creates small chunks.
• Reduce Stage: combination of the Shuffle and Reduce stages
• processes data that comes from the mapper.
• After processing, produces a new set of output, which is stored in HDFS.

Big Data Problem

• Input/Output Bound: the time to complete a computation is determined by the time spent waiting for input/output operations.

Distributed Architecture Challenges

• Nodes failing with 1 in 1000 failing per day
• Networks can be a bottleneck with 1-10 Gb/s throughput
• Traditional distributed programming can be ad-hoc and complicated

Distributed File System (DFS)

• Distributed File System (DFS): classical model of a file across multiple machines
• Purpose: promote sharing of the system distributed across dispersed files.
• Files split into contiguous chunks, commonly 16-64MB, with each chunk replicated 2x or 3x
• Replicas kept in different racks.

High-Level Computation

• Challenges for IO Cluster Computing addressed by:
• Duplicate Data → Distributed File System
• Bring computation to nodes, rather than data to nodes → SORT and SHUFFLE
• Stipulate a programming system that can easily be distributed → Map Reduce

HDFS

• Hadoop Distributed File System (HDFS) is used for storage.
• NameNode (Master node)
• DataNode (Slave node)

NameNode

• NameNode (Master node): manages all services and operations.
• One Master node is sufficient, but a secondary increases scalability and high availability.
• It coordinates Hadoop storage operations
• Name node: part of Master node that coordinates HDFS functions.
• When a file block's location is requested, the Master node obtains it from the Namenode process.

DataNode

• DataNode (Slave/worker node): stores data in a Hadoop cluster, providing CPU, memory, and local disk. Data node: a process handling actual reading and writing of data blocks to storage.
• There are many other components like Job Tracker and Task Tracker.

HDFS Features

• Easy access to stored files
• High availability and fault tolerance
• Scalability to scale-up or scale-down nodes
• Data stored in a distributed manner, with various Datanodes responsible.
• Data replication to prevent data loss
• High reliability, allowing storage of petabytes of data
• Name node and Data Node for easy retrieval of cluster information
• High throughput

Hadoop Overview

• Apache Hadoop software library: a framework for distributed processing of large datasets across computer clusters using a simple programming model.
• Hadoop provides fast and reliable analysis of structured and unstructured data
• Hadoop scales up from single servers to thousands of machines, each offering local computation and storage.

Hadoop Benefits

• Scalability: supports thousands of compute nodes and petabytes of data
• Cost-Effective: runs on low-cost commodity hardware.
• Efficient: distributes data and processes it in parallel on the nodes where the data is located.

HDFS Architecture

• HDFS is a distributed file system providing high-throughput access to application data.
• It follows a master/slave architecture:
• NameNode (master) controls DataNodes (slaves).
• It breaks data/files into small blocks (128 MB each), storing them on DataNode, which replicates blocks on other nodes for fault tolerance.
• NameNode tracks blocks written to DataNode.