Distributed Systems and Hadoop Quiz

Apache Hadoop Overview

• Hadoop is a software framework for storing, processing, and analyzing "big data".
• It's a distributed, scalable, and fault-tolerant system.
• It's open-source.

Hadoop Components

• Hadoop consists of two core components:
  • Hadoop Distributed File System (HDFS) which stores data on the cluster.
  • MapReduce which processes data on the cluster.
• There are many other projects built around core Hadoop, often referred to as the Hadoop Ecosystem (e.g., Pig, Hive, HBase, Flume, Oozie, Sqoop, etc).
• A set of machines running HDFS and MapReduce is known as a Hadoop Cluster. Individual machines are known as nodes. More nodes generally mean better performance.

Hadoop History

• Hadoop is based on work done by Google in the late 1990s/early 2000s (Google File System (GFS) and MapReduce).
• This work presents a radical new approach to distributed computing.
• This approach meets requirements for reliability and scalability in the system.

Core Hadoop Concepts

• Applications are written in high-level code eliminating the need for programmers to worry about network programming, temporal dependencies, or low-level infrastructure.
• Nodes talk to each other as little as possible.
• Data is distributed in advance.
• Computation occurs where the data is stored whenever possible.
• Data is replicated multiple times for increased availability and reliability.
• Hadoop is scalable and fault-tolerant.

Hadoop: Very High-Level Overview

• When data is loaded into the system, it is split into "blocks" (typically 64MB or 128MB).
• Map tasks (part of MapReduce) process relatively small portions of the data.
• Typically, a single block is processed by a map task.
• A master program assigns work to nodes such that a Map task will work on a locally stored block of data whenever possible.
• Many nodes work in parallel to process the entire dataset.

Fault Tolerance

• If a node fails, the master detects the failure and reassigns the work to a different node.
• Restarting a task does not require communication with other nodes.
• When a failed node restarts, it is automatically added back to the system and assigned new tasks.
• If a node appears to be running slowly, the master may redundantly execute another instance of the same task (known as speculative execution).

Data Recoverability

• If a component of a Hadoop system fails, the workload is assumed by still-functioning units in the system.
• This prevents data loss.

Data Storage in Hadoop

• HDFS is the Hadoop Distributed File System which is responsible for storing data on the cluster.
• Data is split into blocks and distributed across multiple nodes.
• Data blocks are typically 64MB or 128MB in size and replicated multiple times (default is 3 times).
• This setup ensures high availability and reliability.
• When a client wants to read a file, it communicates with the NameNode to locate the necessary blocks and then directly communicates with the DataNodes to read the data.

HDFS NameNode Availability

• The NameNode daemon must run at all times.
• If the NameNode stops, the cluster is inaccessible.
• High availability mode has two NameNodes (one active, one standby).

Hadoop: Basic Concepts

• What is Hadoop?
• What features does the Hadoop Distributed File System (HDFS) provide?
• What are the concepts behind MapReduce?
• How does a Hadoop cluster operate?

Hadoop Components (cont'd)

• Hadoop consists of two core components: HDFS and MapReduce. Many other projects build on top of the Hadoop ecosystem (e.g., Pig, Hive, HBase, Flume, Oozie, Sqoop).

Hadoop Components: MapReduce

• MapReduce is a system used to process data in the Hadoop cluster.
• It consists of two phases:
  • Map: Each map task operates on a discrete portion of the dataset. The initial portion of the dataset is typically a single block.
  • Reduce: After all map tasks are complete, the MapReduce system distributes intermediate data to reducers. The reducers perform the final calculation and writing to disk.

Hadoop Environments

• Cloudera's Quickstart VM offers a preconfigured environment for developing Hadoop solutions.
• When ready for production, solutions can be run on a Hadoop cluster managed by a system administrator.

The Hadoop Ecosystem (cont'd)

• Various components exist around core Hadoop.
• Components are characterized by their use case: data processing, data analysis, machine learning, etc.

HBase, Flume, Sqoop

• HBase is the Hadoop database, a NoSQL datastore.
• Flume is a service for moving large amounts of data into HDFS as it is generated (for example, log files from a webserver).
• Sqoop is used to transfer data between RDBMS (e.g., MySQL, PostgreSQL, Teradata, Oracle, etc) and Hadoop.

Hive, Pig, and Impala

• Hive: SQL-like interface to Hadoop.
• Pig: Dataflow language for transforming large datasets.
• Impala: High-performance SQL engine for querying vast amounts of data for Hadoop storage.

Oozie

• Oozie is a workflow engine for scheduling and managing MapReduce jobs on Hadoop.

Mahout

• Mahout is a machine learning library written in Java.

Common Types of Analysis with Hadoop

• Text mining, collaborative filtering, index building, prediction models, graph creation & analysis, sentiment analysis, pattern recognition, risk assessment.