MapReduce Framework Overview

MapReduce (MR)

• MapReduce is a programming model and a software framework for processing large datasets in parallel.
• It divides the processing into two main phases: map and reduce.
• MapReduce is designed to handle big data efficiently and is used for tasks like search, analytics, and machine learning.

Phases of MapReduce

• Split: Data is partitioned across multiple computer nodes.
• Map: A map function is applied to each chunk of data.
• Sort & Shuffle: The output of the mappers is sorted and distributed to the reducers.
• Reduce: A reduce function is applied to the data, producing an output.

Example of MapReduce

• The text gives an example of how MapReduce might work, but it does not provide details about the specific task or data being processed.

MapReduce Framework

• The framework handles the splitting, sorting, and shuffling phases.
• The user defines the map and reduce functions.
• The user can customize the splitting, sorting, and shuffling phases.

Map and Reduce Functions

• The same map and reduce functions are applied to all data chunks.
• The map and reduce computations are independent and can be carried out in parallel.

Data Processing

• The splitting phase is separate from the internal partitioning into blocks.
• The sorting and shuffling phase can be the most resource-intensive part of a MapReduce job.
• The map function takes unsorted data as input and emits key-value pairs.
• The sorting process groups data by key, making it easier for the reducers to work with.
• Reducers can start processing a group of data as soon as the group is complete.

Map Task

• More details about Map tasks are needed; the text does not provide enough information.

Reduce Task

• More details about Reduce tasks are needed; the text does not provide enough information.

MapReduce Daemons

• The text mentions two important daemons in Hadoop's MapReduce implementation.

JobTracker

• JobTracker is a daemon service that manages and tracks MapReduce jobs in Hadoop.
• It accepts jobs from client applications.
• It communicates with NameNode to determine data location.
• It allocates tasks to available TaskTracker nodes.

TaskTracker

• TaskTracker is a daemon service that executes individual map, reduce, or shuffle tasks.
• It has a set of slots that represent the number of tasks it can handle concurrently.
• JobTracker assigns tasks to TaskTracker nodes based on available slots.
• TaskTracker notifies JobTracker about the status of completed tasks.