Hadoop and MapReduce Concepts Quiz

The process of transforming and mapping data from one 'raw' data form into another format with the intent of making it more appropriate and valuable for various downstream purposes like analytics.

A task is divided into smaller sub-tasks that run concurrently, aiming to reduce execution time. This happens within a single machine with multiple processors.

Similar to parallel processing, but tasks are distributed across physically separate machines connected in a cluster, enabling processing large datasets.

The amount and nature of data processed within a specific timeframe, defining the load and type of processing.

Collecting, processing, manipulating, and managing data from various sources to generate meaningful information for the end user.

Refers to a centralized approach to data processing where all operations occur within a single computer system.

Involves distributing data processing across multiple machines, using a network to connect them. This allows for handling larger datasets and enhancing processing speed.

A processing approach where data is processed in batches and often involves delays. Think of it like a big batch of laundry - you wait for everything to be collected before you start washing.

A workload type characterized by large data volumes, sequential read/writes, and the execution of complex queries. It's like analyzing a whole company's sales history.

A processing approach where data is processed interactively and without delay, leading to quick responses. Imagine buying a product online - you get instant feedback.

A workload type involving small amounts of data, random reads and writes, and the execution of simple queries. Think of making a purchase online - small, individual actions.

A type of system designed for transactional workloads. It's like an online store where each customer's purchase is processed immediately.

A type of system designed for batch workloads. It's like analyzing a company's annual sales data to gain insights.

A method for efficiently handling large datasets. It involves breaking down the data into smaller parts, Processing each part independently and then combining the results. Think of dividing a big puzzle into smaller pieces.

A processing run within the MapReduce framework. Think of it as one complete cycle of dividing, processing, and combining data.

The process of dividing a file into key-value pairs, where the key is the offset and the value is the line of data.

The logic that a programmer writes to process the value of each line, typically involving counting occurrences of specific items.

The process of grouping values together based on their corresponding keys. This happens after the mapping phase and before the reduction phase.

The process of combining the values for each key, typically by summing them up. This happens after shuffling.

A method of parallelization where a task is broken into smaller sub-tasks, and each sub-task is run on a separate processor, usually in a cluster. Sub-tasks usually execute different algorithms.

A method of parallelization where a dataset is split into multiple smaller datasets, and each sub-dataset is processed in parallel, usually on different nodes in a cluster. All sub-datasets are processed using the same algorithm.

A general approach to managing large datasets using a collection of computers working together. The data is distributed across nodes, and operations are performed in parallel on different parts of the data.

A type of data processing that handles data as it arrives, allowing for immediate analysis and actions. For example, fraud detection systems.

A data processing method where data is stored and manipulated in the computer's main memory, resulting in significantly faster processing. This is suitable for smaller datasets.

The ability to expand or shrink the size of a system (like a Hadoop cluster) based on workload demands.

Adding more resources to existing nodes in a cluster, for example, adding more disks to a Hadoop node.

Adding more nodes to a cluster on the fly, without downtime, to increase the cluster's processing power. For example, adding more servers to a Hadoop cluster.

The ability of a system to remain functional even if one or more components fail, ensuring continuous operation and data availability. This is achieved in Hadoop HDFS through redundant data replication.

A single point of failure refers to a component in a system whose failure can lead to the entire system's failure. In Hadoop, the NameNode was a single point of failure in older versions.

When a Hadoop mapper is run on a node in a different rack than the data it needs to process due to resource constraints.

A scenario in Hadoop where data is processed on a node in a different rack than the node where the data resides, leading to slower performance due to increased network latency.

Financial institutions utilize Hadoop to analyze vast amounts of data to assess risk, create investment models, and develop complex trading algorithms.

Retailers leverage Hadoop to analyze various types of customer data to understand customer behavior, preferences, and trends to improve marketing and sales strategies.

The energy industry uses Hadoop to analyze sensor data from Internet of Things (IoT) devices for predictive maintenance, optimizing equipment performance and reducing downtime.

The NameNode is a critical component in a Hadoop cluster responsible for managing data blocks and metadata. To ensure high availability, Hadoop clusters run two NameNodes in an active/passive configuration, with one active and the other in standby. If the active NameNode fails, the standby node automatically takes over, ensuring uninterrupted file system operations.

Fault tolerance in Hadoop refers to the system's ability to continue working even if some components fail. It's achieved through data replication, where data is copied onto multiple data nodes within the cluster. This redundancy ensures data availability even if a node goes down.

Data Locality in Hadoop is a technique that minimizes network traffic by bringing the processing power closer to the data. Instead of moving massive data to the processing nodes (like mappers and reducers), the data processing jobs are scheduled on the same nodes where the data resides.

Data Local Data Locality refers to the optimal scenario where the mapper processing the data is located on the same node as the data itself. This minimizes network overhead and maximizes the efficiency of data processing.

Intra-Rack Data Locality occurs when the mapper processing the data is on a different node than the data but within the same physical rack. This is a less optimal scenario than Data Local, but it's still better than having the data and the mapper on different racks.

Inter-Rack Data Locality is a data locality strategy in Hadoop where the mapper processing the data is located on a different node than the data and also on a different physical rack. This results in the highest network overhead and is generally least preferable.

Data Locality is crucial for efficient Hadoop operations. It minimizes network traffic, improves processing speeds, and maximizes cluster utilization by scheduling jobs on the nodes where the data is stored.

The Hadoop ecosystem provides various techniques to optimize cluster performance, including proper cluster configuration, effective data compression using LZO, and optimized data types for efficient storage and retrieval.