Cloud Computing - Lesson 10: Map/Reduce

<p>Web search and indexing (B)</p> Signup and view all the answers

<p>The exponential growth of data being produced. (C)</p> Signup and view all the answers

<p>By using user-generated data for recommendations. (C)</p> Signup and view all the answers

<p>1 sextillion bytes (B)</p> Signup and view all the answers

<p>To increase efficiencies through data analysis. (C)</p> Signup and view all the answers

<p>It does not consider the overall frequencies from all mappers. (B)</p> Signup and view all the answers

<p>A pair for each link to the target URL. (B)</p> Signup and view all the answers

<p>Sorts the document IDs associated with a keyword. (C)</p> Signup and view all the answers

<p>To group items into k clusters. (D)</p> Signup and view all the answers

<p>Source URLs pointing to target URLs. (D)</p> Signup and view all the answers

<p>Local reducers may overlook lower-frequency items. (A)</p> Signup and view all the answers

<p>A sorted list of all document IDs for a given keyword. (C)</p> Signup and view all the answers

<p>Clusters must minimize the distance between contained points. (D)</p> Signup and view all the answers

<p>Handling faults and slow machines (C)</p> Signup and view all the answers

<p>Map/Reduce framework (A)</p> Signup and view all the answers

<p>Data curation and data authenticity (C)</p> Signup and view all the answers

<p>JSON documents (C)</p> Signup and view all the answers

<p>To find the average generation time for different page types (D)</p> Signup and view all the answers

<p>The speed of data generation and processing (C)</p> Signup and view all the answers

<p>Home, product, cart, checkout (D)</p> Signup and view all the answers

<p>Understanding volume is crucial for handling large data sets. (C)</p> Signup and view all the answers

<p>To manage unprecedented scale of data processing (B)</p> Signup and view all the answers

<p>Simple programming model (C)</p> Signup and view all the answers

<p>Map function (A)</p> Signup and view all the answers

<p>Large data set processing and generation (C)</p> Signup and view all the answers

<p>Choose centroids randomly (C)</p> Signup and view all the answers

<p>Complexities in parallel computation and data distribution (B)</p> Signup and view all the answers

<p>Webpage PageRank computation (B)</p> Signup and view all the answers

<p>By calculating the distance to each centroid (D)</p> Signup and view all the answers

<p>New centroids are computed based on assigned points (D)</p> Signup and view all the answers

<p>Parallel processing model (A)</p> Signup and view all the answers

<p>Distributed computation complexity (D)</p> Signup and view all the answers

<p>Centroids have converged and no change occurs (A)</p> Signup and view all the answers

<p>It assumes no global shared state, which k-Means requires (C)</p> Signup and view all the answers

<p>Saves the global centroids and checks for changes (A)</p> Signup and view all the answers

<p>They are selected from the dataset randomly (D)</p> Signup and view all the answers

<p>To group the points by their closest centroid (B)</p> Signup and view all the answers

<p>Fork the user program (A)</p> Signup and view all the answers

<p>Split the input into smaller segments (B)</p> Signup and view all the answers

<p>Reduce phase (D)</p> Signup and view all the answers

<p>Perform local writes of intermediate files (D)</p> Signup and view all the answers

<p>They inform the master (D)</p> Signup and view all the answers

<p>Transformed raw input data (D)</p> Signup and view all the answers

<p>Distribute tasks and manage workers (D)</p> Signup and view all the answers

<p>Final output files from computations (A)</p> Signup and view all the answers

The ability of cloud computing to support applications processing vast amounts of data.

The continuous increase in data volume generated by users and applications in cloud environments.

A framework used to process massive datasets by dividing them into smaller tasks (map) and combining the results (reduce).

The four key characteristics of big data: volume, velocity, variety, and veracity.

The massive amount of data generated globally, measured in zettabytes (ZB).

Applications that utilize vast amounts of publicly available data, like web search engines and large language models.

Applications that rely on data generated by users, including social media, recommendation engines, and ride-hailing services.

Applications that leverage both public and user-generated data, such as web search engines that personalize results based on user preferences.

A method of grouping items into clusters (k) based on the minimum distance between points within each cluster. The goal is to minimize the distance within each cluster.

A web graph that is reversed, showing all links that point to a specific page. This is the foundation for the PageRank algorithm, Google's original web ranking system.

A process of finding all documents containing a particular keyword. This is used by search engines like Google, Yahoo!, and others.

An approach to k-Means computation that involves first calculating a global top-k, which is the overall most frequently occurring items across all data.

In the context of k-Means computation, it refers to the calculation of the top k most frequent items within each mapper. These top-k items are then processed by the reducer to form the final clusters.

A computation where the top k most frequent items are determined globally across all data. This global top-k can be used to guide cluster formation in k-Means computation.

The four characteristics associated with big data are Volume, Variety, Velocity, and Veracity.

Large datasets are characterized by their sheer size, which makes them difficult to process using traditional methods.

Big data is often heterogeneous, containing different data formats like text, images, video, and more.

The rapid rate at which data is generated and needs to be analyzed is known as Velocity.

The accuracy and trustworthiness of data are central to Big Data, ensuring the data is reliable and can be used for informed decisions.

MapReduce is a programming model designed to process large static datasets by dividing the data into smaller pieces for parallel processing.

Stream processing frameworks are designed to handle dynamic data that continuously changes, often used for real-time analysis.

Logging is the process of recording events and activities within a system, typically in text files, for analysis and troubleshooting.

A method for dividing a dataset into smaller tasks (map) and combining the results (reduce).

The process of classifying each data point to its nearest centroid.

The process of recalculating the centroid based on the data points assigned to it.

The step where each point is assigned to the closest centroid, based on distance.

The step where the centroids are recalculated based on the points assigned to them.

A method to implement k-Means in MapReduce that uses a global shared file to store centroids between iterations.

The process of repeatedly applying the 'classify' and 'recenter' phases until the centroids stop changing.

The process of comparing the new centroids to the previous ones to see if they have converged.

A central component in the MapReduce framework, responsible for coordinating the execution of map and reduce tasks, assigning work to workers, and monitoring the progress of the job.

In the MapReduce framework, these are nodes that carry out individual map or reduce tasks.

The initial stage of a MapReduce job, where data is processed in parallel by multiple workers. Each worker receives a split of the input data and applies a map function to it, generating key-value pairs.

The intermediate data generated by the map phase, stored on local disks of the workers for the reduce phase.

The second stage of a MapReduce job after the map phase. Workers receive the intermediate key-value pairs from the map stage, group them based on the keys, and apply the reduce function to generate the final output.

Input files are split into smaller chunks or splits, distributing work for parallel processing.

The master informs the workers about the location of fresh data, ensuring that data is processed efficiently in a distributed manner.

The Master coordinates sending the initial data splits to the workers for processing.

A function that processes individual data items, transforming them into key-value pairs (intermediate data).

A function that aggregates intermediate results with the same key, combining them into a final output.

A program designed to calculate the PageRank of web pages, which measures a page's importance based on incoming links.

The process of transforming large datasets into smaller units that can be processed in parallel by several computers.

The ability to process data independently on multiple machines, allowing for faster processing of massive datasets.

MapReduce's simplicity allows even individuals with limited programming experience to work with large datasets by focusing on the core logic of the data processing task.