RDD Actions and Transformations Quiz

Spark Basic Concepts

• Spark is a unified analytics engine for large-scale data processing.
• It provides a resilient distributed dataset (RDD) abstraction.

Resilient Distributed Datasets (RDDs)

• RDDs are the primary abstraction in Spark.
• RDDs are distributed collections of objects spread across the nodes of a cluster.
• RDDs are split into partitions.
• Each node in the cluster running an application contains at least one partition of the RDD(s) defined in the application.
• RDDs are stored in the main memory of the executors running in the cluster. If not possible, they are stored in the local disk of the nodes.
• RDDs allow executing code in parallel.
• Each executor of a worker node runs specified code on its partition of the RDD.
• RDDs are immutable once constructed.
• Spark tracks lineage information to efficiently recompute lost data (due to executor failures).
• This information is represented as a Directed Acyclic Graph (DAG) connecting input data and RDDs.
• RDDs can be created from collections in Scala, Java, Python, or R.
• RDDS can be created from files stored in HDFS, other systems, or databases.
• Number of partitions depends on type of transformations or user specification.
• Spark programs operate on RDDs; this includes transformations (creating a new RDD) and actions (obtain results).

Spark Programs

• Spark programs are written using operations on resilient distributed data sets.
• Transformations : map, filter, join
• Actions : count, collect, save

Spark Framework

• Manages scheduling and synchronization.
• Splits RDDs into partitions and allocates them among cluster nodes.
• Hides complexities of fault-tolerance and slow machines.
• RDDs are automatically rebuilt in case of machine failure.

Spark Official Terminology

• Application: User program built in Spark with a driver program and executors.
• Driver Program: The process running the main() function that creates the SparkContext.
• Cluster Manager: The external service managing cluster resources (e.g., standalone manager, Mesos, YARN).
• Deploy Mode: Defines where the driver process operates (inside or outside the cluster).
• Worker Node: Any cluster node that can run application code.
• Executor: A process running tasks.
• Task: A unit of work sent to an executor.
• Job: A parallel computation composed of tasks.
• Stage: Each job is divided into sets of tasks.
• Shuffle: A heavy operation involving data grouping/repartitioning.

Spark Programs: Examples (Count Line)

• Counts lines in an input file ("myfile.txt").
• Prints the count to standard output.
• Shows examples of PySpark code.
• Basic operations on RDD(from file).

Spark Program: Word Count

• Implements word count by using Spark operations.
• Takes input filename and output folder as command-line arguments.

RDD-based Programming

• Explains RDD-based programming concepts, which are at the core of how Spark works.
• Focuses on Spark context, and the key details about programming.

SparkContext

• A connection between the driver and the cluster.
• Built using the SparkContext class constructor (in Python).
• Allows creating RDDs and invoking operations on them.

RDD Basics

• An RDD is an immutable distributed collection of objects.
• Each RDD is split into partitions.
• Code runs on individual partitions in isolation.
• RDDs support various data types (Scala, Java, Python and user-defined types), not limited to simple types.

RDD: Create and Save

• RDDs can be created from external datasets or by parallellizing in-memory objects.

Create RDDs from Files

• Creates RDDs from textual files.
• Explains the textFile() method of SparkContext.
• Discuses the importance of data locality.
• Provides examples of reading from files and folders.

Create RDDs from a Local Python Collection

• Describes parallelize() to create RDDs from Python lists, ensuring data distribution.

Save RDDs

• Details for saving the contents of distributed datasets (an RDD) to file system or any other available storage (e.g., HDFS) or database.
• Uses the saveAsTextFile() method.

Retrieve the content of RDDs and "store" it in local python variables

• Describes retrieving contents into local variables.
• Explains the collect() method.
• Notes on potential issues with very large RDDs.

RDD Operations

• Describes transformation operations (new RDD), and actions (results).
• Explains how these operations work in the context of RDD immutability.
• Highlights the concept of lineage graph and its use for optimization.

Actions

• Describes actions returning results to the driver.
• Emphasizes the importance of handling the size of returned data.

Example of lineage graph (DAG)

• Illustrates how Spark computes the content of an RDD only when needed.

Passing Function to Transformations and Actions

• Details on the use of lambda functions and user-defined functions to apply transformations and/or actions on RDDs.

Basic Transformations

• Summarizes fundamental transformations on a single RDD.

Filter Transformation

• Describes the filter() transformation to create new RDDs based on predicates that return True or False.
• Provides examples illustrating the usage of this operation using lambda functions and user-defined functions.

Map Transformation

• Describes the map() transformation applied on each element from the input RDD to return a new element.
• Explains that the input and output of map() can be of a different type.
• Presents examples.

FlatMap Transformation

• Describes the flatMap() transformation.
• Explains how to use it and its differences compared to the map() operator.
• Provides examples.

Distinct Transformation

• Explains the distinct() transformation that returns a new RDD containing the unique elements from the input RDD.
• Emphasizes the shuffle operation.
• Presents examples.

SortBy Transformation

• Describes the sortBy() transformation for sorting an RDD.
• Shows the use of a custom sorting key function.
• Provides examples to sort RDDs based on ascending or descending order, including examples with custom sorting keys.

Sample Transformation

• Describes the sample() transformation.
• Explains whether or not the sampling is with or without replacement and the meaning of the fraction parameter.
• Provides examples.

Set Transformations

• Describes operations (union, intersection, subtract, cartesian) that are applied on two RDDs.

Basic Actions

• Describes actions, their purpose, and examples, highlighting efficient ways to obtain results from transformations.
• The list of available actions.

Collect Action

• Goal of retrieving all RDD elements into the driver.
• Method used in Spark for retrieving RDD contents.
• Example of how to use collect().
• Important consideration on the size of RDD when collect() is used.
• Alternative actions for large RDDs.

Count Action

• Goal of this action.
• Method used by Spark.
• Example of when to use it in a program.

CountByValue Action

• Goal of retrieving the frequency of each RDD element.
• Method that returns a dictionary mapping elements to their frequency.
• Example of how to use it.

Take Action

• Goal of retrieving the first n elements of an RDD.
• Shows how to use take().

First Action

• Goal of retrieving the first element of an RDD.
• Shows how to use first().

Top Action

• Goal of retrieving the top n largest elements of an RDD.
• Methods and examples.

TakeOrdered Action

• Goal of retrieving the top n elements from an RDD in a specific order.

TakeSample Action

• Goal of retrieving a sample of an RDD, either with or without replacement.
• Methods and examples.

Reduce Action

• Goal of combining all elements in an RDD using a custom function.
• The reduce() method and its requirements for associative and commutative functions.
• Example of usage.

Fold Action

• Goal of combining all elements in an RDD with an initial value.
• The fold() method, handling the initial value.
• The example in this case demonstrates an RDD containing strings. Shows how to use fold()
• Explanation of the difference between reduce() and fold().

Aggregate Action

• Goal combining elements of an RDD using a custom function and initial value.
• Unlike reduce() and fold(), aggregate can handle cases where the input and output data type differ and work on partitions in parallel.
• Detailed explanation of the use cases for this operation.

Test your understanding of key actions and transformations in Resilient Distributed Datasets (RDDs) within Apache Spark. This quiz covers various functions and their outputs, helping you grasp how RDD operations work in practice. Perfect for those studying Spark or data processing concepts.