(Spark)[Medium] Chapter 15: How Spark Runs on a Cluster

Podcast

Play an AI-generated podcast conversation about this lesson

Questions and Answers

In cluster mode, which process is launched by the cluster manager on a worker node?

Spark Application process

Client process

Driver process (correct)

Executor process

In client mode, which machine is responsible for maintaining the Spark driver process?

Worker node

Gateway machine

Client machine (correct)

Cluster manager

What is the primary difference between cluster mode and client mode?

Type of Spark Application

Location of the executor processes

Number of worker nodes

Location of the driver process (correct)

What is the primary characteristic of local mode?

Achieves parallelism through threads on a single machine Signup and view all the answers

What is the purpose of local mode?

To test and experiment with Spark Applications Signup and view all the answers

What is the primary responsibility of the cluster manager during Spark Application execution?

To manage the underlying machines that the application is running on Signup and view all the answers

What determines the physical location of resources when running a Spark Application?

The execution mode selected Signup and view all the answers

How many execution modes are available when running a Spark Application?

3 Signup and view all the answers

What is the most common way of running Spark Applications?

Cluster mode Signup and view all the answers

What is the primary process responsible for managing the cluster during a Spark application deployment?

Cluster manager driver Signup and view all the answers

What is the first step in the Spark application life cycle?

Submitting the Spark application Signup and view all the answers

What type of communication is represented by dashed lines in the illustrations?

Cluster management communication Signup and view all the answers

Why is local mode not recommended for production applications?

The reason is not specified in the text Signup and view all the answers

What is the purpose of the spark-submit tool?

To submit a Spark application to the cluster manager Signup and view all the answers

What happens after a shuffle operation in Spark?

A new stage is started Signup and view all the answers

What can be accessed on the SQL tab in the Spark UI after running a query?

The physical plan of the query Signup and view all the answers

What determines the number of tasks in each stage in Spark?

Number of partitions in the DataFrame Signup and view all the answers

What is the purpose of repartitioning in Spark?

To change the number of partitions Signup and view all the answers

What happens when you call collect or any action in Spark?

The execution of a Spark job that individually consists of stages and tasks Signup and view all the answers

When does Spark start a new stage?

After every shuffle operation Signup and view all the answers

What is the relationship between Spark jobs and actions?

There is one Spark job for one action Signup and view all the answers

What determines the number of stages in a Spark job?

The number of shuffle operations needed Signup and view all the answers

What happens by default when creating a DataFrame with range in Spark?

It has 8 partitions Signup and view all the answers

What is the primary role of the driver in a Spark cluster?

To schedule tasks onto each worker Signup and view all the answers

What happens to the executors in a Spark cluster after a Spark Application completes?

They are shut down by the cluster manager Signup and view all the answers

How are Spark jobs within an application executed?

Serially, one after another Signup and view all the answers

What is the first step in creating a Spark Application?

Creating a SparkSession Signup and view all the answers

What is the outcome of a Spark Application after completion?

The driver process exits with either success or failure Signup and view all the answers

What is the primary responsibility of Spark executors?

To run tasks assigned by the Spark driver and report back the state and results Signup and view all the answers

What is the main difference between the Spark driver and the cluster manager?

The Spark driver is responsible for assigning tasks, while the cluster manager manages the cluster Signup and view all the answers

What is the role of the cluster manager in a Spark Application?

To manage the cluster of machines Signup and view all the answers

How many execution modes are available in Spark?

3 Signup and view all the answers

In which mode is the Spark driver process launched by the cluster manager on a worker node?

Cluster mode Signup and view all the answers

What is the role of the Spark driver in a Spark Application?

To assign tasks to executors and manage the application Signup and view all the answers

What is the primary difference between client mode and cluster mode?

The location of the Spark driver process Signup and view all the answers

What is the purpose of local mode in Spark?

To run a Spark Application on a single machine Signup and view all the answers

What is responsible for maintaining the state of a Spark Application?

The Spark driver Signup and view all the answers

How many separate executor processes are there in a Spark Application?

Multiple Signup and view all the answers

Study Notes

Spark Application Modes

Cluster mode: submits a pre-compiled JAR, Python script, or R script to a cluster manager
- Cluster manager launches the driver process on a worker node and executor processes
- Cluster manager is responsible for maintaining all Spark Application-related processes
Client mode: similar to cluster mode, but the Spark driver remains on the client machine
- Client machine is responsible for maintaining the Spark driver process
- Cluster manager maintains the executor processes
Local mode: runs the entire Spark Application on a single machine
- Achieves parallelism through threads on that single machine
- Common way to learn Spark, test applications, or experiment iteratively with local development

Cluster Managers

Three cluster managers supported by Spark: standalone, Apache Mesos, and Hadoop YARN
Cluster manager responsible for managing underlying machines that a Spark Application is running on

Execution Modes

Three execution modes: cluster mode, client mode, and local mode
Execution mode determines where resources are physically located when running an application

Spark Application Life Cycle

Spark Application breaks down into stages and tasks
Stages: groups of tasks that can be executed together to compute the same operation on multiple machines
Tasks: individual units of work that are executed on executors
Spark starts a new stage after each shuffle operation (e.g., sorting a DataFrame, grouping data)

Spark Job

A Spark job consists of one or more stages and tasks
Each job breaks down into a series of stages, with the number of stages depending on the number of shuffle operations
Actions always return results

SparkSession

Creating a SparkSession is the first step in any Spark Application
SparkSession is a process on a physical machine responsible for maintaining the state of the application running on the cluster

Studying That Suits You

Use AI to generate personalized quizzes and flashcards to suit your learning preferences.

Description

Learn about the different modes of Spark cluster management, including cluster mode and client mode. Understand how the cluster manager launches driver and executor processes in a Spark application.