Spark RDD Concepts Quiz

Spark Structured API

• Spark Structured API provides a high-level interface to work with data.
• It offers DataFrames, DataSets, and SQL.
• These APIs handle data structure, making code easier to read and understand.
• Spark Structured API is more efficient compared to RDDs.
• RDDs are low-level objects that track lineage and act like low-level code in Scala or other languages.

Problems with Coding Spark Jobs with RDDs

• Using RDDs to code Spark jobs presents both understanding and efficiency challenges.
• RDD code can be challenging to read.
• Inefficient execution sometimes results from RDD methods.

Problem Example (Scala RDD)

• The code shown in the presentation demonstrates a problematic RDD implementation for achieving a calculation.
• A series of map and reduceByKey operations is used for data transformation.
• This example illustrates the lack of type safety and the verbose nature of RDD code.
• Sample code included (val dataRDD, .map, .reduceByKey, .collect)

Problem with RDDs

• RDD code is often difficult to understand.
• Execution of the code as is frequently leads to performance problems.

Another Problem Example (Scala RDD)

• A different example of RDD code is provided showing the potential issues.
• It utilizes various transformations (e.g., map, reduceByKey, filter).
• These transformations represent more complex data manipulation than the first example.

Spark SQL Module

• Spark SQL adds structure to data for easier analysis with structured APIs.
• The three primary APIs include DataFrames, DataSets, and SQL.

What is a Query Language?

• High-level instructions define typical data analytics in query languages.
• Translation process converts a single query into multiple procedural code lines.

Example: SQL or HiveQL Code

• SQL code (or HiveQL) shows improved readability and type checking compared to RDDs.
• The SELECT dept, avg(salary) query extracts the average salary in the 'IT' department.
• Benefits include better readability, type checking, and faster execution.

What's the Catch with Query Languages?

• Granularity in manipulating data is limited for query languages when compared to procedural programming.
• More specific manipulations require using procedural programming if granular operations are needed or desired.

Spark SQL

• Spark SQL is a module to facilitate data analysis using structured APIs like DataFrames, DataSets, and SQL.
• This allows applying database system benefits in data analysis.

Spark SQL - DataFrames

• DataFrames represent tables with columns in Spark SQL.
• They allow loading and treating datasets as tables for manipulation.
• DataFrames are popular in other languages (e.g. Python and R) and were implemented in a distributed setting for Spark.

DataFrames Code Example

• In the example, a DataFrame ("dataDF") is created by converting an RDD ("dataRDD") into a tabular format.
• The code then aggregates data, calculates average salary for the "IT" department, and collects the results for display.

Interview Question

• A key difference lies in how DataFrames are managed.
• Python's DataFrames are typically local to the execution environment.
• Spark DataFrames are distributed across the cluster nodes for broader processing capabilities.

Behind the Scenes

• Spark SQL utilizes the Catalyst Optimizer to optimize code for efficient execution.
• The optimizer converts high-level APIs such as DataFrames, DataSets, and SQL into optimized RDD operations.

Spark SQL - API Overview

• API to work with RDDs is considered low-level.
• Structured APIs (DataFrames, DataSets and SQL) are high-level interfaces, and ultimately operations are translated into RDDs.
• The structured API is favored in Spark.

Spark SQL - Multiple APIs

• Spark SQL offers a choice of DataFrames, DataSets, or SQL for implementing operations.
• These structured APIs excel compared to the low level RDD methods in terms of readability and efficiency.
• Examples (e.g. DataFrames) in Python, Scala, and R are possible.

Example of Speed Comparison

• A performance comparison of DataFrames and RDDs shows DataFrames to be generally faster for data aggregation involving 10 million integer pairs

Summary

• Spark jobs written using RDDs may exhibit performance issues and be difficult to follow.
• The use of structured APIs (such as DataFrames, DataSets, and SQL) streamlines code readability and execution speed.
• Spark supports these APIs in multiple areas.

DataFrames vs DataSets vs SQL

• Differences in syntax structure and run-time processing may exist among DataFrames, DataSets, and SQL.

Code in DataFrames Example

• A DataFrame example illustrates typical operations.
• Import statements from org.apache.spark.sql.functions provide functional capabilities when used with DataFrames.
• Conversion of data from RDD to DataFrame ("toDF") occurs.
• groupBy, agg, filter operations manipulate data.
• Final data collection is implemented by .collect method.

Same code in DataSets Example

• DataSets example illustrates similar functionality as DataFrames.
• A key difference is how data is managed inside the DataSet.
• Usage of typed for aggregating data inside the DataSet operations.
• resultDS output will be of "DataSet" type.

Same code in SQL Example

• SQL code performs similar manipulations in a declarative, higher level fashion compared to RDD or DataFrame implementations.
• Operations are defined on tables through SQL queries.

Summary of Structured Data Types

• Analysis errors are possible during compile time, runtime, or in the compilation phase depending on the API usage.

Introduction (Spark Structured API)

• Spark offers these three high-level data operation APIs (DataFrames, DataSets and SQL).
• All of them go through the Catalyst optimizer to optimize for execution speeds.
• DataFrames is the most common choice by Spark users.

Spark RDD

• Example code defines an RDD with data.
• RDD operations (e.g. map, reduceByKey, filter) handle data processing.

Spark RDD (Calculating Average Salary by Department)

• RDD operations continue, transforming dataRDD with data.
• Using operations (e.g. map, reduceByKey, filter, map, collect) to find the average salary.

DataFrames (Calculating Average Salary by Department)

• Example includes code for creating DataFrames and performing calculations. These functions calculate average salary by department ("IT").

SQL (Calculating Average Salary by Department)

• SQL shows similar calculations of average salary.
• Code includes statements using SQL operations in a declarative form.

DataFrame vs DataSet

• DataFrames vs DataSets differ in management and processing of individual column data vs properties inside a specific class. The example code show typical methods.

DataFrame vs DataSet (Type Safety)

• Type safety considerations emphasize the benefits of using DataSets when data type handling is important.
• This is emphasized by the examples (e.g. changing the filter, and data handling types).

DataFrame vs DataSet (Typo Example)

• Example showing what happens when a typo occurs in code, and different ways code behaves between DataFrames vs DataSets.

Dataframe vs DataSet (Trying the same with SQL)

• Example code shows a SQL implementation similar in functionality compared to the DataFrame and DataSet examples.

DataFrame vs DataSet (Handling Errors)

• Example showing the difference in how SQL, DataFrames and DataSets handle syntax errors.

DataFrame vs DataSet (Same for DataFrame)

• Example code replicates DataFrame equivalent to the example shown for DataSets, in dealing with syntax issues.

DataFrame vs DataSet (SQL Operations)

• Example code shows how different code executes when applying SQL operations in the examples.