SparkSQL and DataFrames

• The quiz covered Spark and SparkSQL.
• The time limit was 60 minutes with 15 questions.
• The quiz was expected to take 30 minutes.

Attempt History

• The latest attempt took 17 minutes and scored 12 out of 15.
• The attempt was submitted on Mar 4 at 6:40pm.

Question 1

• In Spark SQL, the main unit of abstraction of the data model is the Data Frame.

Question 2

• "Extensibility" of the CATALYST query optimizer for SparkSQL means new optimization rules can be added.

Question 3

• In Core Spark, the main unit of data abstraction is the RDD.

Question 4

• The optimizer of Spark SQL is rule-based.

Question 5

• A Data Frame is an RDD with schema information.

Question 6

• When a table is read from a subsystem such as HBase in Spark, it does not automatically become a Data Frame.

Question 7

• A "Temporary Table" in SparkSQL is a Data Frame defined as an in-memory table for the current session.

Question 8

• Row format storage is better compared to column format when selecting all or the majority of columns of a table.

Question 9

• End-users cannot define new optimization rules in the CATALYST optimizer.

Question 10

• In Spark SQL, SQL transformations can be expressed as an SQL query statement within the .sql() function or as a procedural workflow of a sequence of operations.

Question 11

• Similar to relational SQL, SparkSQL's operators and optimizations are applied to structured data.

Question 12

• A transformation operation in Spark typically results in a narrow dependency between the input and the output.

Question 13

• Word Count Query yields the least performance gain when implemented in Spark compared to Hadoop's implementation.

Question 14

• One drawback of specialized systems such as Impala, Storm, and Giraph is that they are hard to integrate with each other and to create a unified workflow.

Question 15

• Apache Spark has specialized libraries beyond the Spark Core.