65 Questions
A DataFrame can be transformed by changing the order of columns based on the values in rows.
False
The most common DataFrame transformations involve changing multiple columns at once.
False
DataFrames can be created directly from raw data sources.
True
Transforming a DataFrame always involves adding or removing rows or columns.
False
The expr function cannot parse transformations from a string
False
Columns are a superset of expression functionality
False
The logical tree representation of a Spark expression is a cyclic graph
False
Col("someCol") + 5 is a valid expression in Spark
True
The expr function is only used to create DataFrame column references
False
The sortWithinPartitions method can be used to globally sort a DataFrame by a specific column.
False
The limit method can be used to extract a random sample from a DataFrame.
False
Repartitioning a DataFrame always results in a reduction of the number of partitions.
False
The orderBy method must be used in conjunction with the limit method to extract the top N rows from a DataFrame.
True
The coalesce method is used to increase the number of partitions in a DataFrame.
False
Repartitioning a DataFrame is a cost-free operation.
False
The filter df.filter(col("count") < 2) is not equivalent to the SQL query SELECT * FROM dfTable WHERE count < 2 LIMIT 2.
False
Chaining multiple filters sequentially in Spark can lead to improved performance due to the optimized filter ordering.
False
The filter df.where(col("count") < 2).where(col("ORIGIN_COUNTRY_NAME") =!= "Croatia") is equivalent to the SQL query SELECT * FROM dfTable WHERE count < 2 OR ORIGIN_COUNTRY_NAME!= "Croatia" LIMIT 2.
False
The collect method in Spark is used to iterate over the entire dataset partition-by-partition in a serial manner.
False
Calling the collect method on a large dataset can crash the driver.
True
The show(2) method is used to display the first 2 rows of the filtered DataFrame.
True
The take method in Spark only works with a Long count.
False
The show method in Spark is used to collect all data from the entire DataFrame.
False
The collect method and toLocalIterator method in Spark have the same functionality.
False
Using toLocalIterator can be more expensive than using collect because it operates on a one-by-one basis.
True
What is the primary purpose of creating a temporary view in Spark?
To register a DataFrame for querying with SQL
What is the advantage of using Spark's implicits in Scala?
It provides a more concise way of creating DataFrames
How can a DataFrame be created on the fly in Spark?
By converting a set of rows to a DataFrame using the createDataFrame method
What is the difference between the createDataFrame method and the toDF method in Spark?
The createDataFrame method is used for creating DataFrames with a manual schema, while the toDF method is used for creating DataFrames with an implicit schema
Why is using the toDF method on a Seq type not recommended for production use cases?
Because it does not handle null types well
What is the purpose of the createOrReplaceTempView method in Spark?
To register a DataFrame for querying with SQL
How can a DataFrame be created from a JSON file in Spark?
By using the read.format method with the json format
What is the primary purpose of the select method in DataFrames?
To manipulate columns in DataFrames
What is the purpose of the show method in Spark?
To display the first few rows of a DataFrame
What is the purpose of the StructType in PySpark?
To define the schema of a DataFrame
What is the difference between the select and selectExpr methods in DataFrames?
select is used for column manipulation, while selectExpr is used for string-based expressions
What is the purpose of the org.apache.spark.sql.functions package in DataFrames?
To provide a set of functions for working with DataFrame columns
What is the result of calling the show method on a DataFrame?
The entire DataFrame is collected and displayed
How can you create a DataFrame from a manual schema in PySpark?
By creating a StructType and using the createDataFrame method
What is the purpose of the Row class in PySpark?
To create a single row of data for a DataFrame
What are the three tools that can be used to solve the vast majority of transformation challenges in DataFrames?
select, selectExpr, and functions
What is the purpose of using backticks in the given Scala and Python code snippets?
To escape reserved characters in column names
How can Spark be made case sensitive?
By setting the configuration 'spark.sql.caseSensitive' to true
What is the purpose of the 'selectExpr' method in Spark?
To select columns from a DataFrame and rename them
How can columns with reserved characters or keywords in their names be referred to in Spark?
By using backticks around the column name
What is the difference between the 'select' and 'selectExpr' methods in Spark?
The 'select' method is used to select columns, while the 'selectExpr' method is used to select expressions
What is the purpose of the 'createOrReplaceTempView' method in Spark?
To create a temporary view from a DataFrame
How can columns be removed from a DataFrame in Spark?
By using the 'drop' method and specifying the columns to remove
What is the purpose of the 'show' method in Spark?
To display the first few rows of a DataFrame
What is the primary difference between using collect and toLocalIterator to collect data to the driver?
collect gathers data all at once, while toLocalIterator gathers data partition-by-partition
When using collect or toLocalIterator, what is the main risk of crashing the driver?
The dataset is too large
What is the main benefit of using show with a specified number of rows?
It prints out a limited number of rows nicely
What is the main difference between take and collect?
take returns a specified number of rows, while collect returns the entire dataset
What is the main limitation of using collect or toLocalIterator?
They can cause the driver to crash if the dataset is too large
What is the main benefit of using DataFrames in Spark?
They provide a simple and intuitive API for data manipulation
When should you avoid using collect or toLocalIterator?
When working with large datasets
What is the main consequence of using collect or toLocalIterator on a large dataset?
The driver will crash due to memory limitations
What is a schema in a DataFrame?
A definition of the column names and types
When is it a good idea to define a schema manually?
When using Spark for production ETL
What is the purpose of schema-on-read?
To let the data source define the schema
What can be a potential issue with schema-on-read?
All of the above
What is the result of running spark.read.format("json").load("/data/flight-data/json/2015-summary.json").schema in Scala?
A StructType object
Why is it important to define a schema manually when working with untyped data sources?
To avoid precision issues
What is the advantage of using schema-on-read for ad hoc analysis?
It is usually sufficient for ad hoc analysis
What is the difference between schema-on-read and defining a schema manually?
Schema-on-read lets the data source define the schema, while defining a schema manually involves manual definition
Study Notes
DataFrame Transformations
- DataFrame transformations can be broken down into several core operations:
- Adding rows or columns
- Removing rows or columns
- Transforming a row into a column (or vice versa)
- Changing the order of rows based on the values in columns
Creating DataFrames
- DataFrames can be created from raw data sources
- Expressions can be used to create DataFrames, where an expression is a column reference
- The
exprfunction can parse transformations and column references from a string and can be passed into further transformations - Columns and transformations of columns compile to the same logical plan as parsed expressions
DataFrame Operations
- The
sortWithinPartitionsmethod can be used to sort DataFrames - The
limitmethod can be used to restrict what you extract from a DataFrame - The
repartitionmethod can be used to partition the data according to some frequently filtered columns - The
coalescemethod can be used to reduce the number of partitions
Filtering DataFrames
- The
filtermethod can be used to filter DataFrames - The
wheremethod can be used to filter DataFrames - Multiple filters can be chained together using the
wheremethod - Spark automatically performs all filtering operations at the same time, regardless of the filter ordering
Collecting DataFrames
- The
collectmethod can be used to collect all data from the entire DataFrame - The
takemethod can be used to select the first N rows - The
showmethod can be used to print out a number of rows nicely - The
toLocalIteratormethod can be used to collect rows to the driver as an iterator, allowing for iteration over the entire dataset partition-by-partition in a serial manner
Schemas
- A schema defines the column names and types of a DataFrame
- Schemas can be defined explicitly or let a data source define the schema (called schema-on-read)
- Deciding whether to define a schema prior to reading in data depends on the use case
- Defining schemas manually can be useful in production Extract, Transform, and Load (ETL) scenarios, especially when working with untyped data sources like CSV and JSON
Understand how to create expressions in SparkSQL using the expr function and how it differs from column references created with the col function. Learn about performing transformations on columns and parsing expressions from strings.
Make Your Own Quizzes and Flashcards
Convert your notes into interactive study material.
Get started for free
