(Spark) Ch 6 Working with Different Types of Data: (Short Quiz)

Data Transformation Tools

• Data transformation tools transform rows of data from one format or structure to another, and can create more rows or reduce the number of rows available.

Reading Data into a DataFrame

• Data can be read into a DataFrame using Scala or Python.
• The read.format() method specifies the format of the data (e.g. CSV).
• The option() method specifies options such as headers and schema inference.
• The load() method specifies the location of the data.

Data Schema

• The data schema is the structure of the data.
• The schema includes information such as column names and data types.
• The schema can be printed using the printSchema() method.

Data Manipulation

• Data can be manipulated using methods such as withColumn() and filter().
• The withColumn() method adds a new column to a DataFrame.
• The filter() method filters data based on certain conditions.

Replacing Null Values

• Null values can be replaced using the fill() method.
• The fill() method replaces null values with a specified value.
• The replace() method can also be used to replace specific values in a column.

Ordering

• Data can be ordered using the asc() and desc() methods.
• The asc() method sorts data in ascending order.
• The desc() method sorts data in descending order.
• The nulls_first() and nulls_last() methods specify how null values are handled.

Complex Types

• Complex types are used to organize and structure data.
• There are three types of complex types: structs, arrays, and maps.
• Structs are similar to DataFrames within DataFrames.
• Arrays store multiple values in a single column.
• Maps store key-value pairs.

Structs

• Structs can be created using the struct() function.
• Structs can wrap a set of columns in a query.
• Structs can be queried using the dot syntax or the getField() method.

Arrays

• Arrays can be created using the split() function.
• The split() function splits a string into an array of values.
• The explode() function converts an array into a set of rows.

Maps

• Maps can be created using the map() function.
• Maps store key-value pairs.

User-Defined Functions (UDFs)

• UDFs define custom functions in Spark.
• UDFs can be used to write custom transformations using Python or Scala.
• UDFs can take and return one or more columns as input.
• UDFs are registered as temporary functions to be used in a specific SparkSession or Context.