(Spark) Chapter 4. Structured API Overview

Overview of Structured APIs

• Structured APIs in Spark manage various types of data: unstructured log files, semi-structured CSV files, and structured Parquet files.
• Core types include Datasets, DataFrames, SQL tables, and views, facilitating data manipulation across batch and streaming computations.
• Migration between batch and streaming processes is seamless with Structured APIs.
• These APIs serve as fundamental abstractions for writing most data flows within Spark.

DataFrames and Datasets

• DataFrames and Datasets are distributed table-like collections ensuring consistent types with defined rows and columns.
• Both represent immutable, lazily evaluated plans guiding the transformation of data.
• Actions trigger the execution of transformations, resulting in actual data manipulation.
• SQL tables and views can be treated as DataFrames, with differences primarily in the syntax used for execution.

Schemas

• A schema defines the structure of a DataFrame, specifying column names and types.
• Schemas can be defined manually or inferred from data sources (schema on read).
• Consistent typing is crucial, and Schema includes definitions for various data types used in Spark.

Spark Types and Programming Interfaces

• Spark operates with its internal engine, Catalyst, which optimizes execution by maintaining type information.
• Manipulations in Spark are largely confined to Spark types, irrespective of the language used (Scala, Java, Python, R).
• Type declarations vary across programming languages, with specific type imports needed depending on the chosen interface.

Data Type References

• Python types: ByteType, ShortType, IntegerType, etc. must align with Spark’s expectations, ensuring values fit specific byte ranges.
• Scala and Java consistency is crucial, utilizing respective libraries to define and manipulate Spark types effectively.
• Each language has slight variations in how types are implemented and accessed.

Execution Process of Structured APIs

• A structured query undergoes a multi-step transformation before execution:
  • User writes DataFrame/Dataset/SQL code.
  • If valid, Spark converts the code into a Logical Plan.
  • This Logical Plan is transformed into a Physical Plan with optimizations.
  • The Physical Plan is executed through RDD manipulations on the Spark cluster.
• The Catalyst Optimizer plays a key role in determining the execution pathway, ensuring efficient processing before returning results.

Overview of Spark’s Structured APIs

• Structured APIs allow manipulation of various data formats including unstructured log files, semi-structured CSV files, and structured Parquet files.
• Three core types of distributed collection APIs are: Datasets, DataFrames, SQL tables and views.
• Majority of Structured APIs support both batch and streaming computations, facilitating easy migration between the two modes.

Fundamental Concepts

• Spark operates on a distributed programming model where the user specifies transformations.
• Transformations build Directed Acyclic Graphs (DAGs) which begin execution upon invoking an action, breaking down tasks across the cluster.
• DataFrames and Datasets are key logical structures manipulated through these transformations and actions.

DataFrames and Datasets

• DataFrames and Datasets are distributed collections that have well-defined rows and columns, maintaining consistent types across columns.
• Spark treats them as immutable plans for data manipulation that inform the transformations to be applied.
• Tables and views can be treated similarly to DataFrames, with SQL executed against them.

Schemas

• A schema defines the structure of a DataFrame, including column names and data types.
• Schemas can be defined manually or derived from data sources, referred to as "schema on read."

Spark Types and Internal Mechanics

• Spark utilizes its own type system and engine called Catalyst, which performs execution optimizations through internal type representations.
• Spark’s operations mainly work on its internal data types rather than native language types, ensuring efficiency in execution.

Difference Between DataFrames and Datasets

• DataFrames are considered "untyped," with type safety enforced at runtime; Datasets provide compile-time type checking for JVM languages (Scala, Java).
• In Scala, DataFrames are represented as Datasets of Type Row, allowing for optimized in-memory computation.
• In Python and R, there are no Datasets; all collections are treated as DataFrames utilizing Spark's optimized format.

Data Types Overview

• Various data types supported by Spark include:
  • Primitive types: IntegerType, FloatType, StringType, BooleanType, etc.
  • Complex types: ArrayType, MapType, StructType.
  • Each type has a corresponding reference in Scala, Java, Python.

Execution Process of Structured APIs

• Writing DataFrame/Dataset/SQL code triggers Spark to convert it into a Logical Plan.
• Logical Plan is transformed into a Physical Plan with optimization checks.
• Spark executes the Physical Plan by manipulating RDDs across the cluster.
• The entire workflow involves submission of code, optimization via Catalyst, and execution yielding results back to the user.

Important Notes

• Spark SQL and its types may evolve, so it's advised to refer to the latest Spark documentation for updates.
• Understanding and utilizing DataFrames and Spark-based operations allows for efficient data analysis and processing within Spark’s architecture.

Spark DataFrames and Datasets

• Each column in a DataFrame must have the same number of rows; nulls can specify missing values.
• DataFrames and Datasets are immutable and represent plans for data transformations in Spark.
• Actions on DataFrames trigger Spark to execute transformations and return results.
• Tables and views in Spark are equivalent to DataFrames, with SQL executed instead of DataFrame code.

Schemas in Spark

• A schema defines column names and data types for DataFrames.
• Schemas can be defined manually or read from a data source, known as schema on read.
• Spark's internal engine, Catalyst, manages type information for optimization during processing.

Spark Types

• Spark uses its own types across various language APIs (Scala, Java, Python, SQL, R).
• DataFrames in Spark Scala are Datasets of type Row, facilitating efficient in-memory computation.
• In Python or R, all data structures are treated as DataFrames, leveraging Spark's optimized format.

DataFrames vs. Datasets

• DataFrames are considered untyped because type validation occurs at runtime.
• Datasets offer type-checking at compile time, available only for JVM languages (Scala and Java).
• The Row type is Spark's optimized representation of data for computation, reducing garbage collection and object instantiation overhead.

Column and Row Definitions

• Columns represent simple (integer, string) or complex (array, map) types, tracked by Spark for transformation.
• Rows are records of data defined by the Row type, created through various methods including SQL and RDDs.

Internal Spark Type Representations

• Spark has numerous internal type representations, essential for data manipulation.
• Various language bindings have specific APIs to instantiate columns of certain types (e.g., ByteType, IntegerType).

Python, Scala, and Java Type References

• Python: Uses standard types like int, string, float, and specific Spark types (e.g., ByteType, StringType).
• Scala: Utilizes specific Scala types (e.g., Int, String) and Spark’s internal types (e.g., ArrayType).
• Java: Follows Java's native types (e.g., int, String) and Spark's DataType classes for structuring data.

Execution Process in Structured APIs

• Execution involves writing DataFrame/Dataset/SQL code which Spark validates.
• Valid code becomes a Logical Plan; transformations lead to a Physical Plan with optimizations.
• The Physical Plan executes RDD manipulations across the cluster, managed by the Catalyst Optimizer.
• The execution process includes user code submission, logical and physical planning, and final execution with returned results.