Spark SQL Performance Tuning

Caching data in memory in Spark SQL can be done using the spark.catalog.createTable method.

Spark SQL will automatically compress data in memory to minimize memory usage and GC pressure when caching data.

The uncacheTable method is used to add a table to memory in Spark SQL.

The join method in Spark SQL can be used to specify a join strategy hint.

The setConf method on SparkSession can be used to configure in-memory caching in Spark SQL.

Spark SQL can cache data in memory using a row-based format.

The SHUFFLE_HASH join strategy hint is used to instruct Spark to use a broadcast join strategy.

Experimental options can be turned on to improve performance in Spark SQL for certain workloads.

The MERGE join strategy hint is used to instruct Spark to use a shuffle replicate NL join strategy.

In-memory caching in Spark SQL can be configured using SQL commands.

When the BROADCAST hint is used on table 't1', Spark will always choose the broadcast join strategy regardless of the size of table 't1'.

The SHUFFLE_REPLICATE_NL hint has a higher priority than the MERGE hint in Spark.

The 'COALESCE' hint in Spark SQL requires both a partition number and column names as parameters.

Adaptive Query Execution (AQE) in Spark SQL is disabled by default since Apache Spark 3.2.0.

The coalescing post-shuffle partitions feature in AQE is enabled by default in Spark SQL.

AQE can convert sort-merge join to shuffled hash join when the runtime statistics of any join side is smaller than the adaptive broadcast hash join threshold.

The spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold configuration determines the threshold for converting sort-merge join to broadcast hash join.

The skew join optimization feature in AQE can only split skewed tasks into roughly evenly sized tasks.

The REPARTITION_BY_RANGE hint in Spark SQL must have a partition number as a parameter.

The REBALANCE hint in Spark SQL can only have an initial partition number as a parameter.