Delta Chapter 6: Time Travel (Short Quiz)(True or False)

RESTORE Considerations and Warnings

• RESTORE is a data-changing operation that can affect downstream jobs, such as Structured Streaming jobs.
• RESTORE can lead to re-processing of previous updates to a Delta table by a streaming job, potentially causing duplicate processing.
• The operation restores previous versions of the data using the add file action with dataChange = true.

Operations Resulting from RESTORE

• Table version 0: INSERT operation with AddFile action and dataChange = true.
• Table version 1: INSERT operation with AddFile action and dataChange = true.
• Table version 2: OPTIMIZE operation with AddFile and RemoveFile actions, and dataChange = false.
• Table version 3: RESTORE operation with RemoveFile and AddFile actions, and dataChange = true.

Change Data Feed (CDF)

• The CDF provides an efficient way to track changes to row-level data over time.
• The CDF enables querying of changes to row-level data, providing a full audit trail of data.
• The CDF is essential for regulatory requirements, such as HIPAA, to track changes to electronic protected health information (ePHI).
• Enabling the CDF for all new tables can be done by setting the Spark configuration property spark.databricks.delta.properties.defaults.enableChangeDataFeed to true.

TABLE_CHANGES() SQL Command

• The TABLE_CHANGES command allows viewing changes to a table and its CDF metadata columns.
• The command takes three arguments: table_str, start, and end.
• table_str is the optionally qualified name of the table.
• start is the first version or timestamp of change to return.
• end is an optional argument for the last version or timestamp of change to return.

Using TABLE_CHANGES() Command

• The TABLE_CHANGES command can be used to view row-level changes to a table, including insert, update, and delete operations.
• The command returns the _change_type and _commit_version columns, which indicate the type of operation and the version of the change.
• The command can be used to view changes to a specific table or vendor, and to track changes over time.

Audit Trail and Time-Series Analysis

• The CDF can be used to create an audit trail of changes to a specific record or vendor over time.
• The CDF can be used for time-series analysis, such as tracking the addition of new vendors and their fare amounts over time.
• The CDF provides an efficient way to query changes to row-level data, making it a powerful tool for auditing and analytics.

Delta Lake Time Travel

• When the DESCRIBE HISTORY command is executed, it returns operation metrics, which include the number of files added and removed during an operation.
• When performing an UPDATE, DELETE, or MERGE on a table, data is not physically removed from the underlying storage.
• Instead, these operations update the transaction log to indicate which files should or should not be read.
• Similarly, when restoring a table to a previous version, it does not physically add or remove data; it only updates the metadata in the transaction log to tell it which files to read.

Transaction Log

• The transaction log commits checkpoint files, which save the state of the entire table at a point in time.
• Checkpoint files are automatically generated to maintain read performance by combining JSON commits into Parquet files.
• The checkpoint file and subsequent commits can then be read to get the current state, and previous states in the case of time travel, of the table, avoiding the need to list and reprocess all of the commits.

VACUUM Command

• The VACUUM command is used to remove logically deleted data files from storage.
• The default retention period for deleted data files is seven days.
• When running the VACUUM command, you can specify VACUUM without any parameters to vacuum files that are not required by version older than the default retention period.
• You can also use the RETAIN num HOURS parameter to vacuum files that are not required by versions greater than the number of hours specified in the parameter.

VACUUM Syntax and Examples

• To vacuum a table, specify the table name or filepath and then add any additional parameters.
• Example: VACUUM taxidb.tripData; or VACUUM './chapter06/YellowTaxisDelta/'; or VACUUM delta.'./chapter06/YellowTaxisDelta/' RETAIN 100 HOURS;
• You can also run VACUUM using the parameter DRY RUN to view the list of files that are to be deleted before deleting them.

VACUUM Warnings and Considerations

• It is not recommended to set a retention interval shorter than seven days.
• If you run VACUUM on a table with a short retention interval, files that are still active, such as uncommitted files that are needed by readers or writers, could be deleted.
• Delta Lake has a safety check to prevent you from running a dangerous VACUUM command.
• You can turn off this safety check by setting the Spark configuration property spark.databricks.delta.retentionDurationCheck.enabled = false.