Udemy 11. Understanding Delta Tables - Creating a Delta Lake Table

Creating a Delta Lake Table

• A Delta Lake table is created using a CREATE TABLE statement with a table name and schema.
• The table schema defines the columns and their data types, e.g., ID of type integer, Name of type String, and Salary of type double.
• Delta Lake is the default format, so the USING DELTA keyword is not required.

Confirming Table Creation

• The table is created in the default database.
• The table schema can be viewed in the Data tab, including columns and metadata information.

Inserting Records

• Records are inserted using an INSERT INTO statement.
• Multiple records can be inserted in a single transaction.

Querying the Table

• The table can be queried using a standard SELECT statement.

Table Metadata

• The DESCRIBE DETAIL command provides metadata information about the table.
• Metadata includes table location, number of files, and other information.
• The table location is where the table files are stored.
• The number of files indicates the number of data files in the current table version.

Exploring Table Files

• The %fs Magic Command can be used to explore the table files.
• The table directory contains data files in parquet format.
• The number of files is determined by the number of parallel executors in the Spark cluster.

Update Operations

• Update operations are performed using a UPDATE statement.
• Updates create new files instead of modifying existing ones.
• The transaction log is used to indicate which files are valid in the current version of the table.

Table History

• The DESCRIBE HISTORY command provides a history of the table.
• The transaction log stores all changes to the table, allowing for easy review of table history.
• The transaction log is located in the _delta_log folder in the table directory.
• Each transaction is a new JSON file written to the Delta Lake transaction log.

Transaction Log

• The transaction log contains JSON files representing each transaction.
• The JSON files contain information about the new files written to the table and the files that have been soft deleted.
• The transaction log allows for easy tracking of changes to the table.