Advanced+Delta+Lake+Features (1).pdf

Advanced Delta Lake Features Learning Objectives u Time travel u Compacting Small Files and Indexing u Vacuum Derar Alhussein © Udemy | Databricks Certified Data Engineer Associate - Preparation Time travel u Audit data changes u DESCRIBE HISTORY command Derar Alhus...

Advanced Delta Lake Features Learning Objectives u Time travel u Compacting Small Files and Indexing u Vacuum Derar Alhussein © Udemy | Databricks Certified Data Engineer Associate - Preparation Time travel u Audit data changes u DESCRIBE HISTORY command Derar Alhussein © Udemy | Databricks Certified Data Engineer Associate - Preparation Time travel u Query older versions of the data u Using a timestamp u SELECT * FROM my_table TIMESTAMP AS OF "2019-01-01" u Using a version number u SELECT * FROM my_table VERSION AS OF 36 u SELECT * FROM my_table@v36 Derar Alhussein © Udemy | Databricks Certified Data Engineer Associate - Preparation Time travel u Rollback Versions u RESTORE TABLE command: u RESTORE TABLE my_table TO TIMESTAMP AS OF "2019-01-01” u RESTORE TABLE my_table TO VERSION AS OF 36 Derar Alhussein © Udemy | Databricks Certified Data Engineer Associate - Preparation Compaction u Compacting Small Files u OPTIMIZE command: u OPTIMIZE my_table OPTIMIZE my_table Derar Alhussein © Udemy | Databricks Certified Data Engineer Associate - Preparation Indexing 55 u Co-locate column information 10 1 99 u OPTIMIZE command: 1 … 15 OPTIMIZE my_table 50 u OPTIMIZE my_table 50 ZORDER BY ID ZORDER BY column_name 75 51 30 … 100 8 100 44 66 Derar Alhussein © Udemy | Databricks Certified Data Engineer Associate - Preparation Vacuum a Delta table u Cleaning up unused data files u uncommitted files u files that are no longer in in latest table state u VACUUM command u VACUUM table_name [retention period] u Default retention period: 7 days u Note: Vacuum = no time travel Derar Alhussein © Udemy | Databricks Certified Data Engineer Associate - Preparation