Delta Lake: Origins, Features, and Applications

• The discussion revolves around Delta Lake, its origins, features, and applications.

Introduction & Event Details

• The session is part of "From Tahoe to Delta Lake," broadcast on Zoom, YouTube, and LinkedIn.
• The Data + AI Online meetup group hosts upcoming online events; check meetup.com/data-ai-online.
• Subscribe to the Databricks YouTube channel.
• Follow Databricks on LinkedIn.
• A Data + AI Online meeting with community lightning talks is scheduled for this Thursday.
• Speakers include: Fraca Patana, Siobhan G, Sristaba Alyssa Vishnek, and Yan Zhang
• Data Brew by Databricks is a podcast/vidcast series with discussions on lake houses, including an episode with Michael Armbrust and Mate Zaharia.
• The Data + AI Summit is next week, featuring training, keynotes, and technical sessions.
• Delta Lake evolved from a project code-named Tahoe.

Origin of Delta Lake

• Delta Lake originated from the structured streaming project in Databricks.
• The "stream team" (Michael, Ryan, Barack, TD) reimagined streaming within Spark.
• Spark streaming had limitations: no event time, inability to change queries and recover, and scalability problems.
• The team aimed to make streaming a first-class part of Spark SQL.
• Streaming to files on S3 was challenging due to S3's eventual consistency.
• The file sync in Apache Spark has a transaction log tracking valid files and versions but is limited to a single writer and lacks transactions.
• The team recognized the need for a scalable transaction log to handle large-scale data ingestion.
• Michael named the project "Tahoe" to represent a massive data lake, inspired by Lake Tahoe's depth and volume.

Tahoe Code Name

• Michael chose the name Tahoe for the project to emphasize its ambition to create a massive data lake.
• Lake Tahoe's depth and the fact that its water would cover California in over a foot of water inspired the name
• The initial pull request for the project was named "Project Tahoe" to generate excitement.

Delta Lake and Streaming Scenarios

• Apache Spark's file stream sync had limitations: no multiple writers to the same location, no combining streaming and batch writers.
• Delta protocol was designed for multiple concurrent writers with ACID guarantees and scalability.
• Delta Lake expanded the use cases for structured streaming, enabling multiple streaming queries on the same table.
• The technology enables streaming queries that generates key value data to absurd into a delta table.

Delta Lake Solves Streaming Challenges

• Streaming workloads involve a trade-off between latency and throughput.
• Writing data quickly for low latency results in smaller files, impacting query performance on cloud object stores.
• Delta Lake allows compaction on data by a separate writer to improve query performance by merging data files into bigger files.

Necessity of Delta Lake

• Delta Lake eliminates consistency issues associated with AWS S3's eventual consistency.
• Delta Lake unifies the data layer.
• Both batch and streaming jobs can read and write from Delta Lake tables simultaneously.

Delta Rust API

• Scribd decided to support the Delta Rust API to enable access to Delta Lake for non-JVM languages such as Ruby, Python, and Go, including integration with their respective code frameworks.
• Rust implementation expands the scope of Delta Lake within Scribd.
• The Delta transaction protocol is open and relatively straightforward to implement.
• Tyler pitched the idea in April of last year and got it finished.
• The technology is now a production-ready library being used at Scribnd, and other companies
• A Python binding allows converting a Delta table into a Pandas data frame for native Python data processing with a contributor named Florian.

Connecting Systems to Delta Lake

• Delta Lake has a Hive connector for reading Delta tables.
• A Delta standalone reader allows reading Delta tables without Spark.
• The technology has spatial integration for Presto via manifest files.
• There is an open-source pull request to use the Hive connector for reading Delta tables in Presto.

Reading Delta Without Spark

• Delta RS allows reading Delta Lake tables without Spark.
• A Java standalone reader is also available.
• A Python binding enables Delta table access from a single local machine or cloud machine.
• The Python binding can convert a Delta table into a Pandas data frame.

Google Cloud Storage (GCS)

• Delta Lake is supported over Google Cloud Storage (GCS).
• It was supported even before Databricks on Google.
• Rust and Python bindings do not currently support Google Cloud Storage.
• All are welcome to contribute.

Migrating Transactional Data to Delta Lake

• SQL Delta Import, an import created for nightly operation, transfers relational data to Delta Lake.
• Kafka is used for streaming workloads.
• Kafka Delta Ingest project is open-source in Delta Lake.
• Debezium is a tool that connects to a relational data store to start streaming changes off of that data store.

Delta Lake and Data Versioning

• Each commit in Delta Lake represents a version of the table.
• Commits are stored and appear as deltas in the transaction log.
• Users can compute the table's state at any version and track changes over time with multi-version concurrency control(MVCC).
• Users can query the latest version of the table or go back in time to a previous version
• Picking a version in the transaction log allows you to go to in time travel and then query from that version.

Naming of Delta Lake

• The name Delta was suggested because rivers flow into deltas, depositing sediments where crops grow.
• It's a metaphor for streams coming in, sediments and quality being controlled.
• It also fits the architecture where massive firepower streams come in and split into materialized views.

Time Travel Implications and Vacuuming

• The Delta log must contain the data you're trying to access for time travel.
• Executing a vacuum command deletes files, impacting time travel capabilities.
• Consider cloning into an archive table monthly and then running vacuum to clean up other tables.
• In streaming tables, compacting partition data will create a new transaction removing and adding data
• Compaction is when a new transaction removes all those little files and then adds four new files.

Cloning Operations

• Shallow clones are cheap, as they only clone pointers using the transaction log.
• Deep clones copy all the data.
• Use deep clone to archive a specific table snapshot to avoid the impact of vacuum operations.
• Shallow clones are useful for development.
• Deep clones are really useful for archiving.

Machine Learning with Delta Lake

• MLflow's autologging feature tracks which Delta table version was used for model training.
• Users can reproduce experiments with the same input parameters.
• Archive table versions using deep clone to compare model performance over time, also known as A/B testing.
• Delta Lake serves as a repository for features for large-scale distributed training and inference.
• High quality data and Delta Lake gives you better machine learning models.

Additional Insights on Delta Lake and Machine Learning

• MLflow allows reproducing experiments with the exact same data, addressing data drift.
• Ensuring data quality with Delta Lake helps mitigate model drift.
• Transactional protection and schema enforcement in Delta Lake maintain data quality.

Batch and Streaming ML Pipelines

• Delta Lake tables allow transitioning from batch to streaming ML pipelines without changing the underlying data store.

Schema Enforcement and Evolution

• Delta Lake enforces schema validation using the schema stored in the transaction log.
• Delta Lake supports schema evolution, allowing new columns to be added.
• Values in new column will return null values when read.

Check Constraints

• Delta Lake supports check constraints for data quality.
• Users can define SQL expressions to ensure data formats are correct.
• Delta Lake rejects data with bad records based on these checks.

Implementing Streaming Pipeline

• Several customers have millions of records per second, for example between 3,000,000 and 5,000,000.
• The latencies on throughput are around 15 seconds or one minute, but possible if you scale well
• Throughputs of 1.2 to 1.3 million records per second are possible with structured streaming as well as Delta.
• Writing to files and reading back from those files incurs a cost of serialization and deserialization.
• Implement Kafka to Kafka with a second timeframe.
• Achieving low latency and high throughput performance requires careful consideration of the instrumentation strategy with streaming workloads.