Ch 10: Build a Lakehouse on Delta Lake (True and False)

Data Lakes

• Offer simple integration through standardized APIs, allowing users with different skills, tools, and programming languages to perform various analytics tasks simultaneously.

Replication

• Cloud providers offer easy setup of replication to different geographical locations for data lakes, useful for:
  • Meeting compliance requirements
  • Failover for business-critical operations
  • Disaster recovery
  • Minimizing latency

Availability

• Cloud systems offer different types of data availability for cloud data lakes, allowing for lifecycle policies to move data across different storage availability classes for:
  • Compliance
  • Business needs
  • Cost optimization
• Availability can be defined through service-level agreements (SLAs), guaranteeing a minimal level of service, typically greater than 99.99% uptime for business-critical resources.

Cost

• With cloud data lakes, users typically pay for what they use, and costs always align with data volumes.
• Costs are minimized through:
  • A single storage layer
  • Minimal data movement
  • Decoupled storage versus compute

Apache Spark Ecosystem

• Consists of different libraries, with Spark Core and Spark SQL engine as the substrate.
• Spark SQL supports ANSI SQL, allowing users to query and analyze data using SQL syntax.
• Spark SQL library allows users to interact with datasets and tables using the DataFrame API.
• Time travel enables easy auditing or reproduction of machine learning models.

Lakehouse Architecture

• Bridges the gap between data engineers and data scientists, unifying workloads and reducing silos.
• Enables organizations to build and manage machine learning models in a faster, more efficient way.

Medallion Architecture

• Consists of Bronze, Silver, and Gold layers:
  • Bronze layer: landing zone for raw data, no schema required, fast and easy to get new data.
  • Silver layer: first layer useful to the business, prioritizes speed to market, enables data discovery, self-service, ad hoc reporting, advanced analytics, and ML.
  • Gold layer: data is in a format easy for business users to navigate, prioritizes business use cases, highly performant, and pre-calculated, business-specific transformations.

The Emergence of the Lakehouse

• In the late 2010s, the concept of the lakehouse emerged as a modernized version of a data warehouse.
• A lakehouse provides all the benefits and features of a data warehouse without compromising the flexibility of a data lake.
• It leverages a low-cost, flexible cloud storage layer, a data lake, combined with data reliability and consistency guarantees through technologies that feature open-table formats with support for ACID transactions.

Storage Layer

• A cloud object store like a data lake is the foundational storage layer for a lakehouse.
• A data lake allows for storing massive amounts of data in a flexible, cost-effective, and scalable manner.
• It enables processing of all data on a single system, preventing the creation of additional copies of data and reducing integration points and errors.

Flexibility of a Data Lake

• Cloud data lakes allow for ultimate flexibility to store data, whether it's velocity (streaming versus batch), volume, or variety (structured versus unstructured).
• They can work with data that enters the data lake at any speed, whether it's real-time data or volumes of data ingested in batches.
• Infrastructure can be provisioned on demand, and quickly scaled up or down elastically.

Decoupled Storage and Compute

• Traditional data warehouses and on-premises data lakes have tightly coupled storage and compute.
• Cloud data lakes allow for decoupling of storage and compute, enabling independent scaling of storage and compute.
• Storage is generally inexpensive, whereas compute is not, making it cost-effective to store vast amounts of data.

Delta Lake and SQL Interface

• Delta Lake ensures serializability, providing full support for concurrent reads and writes.
• The Spark SQL engine generates an execution plan to optimize and execute queries on the cluster.
• Users can leverage Spark SQL to perform queries on Delta tables, taking advantage of the performance and scalability of Delta tables.

Spark SQL Execution Plan

• The execution plan uses Spark SQL RDDs, also referred to as DataFrames or datasets, optimized for structured data.
• The DataFrame API is recommended for ETL and data ingestion processes, while Spark SQL is suitable for most data consumers.

Lakehouse Architecture

• A lakehouse architecture combines the capabilities of a data lake with the features of a data warehouse.
• It provides a single unified platform for diverse workloads, enabling a single security and governance approach for all data assets.
• The architecture includes a cloud-based data lake, a transactional layer, and a high-performance query engine.