(Delta) Chapter 9: Delta Sharing (Multiple Choice)

Managing Data Sharing Solutions

• Commercial data-sharing solutions are widely chosen by companies as an alternative to building in-house solutions.
• They offer a balance between not wanting to allocate extensive time and resources to developing a proprietary solution and desiring greater control than what cloud object storage can provide.
• These solutions provide simplicity for users to share data with others on the same platform.

Limitations of Commercial Data-Sharing Solutions

• Vendor lock-in: Commercial solutions often lack interoperability with other platforms, making it difficult to share data with users of competing solutions.
• Data movement: Data needs to be loaded onto a specific platform, which involves additional steps, such as ETL and creating copies of the data.
• Scalability: Commercial data-sharing solutions may have limitations on scaling imposed by the vendors.
• Cost: The challenges mentioned above contribute to additional costs for sharing data with potential customers, as data providers need to replicate data for different recipients across various cloud platforms.

Cloud Object Storage

• Object storage is highly regarded as a well-suited solution for cloud environments due to its elastic nature and seamless scalability.
• It allows handling vast amounts of data and effortlessly accommodating unlimited growth.

Open Source Delta Sharing

• Open source data sharing is not associated with a vendor-specific technology that introduces unnecessary limitations and financial burdens.
• Delta Sharing is an open source protocol designed to provide open cross-platform data sharing, allowing data sharing in Delta Lake and Apache Parquet formats with any platform, whether on premises or another cloud.

Delta Sharing Goals

• Open cross-platform data sharing: Delta Sharing provides an open source, cross-platform solution that avoids vendor lock-in.
• Share live data without data movement: Data recipients can directly connect to Delta Sharing without replicating the data.
• Support a wide range of clients: Delta Sharing supports a diverse range of clients, including popular tools like Power BI, Tableau, Apache Spark, pandas, and Java.
• Centralized governance: Delta Sharing provides robust security, auditing, and governance capabilities, allowing data providers to have granular control over data access.

Delta Sharing Protocol

• Delta Sharing lets data providers share existing tables or parts thereof stored on their cloud data lake in Delta Lake format.
• The data provider decides what data they want to share and runs a sharing server in front of it that implements the Delta Sharing protocol and manages access for recipients.
• As a data recipient, you only need one of the many Delta Sharing clients supporting the protocol.

Delta Sharing Under the Hood

• The recipient's client authenticates to the sharing server and asks to query a specific table.
• The server verifies whether the client is allowed to access the data, logs the request, and then determines which data to send back.
• The server generates short-lived pre-signed URLs that allow the client to read these Parquet files directly from the cloud provider.

Reading a Shared Table

• A shared table can be accessed using a Delta Sharing client, passing it the profile path, and listing all shared Delta tables.
• A URL to access a shared table can be created using the syntax: profile_path + "#delta_sharing..".
• The shared table can be read as a pandas DataFrame or a standard PySpark DataFrame using the load_as_pandas() or load_as_spark() method, respectively.

This quiz assesses your understanding of scalability issues in data management and commercial data-sharing solutions. It covers the limitations of in-house and cloud-based solutions.