Section 2: 14. Databricks Relational Entities

Podcast

Play an AI-generated podcast conversation about this lesson

Questions and Answers

What is a database in Databricks equivalent to?

A schema in Hive metastore (correct)

A default folder for data storage

A collection of tables in a file system

An external data source connection

What command is used to create a database in Databricks?

CREATE SCHEMA (correct)

CREATE METADATA

CREATE DATABASE (correct)

CREATE TABLE

Where is the central Hive metastore accessed in Databricks?

On a user-local machine

Only by a single cluster

Through external applications only

By all clusters in a workspace (correct)

What is the default location for table data in Databricks?

/user/hive/warehouse Signup and view all the answers

What extension is used for database folders in the Hive metastore?

.db Signup and view all the answers

If a table is created without specifying a database name, where will its metadata be stored?

Under the default database in the Hive metastore Signup and view all the answers

How do you create other databases apart from the default one in Databricks?

By using the CREATE SCHEMA command Signup and view all the answers

What does the Hive metastore primarily store information about?

Metadata for databases, tables, and partitions Signup and view all the answers

What is indicated by the LOCATION keyword when creating a schema?

The path where the database will be stored. Signup and view all the answers

What happens to the data files of a managed table when the table is dropped?

They are deleted along with the table. Signup and view all the answers

How does Hive manage an external table's data?

Hive manages only the metadata, not the data files. Signup and view all the answers

Which command is used to specify which database to use when creating an external table?

USE Signup and view all the answers

Where is the definition of an external table stored?

In the Hive metastore. Signup and view all the answers

What is the default case for table creation in Databricks?

Creating a managed table. Signup and view all the answers

Can you create an external table in a database created in a custom location?

Yes, external tables can be created in any database. Signup and view all the answers

What is the primary responsibility of Hive regarding managed tables?

It oversees the complete lifecycle of both metadata and the data. Signup and view all the answers

Study Notes

Overview of Databricks and Hive Metastore

Databricks utilizes a Hive metastore to manage database and table metadata.
A database in Databricks is equivalent to a schema in the Hive metastore.
Creating a database can be done using either CREATE DATABASE or CREATE SCHEMA syntax; both are functionally identical.

Hive Metastore

The Hive metastore serves as a repository for metadata, storing information about databases, tables, and partitions.
It contains details regarding table definitions, data formats, and the physical storage locations of the data.

Default Database in Databricks

Each Databricks workspace has a default database named "default."
Tables can be created in the default database using the CREATE TABLE statement without specifying a database name.
By default, the table data is stored in the /user/hive/warehouse directory.

Creating Additional Databases

Additional databases can be created using CREATE DATABASE or CREATE SCHEMA syntax.
New databases will be located in the Hive metastore and stored in the default directory with a .db extension to identify them.

Custom Database Locations

Databases can be created outside of the default Hive directory by specifying a custom path using the LOCATION keyword.
The database definition remains in the Hive metastore, while the folder for the database data will be located at the specified custom path.

Types of Tables in Databricks

Two main types of tables exist in Databricks: managed tables and external tables.
Managed tables store their data in the database directory and are owned entirely by Hive, which manages their lifecycle.
Dropping managed tables will delete both the table definition and the underlying data files.

External Tables

External tables store metadata in the Hive metastore but the actual data resides outside the database directory, specified by the LOCATION keyword.
Dropping an external table does not delete the underlying data files, allowing for data persistence even after the table’s removal.

Creating Tables

External tables can be created within the default database or any specified database by using the CREATE TABLE statement with the LOCATION keyword.
Switching between databases can be done using the USE keyword, followed by the database name.

Summary of Operations

Both managed and external tables function through similar creation syntax, but their data management and lifecycle differ significantly, affecting storage and deletion behavior.
Understanding how to effectively manage tables and databases within Databricks is essential for efficient data organization and manipulation.

Studying That Suits You

Use AI to generate personalized quizzes and flashcards to suit your learning preferences.

Description

This quiz explores relational entities in Databricks, focusing on how databases and tables operate within the platform. Learn about the impact of the LOCATION keyword on the default storage directory and the relationship between databases and schemas in Hive metastore.