Databricks and Apache Spark Overview

Databricks Overview

• Databricks is a cloud-based data analytics platform built on Apache Spark.
• It unifies data, analytics, and AI workloads.
• It offers a workspace environment
• It provides a lakehouse platform
• It has a control plane and data plane.

Apache Spark

• A multi-language engine for data engineering, data science, and machine learning.
• Runs on single-node machines or clusters.
• Uses a distributed data processing framework.
• The driver program coordinates processing across multiple executors.
• Executors process data in a distributed file system.
• Spark uses a "Driver" JVM for application execution.
• Parallelism is key to Spark's performance.
• Spark can scale horizontally by adding worker nodes.
• Spark uses Executors and Slots for parallelism.
• Each executor has slots to which tasks can be assigned by the driver.
• Spark has an API for different languages like Python, Scala, R, Java and SQL.
• DataFrames are the higher-level API using SQL queries.
• Resilient Distributed Datasets (RDDs) are the low-level representation of datasets.
• The SparkSession class is the main entrypoint for DataFrame API.

Databricks Lakehouse

• Combines the best of data warehouses (structured data) and data lakes (unstructured data).
• Provides fine-grained governance for data and AI.
• Delivers data reliability and performance.
• Includes data warehouses, structured tables and data lake, unstructured files.

Databricks File System (DBFS)

• A distributed file system mounted in a Databricks workspace.
• Provides storage for data lakes.
• Allows seamless data access without credentials.
• Uses directory and file semantics instead of storage URLs.
• Files persist even after cluster termination.

Databricks Workspace and Services

• Unity Catalog provides fine-grained governance
• Cluster Management allows to manage clusters.
• Workflow Management manages Spark workflows
• Access control manages user access
• Lineage provides historical data.
• Notebooks, repos, and DBSQL are provided by the service.
• Cloud storage is available.

Azure Databricks

• A fully managed, cloud-based data analytics platform.
• Built on Apache Spark.
• Provisioned as an Azure resource.
• Offers standard and premium tiers, also trials.
• It's a fully-managed service offering data science, machine learning, and SQL workloads.
• The user has notebooks for coding.
• The user can train predictive models using frameworks like SparkML.
• Data can be queried and stored in relational tables with SQL.
• Only available for Premium tier workspaces.

Key Concepts

• Apache Spark clusters: provide highly scalable parallel compute for distributed data processing.
• Databricks File System (DBFS): provides distributed shared storage for data lakes, allowing seamless data access and persistence.
• Notebooks: provide an interactive environment to combine code, notes, and images, ideal for exploration.
• Metastore: provides a relational abstraction layer, enabling common database operations, and storing data from files.
• Delta Lake: builds on the metastore to enable common relational database capabilities.
• SQL Warehouses: provide relational compute endpoints for querying data in tables.

Cluster types

• All-purpose clusters: used for interactive data analysis, and have configuration information stored for 70 recently created clusters.
• Job clusters: used for running automated jobs, and have configuration information stored for the 30 most recently terminated clusters.

Cluster Configuration

• Standard(Multi Node): is the standard configuration catering to any supported language.
• Single node: is a low-cost cluster suitable for low-scale machine learning.

Notebook Magic Commands

• Used to override language settings, run utilities, and auxiliary commands.
• %python, %r, %scala, %sql: change notebook's default language
• %sh: executes shell commands on the driver node
• %fs: shortcut for Databricks file system command
• %md: markdown for styling the display
• %run: execute a remote notebook from a notebook
• %pip: install python libraries

dbutils Utilities

• A set of methods to perform tasks in Databricks using notebooks (filesystem operations, secret management, and job management).

Git Versioning with Databricks Repos and CI/CD

• Native integration with Git platforms enables version control and collaboration.
• Supports CI/CD workflows for automation.
• Databricks repos manage customer accounts,datasets and clusters.
• Provides Git and CI/CD systems for version, review, and testing features.

Databricks Notebooks

• Collaborative, reproducible, and enterprise-ready
• Multi-language: Python, SQL, Scala, R
• Collaborative: real-time co-presence, co-editing
• Ideal for exploration: visualization, data profiles
• Adaptable: standard libraries and local modules
• Reproducible: track version history, Git version control