Spark Core Concepts

Spark Core

• Resilient Distributed Datasets (RDDs): fundamental data structure in Spark, representing a collection of data that can be split across multiple nodes
• RDD Operations: two types of operations:
  • Transformations: operations that create a new RDD, e.g., map, filter, groupBy
  • Actions: operations that trigger computation and return a result, e.g., count, collect, reduce
• DAG (Directed Acyclic Graph) Scheduler: Spark's scheduling engine, responsible for executing RDD operations
  • Breaks down RDD operations into a DAG of tasks
  • Schedules tasks on the cluster, considering dependencies and node availability

Memory Management

• Memory Hierarchy: Spark uses a hierarchical memory management system to optimize performance
  • Execution Memory: used for caching and storing intermediate results
  • Storage Memory: used for storing persisted RDDs
  • System Memory: used for JVM overhead and other system-level memory allocation
• Memory Modes:
  • In-Memory Mode: Spark stores data in memory for faster access
  • Disk Spill Mode: Spark spills data to disk when memory is limited
• Cache and Persist:
  • Cache: temporarily stores an RDD in memory for faster access
  • Persist: stores an RDD in memory and/or disk for later reuse
  • StorageLevel: defines the storage behavior for an RDD, e.g., MEMORY_ONLY, DISK_ONLY, MEMORY_AND_DISK

Spark Core

• Resilient Distributed Datasets (RDDs): a fundamental data structure in Spark, representing a collection of data split across multiple nodes.

RDD Operations

• Transformations: operations that create a new RDD, e.g., map, filter, groupBy.
• Actions: operations that trigger computation and return a result, e.g., count, collect, reduce.

DAG (Directed Acyclic Graph) Scheduler

• Scheduling Engine: responsible for executing RDD operations.
• Task Scheduling: breaks down RDD operations into a DAG of tasks and schedules them on the cluster, considering dependencies and node availability.

Memory Management

Memory Hierarchy

• Hierarchical Memory Management: Spark's memory management system optimizes performance.
• Execution Memory: used for caching and storing intermediate results.
• Storage Memory: used for storing persisted RDDs.
• System Memory: used for JVM overhead and other system-level memory allocation.

Memory Modes

• In-Memory Mode: Spark stores data in memory for faster access.
• Disk Spill Mode: Spark spills data to disk when memory is limited.

Cache and Persist

• Cache: temporarily stores an RDD in memory for faster access.
• Persist: stores an RDD in memory and/or disk for later reuse.
• StorageLevel: defines the storage behavior for an RDD, e.g., MEMORY_ONLY, DISK_ONLY, MEMORY_AND_DISK.

Azure Databricks Overview

• Fast, easy, and collaborative analytics platform based on Apache Spark
• Provides a managed platform for data engineering, data science, and data analytics

Data Engineering with Azure Databricks

• Ingest data from various sources (e.g., Azure Storage, Azure Data Lake Storage, Azure Cosmos DB) using Azure Databricks' connectors
• Process and transform large datasets using Apache Spark for scalability and performance
• Create, schedule, and manage data pipelines using Azure Databricks' workflows and jobs
• Store and manage data in various formats (e.g., CSV, JSON, Avro) using Azure Databricks' data storage solutions (e.g., Databricks File System (DBFS))

Apache Spark on Azure Databricks

• Create and manage Spark clusters on Azure Databricks with automatic scaling and termination
• Supports multiple Apache Spark versions for flexibility and compatibility
• Leverage Spark's APIs (e.g., DataFrame, Dataset, RDD) for data processing, machine learning, and graph processing
• Use popular Spark libraries, such as MLlib (machine learning), GraphFrames (graph processing), and Structured Streaming (streaming data processing)

Additional Features

• Supports collaborative workspaces for data engineers, data scientists, and data analysts
• Provides enterprise-grade security features, such as Azure Active Directory integration and encryption
• Integrates with other Azure services (e.g., Azure Synapse Analytics, Azure Storage) and popular data tools (e.g., Jupyter, Git)