Apache Kafka Overview and Key Features

Apache Kafka Overview

• Kafka is a distributed streaming platform designed for real-time data processing.
• It enables publish-subscribe messaging and is fault-tolerant, scalable, and high-throughput.

Kafka Key Features

• High Throughput and Scalability:
  • Supports processing millions of messages per second with low latency.
  • Achieves horizontal scaling across clusters using partitions.
• Fault Tolerance:
  • Message replication ensures data reliability even with hardware or node failures.
  • Leaders and followers architecture ensures continuous operation.
• Durability:
  • Writes messages to disk and retains them based on user-defined policies.
• Flexible Messaging Model:
  • Supports both publish-subscribe and point-to-point messaging, suitable for various use cases.
• Stream Processing:
  • Provides libraries like Kafka Streams for real-time data transformation and aggregation.
• Decoupled Architecture:
  • Producers and consumers operate independently, enabling seamless integration across distributed systems.
• Integration-Friendly:
  • Works well with external systems using Kafka Connect, supporting plugins for databases, file systems, and more.

Kafka in the Big Data Ecosystem

• Central Data Hub:
  • Acts as a backbone for managing and transferring real-time data between various components in a big data stack.
• Key Integrations:
  • Apache Hadoop/Spark: Used to feed raw data streams for batch or stream processing.
  • Machine Learning Pipelines: Streams data for real-time model updates and predictions.

Why Kafka is Preferred

• Designed for Real-Time Streaming:
  • Excels in scenarios needing high throughput, fault tolerance, and real-time data processing.
• Integration with Big Data Ecosystem:
  • Works seamlessly with systems like Apache Spark, Flink, and Hadoop for building advanced data pipelines.
• Adoption by Industry Leaders:
  • Companies like LinkedIn, Netflix, Uber, and Spotify use Kafka for scalability, speed, and reliability in mission-critical systems.

Who Uses Kafka?

• LinkedIn: Activity data and operational metrics.
• Twitter: Part of Storm – stream processing infrastructure.
• Many other companies including Spotify, Uber, Tumbler, Goldman Sachs, PayPal, Box, Cisco, CloudFlare, DataDog, LucidWorks, MailChimp, Netflix, and others.

Kafka in LinkedIn

• More than 1 petabyte of data in Kafka.
• Over 1.2 trillion messages per day.
• Thousands of data streams.
• Source of all data warehouse and Hadoop data.
• Over 300 billion user-related events per day.

Kafka in Fortune 500's

• Over 35% of Fortune 500 companies use Kafka (2018).
• Strong presence in various sectors: Travel, Global Banks, Insurance, Telecom.

Kafka Advantages

• Low Latency
• High Throughput
• Fault Tolerance
• Scalability
• Batch Handling
• Real-time Handling
• Message Broker Capabilities:
• Distributed By default
• Durable and by default Persistent
• Consumer Friendly

Kafka Disadvantages

• Issues with Message Tweaking
• No Complete Set of Monitoring Tools
• Not Support Wildcard Topic Selection
• Behaves Clumsily
• Lacks Some Messaging Paradigms
• Lack of Pace
• Performance Reduction

Kafka Architecture - Brokers

• Kafka Broker is a server that manages incoming and outgoing messages.
• Key Features:
  • Message Storage: Retains messages for a specified retention period.
  • Request Handling: Handles read and write requests from producers and consumers.
  • Scalability: Kafka clusters consist of multiple brokers for handling large workloads.
  • Each broker has a unique ID and coordinates with others in a distributed system.

Kafka Architecture - Producers

• Client applications responsible for publishing messages to Kafka topics.
• Key Features:
  • Partitioning: Sends messages to specific partitions based on custom logic/random assignment.
  • Acknowledgment: Can request acknowledgments from brokers for data delivery.
  • Compression: Supports message compression for reduced storage and network usage.

Kafka Architecture - Consumers

• Applications subscribing to topics and processing messages.
• Key Features:
  • Consumer Groups: Multiple consumers in a group, with Kafka ensuring each partition is assigned to only one consumer.
  • Offset Management: Tracks message offsets, enabling sequential data processing or resuming from the last read point.
  • Pull Model: Consumers request data from brokers.

Kafka Architecture - Partitions

• Topics are divided into partitions for scalability and parallelism.
• Key Features:
  • Parallel Processing: Multiple partitions enable distributed message handling.
  • Ordering: Guarantees order within a single partition, not across partitions.
  • Load Distribution: Producers distribute messages for even workload handling across partitions.

ZooKeeper's Role

• Distributed coordination service used by Kafka.
• Key Roles:
  • Cluster Management: Maintains metadata about brokers, topics, and partitions.
  • Leader Election: Facilitates leader election for partitions and brokers.
  • Configuration Management: Tracks/propagates configuration changes across the cluster enabling fault tolerance and managing broker failures.