Apache Flume: Data Collection and Streaming

• Apache Flume is a high-performance system for data collection.
• Flume is good at extracting and streaming real-time data.
• Flume collects and moves large amounts of data to HDFS.

Key Features of Flume

• High performance system for data collection.
• Widely used for collecting any streaming event data.
• Supports aggregating data from different sources into HDFS.
• Horizontally scalable, extensible and reliable.
• Can add new functionality.
• Can configure multiple Agents with the same task.

History

• Originally developed by Cloudera.
• Donated to Apache Software Foundation.
• Became a top-level Apache project.
• Flume OG gave way to Flume NG(Next Generation).

Flume Reliability

• Channels provide reliability by acting as a middleman.
• Disk-based channels guarantee data durability in the face of power loss.
• Memory channels, though high performance, lose data if power is lost.
• Data transfer between Agents and Channels is transactional.
• Failed data transfer to a downstream agent rolls back and retries.
• Disk-based channels ensure data is safely queued before saying "done".

Flume Scalability

• Scalable to handle the increase of data volume without slowing down or breaking down.
• Agents can be added when needed, both vertically and horizontally.
• Ability to increase linearly or better by adding more resources to the system.
• As load increases, more machines can be added to the configuration.
• Vertical scalability involves adding more power, like CPUs.
• Horizontal scalability involves adding more machines or agents.

Flume Extensibility

• Ability to add new functionality to a system.
• Sources and Sinks can be added to existing storage layers or data platforms.
• Custom Sources or Sinks can be added for new types of storage or platforms.
• General Sources include data from files, syslog, and standard output.
• General Sinks include files on the local filesystem or HDFS.

Key Data Sources

• Log files.
• UNIX syslog.
• Program output.
• Sensor data from devices.
• Status updates.
• Network sockets.
• Social media posts.

Flume Agents

• Enable Flume to collect data using configurable agents.
• Receive data from multiple sources, including other agents.
• Handle large-scale deployments using tiers for scalability and reliability.
• Support inspection and modification of in-flight data.
• Employ filtering and cleaning of the data before final destination.
• Are responsible for collecting and processing data from various sources.
• Responsible for converting to UTC to standardize time for uniformity.
• Can Encrypt/Decrypt to protect data Transmission.

Flume Events

• The smallest piece of data that Flume moves.
• Fundamental unit of data and consist of a body (payload) and headers(metadata).
• Headers are name-value pairs mainly for directing output to the correct destination.

Flume Architecture Components

• Source: collects data and places it into a channel before sending to sink and receives external actor events.
• Sink: Sends an event to its destination.
• Channel: Moves the data from source to sink and buffers events from the source, temporarily storing until fully processed.
• Agent: Java process that configures and hosts the source, channel and sink in a process and ensures data flows smoothly.

Syslog

• Syslog is a system used to collect and store messages about events in a computer system(warnings, errors)
• Flume collects the syslog message, stores it temporarily in memory and saves to HDFS for further processing.

Built-In Flume Sources

• Syslog: Captures messages from UNIX syslog daemon.
• Netcat: Captures data written to a socket.
• Exec: Executes a UNIX program to read events for standard output.
• Spooldir: Extracts events from files appearing in a specified directory.
• HTTP Source: Collects data from HTTP requests.

Built-In Flume Sinks

• Null: Discards all events.
• Logger: Logs event to INFO level using SLF4J.
• IRC: Sends event to a specified Internet Relay Chat channel.
• HDFS: Writes event to a file in the specified directory in HDFS.
• HBaseSink: Stores event in HBase.

Built-In Flume Channels

• Memory Channel fast , not reliable stores events in machine RAM which is volatile.
• File Channel is slower than RAM but reliable as it stores events on the machine local disk .
• JDBC: Stores events in a database table using JDBC, slower than file channel.

Flume Agent Configuration

• Is configured through a Java properties file.
• Multiple agents can be configured in a single file.
• The configuration file uses hierarchical references.
• The parameters for the different component types (source, channel and sink).

Flume Agent Configuration Parameters

• Parameters also vary by subtype
• See the Flume user guide for details on configuration.
• Path may contain patterns based on event headers, such as timestamp.
• By default, HDFS sink uses uncompressed SequenceFiles unless specifying the file type for raw data.

Starting Flume Agent

• Typical command line invocation uses the name arg that must match the agent's name in the configuration file.

Command Line Example:

$ flume-ng agent \ tells Flume to start an agent
- -conf /etc/flume-ng/conf \ specify the directory where Flume
configured file are located
- -conf-file /path/to/flume.conf \ tell flume the exact path to the configure file
- -name agent1 \ name of the agent that you want to run
- Dflume.root.logger=INFO,console

• *ng = Next Generation (prior version now referred to as og).