Spark Streaming Late Events and Timing

Study Notes

Spark streaming handles data arriving at a later time than expected, a key concern in real-time processing.
Different output modes exist for Spark streaming.
Non-aggregate queries and aggregate queries were covered in previous sections.
Understanding what's possible and impossible with these different output modes is important.
The concept of "windowing" involves summarizing data over a specific time period.
Total action counts are calculated for a given time point but not for specific windows.
This is an important concept especially in Internet of Things (IoT) applications. For example, a website is gathering order data in real time; total orders is important but knowing order counts by the hour permits observing spikes and potentially adding more resources.

Two types of timing exist in streaming applications:
- Event time: The time when an event originates at its source.
- Processing time: The time when an event is processed by the system, determined when it reaches Spark streaming, or just prior to processing.
Developers are more interested in event time because it represents when the event actually happened.
Processing time indicates when the event is picked up for processing.
The concept of 'late events' arises because events can arrive at the streaming or processing centre considerably later than anticipated.

The process of calculating event times is studied before looking at its application to windowing events.
Spark gathers the total number of events to date.
The focus now shifts to grouping events into 10-minute intervals.

Event times are crucial in streaming applications.
Late events are challenging to manage because events might arrive significantly later than their processing time.
For illustration, consider an accelerometer sending step information. If the Wi-Fi connection is interrupted, events generated earlier might arrive much later, raising issues.
Spark needs to maintain states for an indefinite duration, causing the resulting table to indefinitely grow in size. This is a major concern.
- It's a significant problem
- Spark needs to update the window to match the late event.
- Spark needs to maintain states for an indefinite amount of time - leading to problems

A cutoff time for late events is required for solution.
If an event arrives beyond a pre-determined threshold, it can be safely ignored.
Implementing this strategy avoids indefinite table growth.
The approach is called "watermarks."

Watermarks help manage late events by defining a cutoff time for considering events.
Examples: Assuming a watermark set to 10 minutes and data up to 10:00 AM was already processed. The threshold for considering new data would be 9:50 AM.
Spark's capability to not process events older than this timestamp effectively restricts late data from entering the current calculations.

For managing late events, Spark indefinitely maintains all windows. This makes state management challenging and may overwhelm storage.
To address this, Spark Streaming introduced the concept of watermarks.

This description uses a visualization of time, showing watermarks for different triggers.
Watermarks determine the threshold for triggering a new process step or updating the calculation.
They are used to detect and ignore events that occur significantly after a timestamp that represents a boundary of a time-based procedure.

The concept of watermarks involves defining a cutoff time or timestamp threshold.
If the event time falls after this timestamp, the event can be safely excluded from consideration in specific processing steps.

Complete output mode in Spark streaming is unaffected by watermarks. All data remains in the result table regardless of whether it is deemed "late."

Append mode only permits appending new entries to the result table. Updates are not allowed.
Therefore, append mode does not directly support query aggregations.