Amazon Kinesis: Real-Time Data Processing

Amazon Kinesis Overview

• Amazon Kinesis facilitates real-time collection, processing, and analysis of streaming data for timely insights.
• Data processing occurs in "shards," with each shard capable of ingesting 1000 records per second.
• Default limit is 500 shards, expandable upon request without limit.
• Records are comprised of a partition key, sequence number, and data blob (up to 1 MB).
• Transient data storage is available with a default retention of 24 hours, extendable to 7 days.

Kinesis Video Streams

• Enables secure video streaming from connected devices for analytics and machine learning.
• Stores, encrypts, and indexes video data streams with easy API access.
• Default data retention period is 24 hours, extendable to 7 days.
• Supports 5 transactions per second for reading, with a maximum read rate of 2MB/second and write rate of 1000 records/second.
• Multiple shards are supported in a stream, with server-side encryption using KMS.

Kinesis Data Streams

• Allows creation of custom applications for processing or analyzing streaming data.
• Ideal for continuous movement and processing of data from producers.
• Data is stored for later application processing, differentiating from Kinesis Firehose which delivers data directly.
• Utilizes records composed of a sequence number, partition key, and data blob.
• Shards serve as the base throughput unit, with each providing 1MB/sec input and 2MB/sec output capacity.

Resharding in Kinesis Data Streams

• Resharding can increase or decrease the number of shards based on data flow rate.
• Shard splitting increases both the number of shards and data capacity, while merging reduces them.
• Partition keys are essential for grouping data by shard.

Kinesis Data Firehose

• Simplifies the process of loading streaming data into data stores and analytics tools.
• Captures, transforms, and loads streaming data without the need for application management.
• Configurable to batch, compress, and encrypt data before delivery.
• Supports delivery streams storing data for up to 24 hours; uses sync replication across three availability zones.

Kinesis Data Analytics

• Helps in processing and analyzing real-time streaming data using standard SQL queries.
• Ingests data from Kinesis Streams and Firehose, with output options to S3, RedShift, Elasticsearch, and Kinesis Data Streams.
• Composed of three application components with configurable destinations for results persistence.

Kinesis Client Library (KCL)

• Java library for reading records from Kinesis Streams in distributed applications.
• Acts as an intermediary, ensuring one record processor per shard while managing multiple shards per worker.
• Checkpoints progress in DynamoDB and supports deployments on EC2 and Elastic Beanstalk.

Security Features

• Access control managed through IAM policies, with encryption in transit using HTTPS.
• Data at rest is encrypted using KMS, with options for client-side encryption.
• VPC endpoints are available for secure Kinesis access within a Virtual Private Cloud.

Key Comparisons: SQS, SNS, and Kinesis

• Each service is optimized for different messaging and streaming use cases, with Kinesis focusing on real-time data streams.