Amazon Kinesis Data Streams

A durable stream is an ordered set of records split across shards

Amazon Kinesis Data Streams is built for collecting and processing records that arrive continuously, such as application logs, clickstreams, telemetry, and market events. Producers write records to a named stream, and consumers read them independently. Unlike a queue whose message is normally deleted after successful processing, a stream retains records for its configured retention period so different consumers can process the same history at their own position.

The core unit of parallelism is the shard. Every record includes a partition key, and Kinesis maps that key through a hash-key space to a shard. Records with the same partition key therefore follow the same routing and can preserve per-key ordering. A poor partition-key strategy can concentrate traffic into a small portion of the stream even when overall capacity appears sufficient, so evenly distributed, high-cardinality keys are a recurring design consideration.

Consumers choose shared reads or dedicated enhanced fan-out

Kinesis assigns a sequence number after a record is written. Consumers use shard iterators or higher-level libraries such as the Kinesis Client Library to move through records. The KCL coordinates multiple workers and redistributes shard-processing assignments as workers or shards change, which is useful for scaling a consumer application without manually assigning each shard.

Standard consumers share a shard's read-throughput path. Enhanced fan-out registers consumers and gives each registered consumer a dedicated read-throughput path per shard, reducing contention when several low-latency consumers must process the same stream. This is a consumer-side decision; it does not change how producers partition records into shards.

• Partition keys affect both load distribution and ordering scope.
• Multiple applications can read the same retained stream independently.
• Enhanced fan-out is useful when consumers should not compete for the same shard read throughput.
• KCL coordinates shard ownership among workers and adapts to resharding.

Capacity mode and resharding determine how the stream scales

Kinesis supports capacity modes that shift how explicitly you manage stream capacity. In provisioned mode, shard capacity is a direct design input and resharding operations such as splitting or merging shards change parallel capacity. On-demand modes automate more of this scaling behavior. Regardless of the mode, partition-key distribution remains important because individual keys still map into the stream's partitioning model.

Resharding changes the hash-key ranges owned by shards. A split divides a parent shard's range between child shards; merging combines adjacent shard ranges. Consumers must be able to follow the resulting shard lineage, which KCL is designed to handle. In exam scenarios, select Kinesis when the requirement is a replayable real-time stream with multiple independent consumers, not merely asynchronous task buffering between two components.