# Kafka vs Kinesis: Shards, Retention, MSK

**Apache Kafka** is an open-source distributed log built around topics and partitions. **Amazon Kinesis Data Streams** is an AWS-managed service built around streams and shards. Both store ordered, replayable event streams. A shard has fixed limits (1 MB/s in, 2 MB/s out) and keeps data at most 365 days; a partition is limited by its broker and its storage.

![Kafka vs Kinesis: a Kinesis stream is split into shards, each with a fixed 1 MB/s in and 2 MB/s out, on servers AWS runs, with retention up to 365 days. A Kafka topic is split into partitions, each led by a broker, with no fixed rate per partition, replicas on other brokers, and retention set per topic. On both, a record key hashes to one shard or partition and order is kept per key.](https://www.conduktor.io/assets/images/glossary/kafka-vs-kinesis-2.webp)

## Kafka vs Kinesis terms: topic is stream, partition is shard

| Concept | Apache Kafka | Amazon Kinesis Data Streams |
|---|---|---|
| Container | Topic | Stream |
| Unit of parallelism | [Partition](https://www.conduktor.io/glossary/kafka-partitions-explained) (count chosen by the operator, bounded by broker hardware) | Shard (fixed 1 MB/s or 1,000 records/s in, 2 MB/s out) |
| Routing key | Message key, hashed by the producer's partitioner | Partition key, MD5-hashed to a 128-bit hash key range |
| Position | Offset, per partition | Sequence number, per shard |
| Reader identity | [Consumer group](https://www.conduktor.io/glossary/kafka-consumer-groups-explained), offsets stored in `__consumer_offsets` | Application name (KCL), checkpoints stored in a DynamoDB lease table |
| Dedicated read throughput | No per-group quota; limited by broker egress | Enhanced fan-out, 2 MB/s per shard per registered consumer |
| Retention | Time or size, default 7 days, unbounded with tiered storage | 24 hours by default, up to 365 days |
| Max record | `message.max.bytes`, default about 1 MB per batch, configurable | 10 MiB per record, large records draw on burst capacity |
| Delivery | At-least-once, idempotent producer on by default, transactions | At-least-once, duplicates handled by the application |
| Protocol | Open Kafka wire protocol, many implementations | AWS-specific API |

Kafka can run on-premises, on any cloud, or as a managed service such as Amazon MSK. Its clients all use the Kafka protocol. Kinesis is AWS-specific: producers use `PutRecord` or `PutRecords`, and consumers use `GetRecords`, enhanced fan-out, Lambda, or the Kinesis Client Library. AWS runs the service and its replication.

## Throughput: 1 MB/s per shard, no fixed limit per partition

Each Kinesis shard accepts up to 1 MB/s or 1,000 records/s and serves up to 2 MB/s. In provisioned mode, the team chooses the shard count. In on-demand mode, AWS manages the shards: as of October 2026, a new stream starts at 4 MB/s of write capacity and absorbs up to double its peak write throughput of the previous 30 days. Traffic that more than doubles within 15 minutes can be throttled, and the On-demand Advantage account setting adds configurable warm throughput for expected peaks.

A Kafka partition has no built-in throughput quota. Its limit comes from the leader broker's disk, network, and CPU, shared with other partitions on that broker.

- **Kinesis splits and merges shards.** A split closes one parent shard and creates two children. Consumers must finish the parent before reading its children to preserve order.
- **Kafka adds partitions.** Existing records do not move, but the new partition count changes key-to-partition mapping for future records. Kafka cannot reduce the partition count.

![Kinesis shard split with parent and child lineage next to Kafka partition expansion where existing keys are rerouted](https://www.conduktor.io/assets/images/glossary/kafka-vs-kinesis-0.webp)

On both, every record with the same key goes to one shard or one partition, so a hot key is capped at that one shard or partition. The fix is a better key or custom routing.

## Retention and replay: up to 365 days on Kinesis, no limit on Kafka

Kinesis keeps records for 24 hours by default and supports up to 365 days. Longer retention is configured per stream and adds storage and retrieval charges.

Kafka's broker default is `log.retention.hours=168` (seven days) with no size cap (`log.retention.bytes=-1`), and both can be set per topic to any value, including infinite. The same seven-day window on each system:

```bash
# Kinesis: stream-level, billed as extended retention above 24 h
aws kinesis increase-stream-retention-period \
  --stream-name orders --retention-period-hours 168

# Kafka: topic-level, bounded only by disk or tiered storage
kafka-configs.sh --bootstrap-server broker-1:9092 --alter \
  --entity-type topics --entity-name orders \
  --add-config retention.ms=604800000
```

Broker storage is the practical Kafka limit. [Tiered storage](https://www.conduktor.io/glossary/tiered-storage-in-kafka) moves older segments to object storage, while log compaction can keep the latest value per key. Kinesis has no equivalent to log compaction.

A Kinesis consumer requests a shard iterator at `TRIM_HORIZON`, `AT_TIMESTAMP`, or a sequence number; a Kafka consumer seeks to an offset or a timestamp. Kinesis replays at most 365 days back, billed as extended (up to 7 days) or long-term retention storage. Kafka replays as far back as retention reaches, which with tiered storage can be the topic's full history.

## Consumers: a shared 2 MB/s per shard versus consumer groups

Standard Kinesis consumers poll with `GetRecords` and share each shard's read capacity: 2 MB/s and five `GetRecords` calls per second per shard, split across every polling consumer. AWS's capacity guidance recommends one polling consumer per stream and enhanced fan-out when more applications need the data.

Enhanced fan-out gives each registered consumer dedicated 2 MB/s per shard, pushed over HTTP/2 with `SubscribeToShard`. As of October 2026, a stream accepts up to 20 registered consumers, or 50 with On-demand Advantage. In provisioned mode it adds a consumer-shard-hour and per-GB charge; on-demand bills it per GB, with no premium under On-demand Advantage. AWS documents an average propagation delay of about 200 ms for one polling consumer, rising to about 1,000 ms with five, versus about 70 ms with enhanced fan-out.

![Kinesis shared GetRecords polling and enhanced fan-out compared with Kafka consumer groups reading a topic](https://www.conduktor.io/assets/images/glossary/kafka-vs-kinesis-1.webp)

Kafka has no per-group fee or call quota. Each consumer group keeps its own offsets, so several teams can read the same topic independently. The cluster still pays the network and broker cost, and the operator must manage rebalancing and lag.

The Kinesis Client Library handles leases, checkpoints, and shard ordering, storing its coordination state in DynamoDB. Lambda can also consume a stream directly through an event source mapping.

## Delivery guarantees: idempotence and transactions versus retry and deduplicate

Kafka producers use idempotence by default to deduplicate retries. Transactions can atomically write to several partitions and commit consumed offsets, which provides [exactly-once semantics](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) inside Kafka-to-Kafka pipelines.

Kinesis is at-least-once. A timed-out `PutRecord` retry can create two records with different sequence numbers, and a `PutRecords` batch can partly fail. Applications must retry failed entries and deduplicate at the sink. Idempotent consumers help on both. Only Kafka can commit produced records and consumed offsets in one transaction.

## Ecosystem and portability: Kafka protocol vs AWS API

Kafka has a public wire protocol supported by [Kafka Connect](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines), Kafka Streams, Apache Flink, [Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium), and many managed services. Produce and consume code moves between Kafka providers unchanged; authentication and authorization settings often do not. MSK's IAM access control needs AWS's client-side auth library and IAM policies, and MSK Serverless requires it, without Kafka ACLs. A team that wants a later exit can run provisioned MSK with SASL/SCRAM or mTLS and Kafka ACLs, at the cost of a second identity system next to IAM. Either way, moving providers still means replicating the data.

Kinesis uses an AWS-specific API and integrates closely with Lambda, Firehose, Managed Service for Apache Flink, Glue Schema Registry, and other AWS services. Those integrations reduce setup inside AWS, but leaving Kinesis means changing producers, consumers, and checkpoints.

Cross-VPC and cross-account access is simpler on Kinesis. A Kinesis client calls one regional HTTPS endpoint, reachable privately through an interface VPC endpoint, and another account gets access through a resource-based policy on the stream. A Kafka client bootstraps once, then connects to every broker the metadata returns, so each broker address must be resolvable and routable from the client's network. On MSK, multi-VPC private connectivity automates this over PrivateLink within one Region, with a per-GB processing charge; across Regions, clouds, or on-premises networks, the per-broker routing remains the team's job. The [MSK cross-VPC connectivity walkthrough](https://www.conduktor.io/blog/kafka-msk-cross-network-connectivity) shows where the bootstrap succeeds and the broker connections time out.

## Kafka vs Kinesis pricing: what each one bills

- **Kinesis provisioned** charges shard-hours, PUT payload units, extended or long-term retention, and enhanced fan-out per consumer-shard-hour plus per GB.
- **Kinesis On-demand Standard** charges a per-stream hourly fee plus per GB written and read.
- **Kinesis On-demand Advantage** drops the per-stream fee and the enhanced fan-out premium and lowers per-GB rates, in exchange for an account-wide minimum of 25 MiB/s ingest and 25 MiB/s retrieval across on-demand streams (as of October 2026).
- **Amazon MSK** charges broker-hours and storage on provisioned clusters, or cluster-hours, partition-hours, and GB on Serverless. Replication between brokers is not billed; client traffic pays standard AWS data transfer.
- **Self-managed Kafka** pays for instances and disks, plus cross-AZ transfer for replication: with a replication factor of 3 across zones, each produced byte is stored three times and [crosses zones at least twice](https://www.conduktor.io/blog/8-new-ways-to-drastically-reduce-your-kafka-costs) before any consumer reads it. Kinesis includes multi-AZ replication in its price.

Kinesis provisioned bills every shard-hour, and MSK Serverless bills every partition-hour. On provisioned Kafka, the total number of partition replicas per broker, more than throughput, often sets the broker count, and MSK publishes a recommended maximum per broker size. Kinesis can merge shards back down; Kafka cannot reduce a topic's partition count, so an over-partitioned topic has to be migrated to a smaller one. Unused partitions ([partition waste](https://www.conduktor.io/blog/the-surprising-cost-of-kafka-partition-waste)) still count toward partition-hours and broker count.

Small and bursty AWS-only workloads often fit Kinesis on-demand because there is no broker fleet to size. Kafka tends to cost less when many applications read the same data, retention is long, or the organization already runs Kafka. Price both with current regional rates, expected traffic, retention, and number of readers.

## Amazon MSK: Kafka run by AWS

[Amazon MSK](https://www.conduktor.io/glossary/amazon-msk-managed-kafka-on-aws) is Kafka operated by AWS: AWS provisions, patches, and replaces the brokers, and clients use the standard Kafka protocol. MSK has three cluster types:

- **Standard brokers** use provisioned compute and storage, with optional tiered storage.
- **Express brokers** separate compute from elastic storage and speed up scaling and recovery.
- **MSK Serverless** has no brokers to size and requires IAM authentication.

MSK Serverless is the MSK option closest to Kinesis on-demand: billed per cluster-hour, partition-hour, and GB. As of October 2026, each cluster is capped at 200 MBps in and 400 MBps out, each partition at 5 MBps in, and messages at 8 MiB. Standard Kafka clients, Connect, and Flink work against it.

## Access control and audit on each

Kinesis uses IAM policies, KMS encryption, and CloudTrail audit. Kafka has ACLs, MSK adds IAM access control, and other controls depend on the provider and tooling. [Conduktor](https://www.conduktor.io/kafka-data-governance) works with Kafka-protocol clusters, including MSK, to add role-based access, field-level controls, and audit. It does not manage Kinesis, so teams using both keep two governance models.

## Migrating from Kinesis to Kafka (or MSK)

- **Map shards to partitions** and keep the same partition key, so per-key ordering survives the move.
- **Checkpoints do not carry over.** KCL state lives in DynamoDB; Kafka consumers start from an offset looked up by timestamp.
- **Run both in parallel**, by dual-writing or by copying Kinesis into Kafka with a source connector, then cut consumers over one at a time.
- **Lambda consumers** can switch event sources: Lambda supports both Kinesis and MSK event source mappings.
- **Replace partial-failure handling.** Per-entry `PutRecords` retries give way to the Kafka producer's retries and idempotence.
- **Check record sizes.** Kinesis accepts 10 MiB records; Kafka defaults to about 1 MB and MSK Serverless caps messages at 8 MiB.

## When to choose Kinesis vs Kafka

Choose Kinesis Data Streams when:

- The workload is AWS-only and consumers are Lambda functions, Firehose deliveries, or Managed Flink jobs.
- The team wants an AWS-native service without broker operations.
- Throughput fits in shard arithmetic and independent readers stay within enhanced fan-out's 20 registered consumers (50 with On-demand Advantage, as of October 2026).
- Retention needs fit within one year.

Choose Kafka (self-managed, MSK, or another provider) when:

- Many teams or applications read the same topics and per-consumer metering would dominate the bill.
- Retention is long, log compaction is needed, or events must remain replayable for years.
- Exactly-once processing across topics, Kafka Streams, or Connect connectors are part of the design.
- Portability across clouds, on-premises, or vendors matters, or the organization already runs Kafka elsewhere.

**Is a Kinesis shard the same as a Kafka partition?**

Mostly. Both are ordered sub-logs selected by hashing a key, and one key's throughput is capped by the one shard or partition it lands on. A shard is a fixed unit (1 MB/s in, 2 MB/s out), bought by the hour in provisioned mode, while a partition's capacity depends on the broker it lives on and carries no quota of its own.

**Which is cheaper, Kafka or Kinesis?**

Kinesis on-demand often fits small, bursty, AWS-only streams because there is no broker fleet to size. Kafka can cost less as fan-out and retention grow because it does not charge per consumer group, although Kinesis On-demand Advantage removes the per-stream fee and the enhanced fan-out premium for accounts that meet its minimum usage commitment. Price both with current regional rates, your traffic, retention, and number of readers.

**Can Kinesis replay data like Kafka?**

Yes, within its retention window. You request a shard iterator at a timestamp, a sequence number, or the oldest record, exactly as you would seek a Kafka offset. The window is 24 hours by default and at most 365 days, billed as extended or long-term retention storage, whereas Kafka retention is bounded only by disk or object storage.

**What is the difference between Kinesis and Amazon MSK?**

Kinesis is an AWS-specific API with its own SDKs and consumer library. MSK is Apache Kafka operated by AWS, so standard Kafka clients, Kafka Connect connectors, and Flink jobs work against it, and the data can move to another Kafka provider. Authentication can stay AWS-specific: MSK Serverless requires IAM access control, while provisioned MSK also supports SASL/SCRAM and mTLS. MSK Serverless is the closest to Kinesis: no brokers to size.

**Does Kinesis support exactly-once delivery?**

No. Kinesis is at-least-once: producer retries create duplicates with new sequence numbers, PutRecords can partially fail, and consumer restarts replay from the last checkpoint. AWS recommends embedding a primary key and making sinks idempotent. Kafka's idempotent producer and transactions give exactly-once inside Kafka-to-Kafka pipelines.

**How does enhanced fan-out compare to Kafka consumer groups?**

Both let several applications read the same data independently. Enhanced fan-out gives each registered consumer a dedicated 2 MB/s per shard over HTTP/2 push, up to 20 consumers per stream (50 with On-demand Advantage, as of October 2026). In provisioned mode it adds a consumer-shard-hour and per-GB charge; on-demand bills it per GB only. Kafka consumer groups pull at their own offsets with no per-group quota or fee; the limit is broker egress.

## Related Pages

- [Amazon MSK: Managed Kafka on AWS](https://www.conduktor.io/glossary/amazon-msk-managed-kafka-on-aws): Standard, Express, and Serverless clusters in depth.
- [Kafka Partitions Explained](https://www.conduktor.io/glossary/kafka-partitions-explained): ordering, parallelism, and choosing a partition count.
- [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained): offsets, rebalancing, and fan-out on Kafka.
- [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka): how KIP-405 decouples retention from broker disks.
- [Cross-AZ Traffic in Streaming](https://www.conduktor.io/glossary/cross-az-traffic-streaming): the transfer line Kinesis hides and Kafka exposes.
- [Kafka vs Pulsar](https://www.conduktor.io/glossary/kafka-vs-pulsar) and [Kafka vs RabbitMQ](https://www.conduktor.io/glossary/kafka-vs-rabbitmq): sibling comparisons in the series.
- [Kafka Alternatives](https://www.conduktor.io/compare/kafka-alternatives): the wider field of Kafka-protocol and non-Kafka options.

## Sources and References

- [Amazon Kinesis Data Streams: Quotas and limits](https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html)
- [Amazon Kinesis Data Streams: Choose the right mode to stream in](https://docs.aws.amazon.com/streams/latest/dev/how-do-i-size-a-stream.html)
- [Amazon Kinesis Data Streams: Develop enhanced fan-out consumers](https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html)
- [Amazon Kinesis Data Streams: Complete the resharding action](https://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-after-resharding.html)
- [Amazon Kinesis Data Streams: Handle duplicate records](https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-duplicates.html)
- [Amazon Kinesis Data Streams: PutRecords API reference](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecords.html)
- [Amazon Kinesis Data Streams pricing](https://aws.amazon.com/kinesis/data-streams/pricing/)
- [Amazon MSK: Supported Apache Kafka versions](https://docs.aws.amazon.com/msk/latest/developerguide/supported-kafka-versions.html)
- [Amazon MSK: Express brokers](https://docs.aws.amazon.com/msk/latest/developerguide/msk-broker-types-express.html)
- [Amazon MSK: Quota](https://docs.aws.amazon.com/msk/latest/developerguide/limits.html)
- [Amazon MSK pricing](https://aws.amazon.com/msk/pricing/)
- [Apache Kafka documentation: message delivery semantics, modifying topics, tiered storage](https://kafka.apache.org/documentation/)
- [Apache Kafka broker configuration reference](https://kafka.apache.org/43/generated/kafka_config.html)
- [KIP-405: Kafka Tiered Storage](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage)