Kafka vs RabbitMQ vs SQS: Message Broker Guide

Q: What is the difference between Kafka and RabbitMQ?

They implement fundamentally different semantics, which matters more than any feature comparison. Kafka is a distributed, partitioned, replayable log: producers append events, the log retains them for a configured period (days, or forever), and consumers read at their own pace by tracking an offset — consuming does not remove anything, so five independent systems can read the same stream, a service can re-read last week after a bug fix, and a new consumer can bootstrap from history. Ordering is guaranteed within a partition, and throughput scales to millions of messages per second by adding partitions. RabbitMQ is a message broker in the classic sense: producers send messages through exchanges that route them to queues, one consumer receives each message, acknowledges it, and it is gone — the broker's intelligence lives in routing (topic patterns, headers, priorities, TTLs, dead-lettering) and in per-message delivery guarantees, with sub-millisecond to low-millisecond latency at tens of thousands of messages per second. The selection question is therefore about your data's nature: if messages are events — facts about what happened that multiple systems care about, possibly retrospectively — you want Kafka's log. If messages are tasks — units of work exactly one worker should perform — you want a queue, and RabbitMQ is the richest self-hosted one.

Q: When is Kafka overkill?

Whenever the workload is a to-do list rather than a shared history — which describes most workloads at most companies. The concrete signals: a single consumer type processes each message (no second team reads the same stream), messages are commands or jobs rather than facts (send this email, resize this image), nobody has ever asked to replay history, volume sits in the hundreds-to-thousands per second rather than hundreds of thousands, and no CDC, event-sourcing, or stream-processing ambitions exist on the roadmap. In that shape, Kafka's costs buy nothing: you operate (or pay for) a distributed cluster with partitions, consumer-group rebalancing, retention tuning, and upgrade choreography, while fighting its semantics — per-message acknowledgment and selective retry, which queues give natively, must be simulated awkwardly on top of offsets. A team running a Kafka cluster to deliver a few hundred emails an hour is the canonical cautionary tale. The inversion is equally real though: teams that pipe genuinely shared events through point-to-point queues end up with six services each maintaining duplicate queue subscriptions and no way to replay when one of them ships a bug. The rule: name each workload log or queue first; the honest answer for many organizations is a small managed Kafka for the few true event streams, and boring queues for everything else.

Q: What are Kafka partitions and why does the partition key matter?

A Kafka topic is split into partitions — independent append-only logs that are the unit of both parallelism and ordering. Producers assign each event to a partition, normally by hashing a partition key you choose; consumers in a group divide partitions among themselves, so throughput scales by adding partitions and consumers. The two guarantees and their coupling: Kafka preserves order only within a partition, and all events with the same key land in the same partition — so choosing the key is choosing what stays ordered. Key by order ID and every event for a given order (created, paid, shipped) arrives in sequence to whoever consumes it; key randomly and events for the same entity scatter across partitions, arriving in any interleaving — a correctness bug that surfaces as intermittent, unreproducible state corruption downstream. The opposite failure is the hot key: keying a B2B platform's events by tenant when one tenant generates 60 percent of traffic sends 60 percent of load to one partition, capping your parallelism at one consumer regardless of cluster size. Design guidance: key by the entity whose event sequence must be causally consistent (order, user, device), verify the key's cardinality and skew against real traffic, and size partition counts for target parallelism with headroom — repartitioning later changes key-to-partition mapping and is operationally disruptive.

Q: Is SQS good enough for most applications?

For teams on AWS whose workloads are queue-shaped, yes — and choosing it by default until requirements prove otherwise is sound engineering, not laziness. What SQS delivers for effectively zero operational effort: unlimited throughput on standard queues, at-least-once delivery with visibility timeouts, dead-letter queues as a checkbox, encryption and IAM-native access control, seamless Lambda integration for serverless consumers, and pricing ($0.40 per million requests after the free tier) that rounds to nothing at moderate volume. Paired with SNS for fan-out, it covers pub/sub-style distribution to multiple queue subscribers. The genuine limits that push you elsewhere: no replay — a consumed message is gone, so shared event history is not what SQS provides; FIFO queues cap at 300 messages per second (3,000 batched), so strict ordering at high volume is a squeeze; latency is tens of milliseconds plus polling, not RabbitMQ's sub-millisecond push; routing is primitive compared to AMQP exchanges — fan-out and filtering live in SNS subscription policies, which get unwieldy past moderate complexity; and cross-cloud portability is nil. The decision shape: SQS when a queue is what you need and AWS is home; RabbitMQ when routing sophistication or latency matters; Kafka when the workload is actually a log. Many mature AWS architectures run SQS for 90 percent of messaging and a small managed Kafka for the streams that earn it.

Q: Why must message consumers be idempotent?

Because every broker in this comparison — Kafka, RabbitMQ, SQS — guarantees at-least-once delivery in practice, which means duplicate processing is a certainty over time, not a tail risk. The duplicates arrive through unavoidable mechanics: a consumer processes a message but crashes before committing the offset or sending the acknowledgment, so the broker re-delivers; a visibility timeout expires mid-processing and a second worker claims the same message; a network partition eats an ack; a consumer-group rebalance replays a batch. If the side effect of processing is "charge the customer" or "decrement inventory," running twice is an incident with a dollar amount. Idempotency converts duplicates into no-ops: track processed message IDs (or business-level event IDs) and skip already-seen ones — a unique constraint in the database is the sturdiest implementation; use provider idempotency keys on external calls (payment APIs support them universally); and prefer absolute state assignments (set status to shipped) over relative mutations (increment counter) where the domain allows. Kafka adds an option worth knowing: exactly-once semantics within Kafka-to-Kafka stream processing via transactions — but the moment a consumer touches an external system (your database, an email API), you are back to at-least-once at that boundary, and consumer idempotency remains the load-bearing defense.

Q: What role does a message broker play in AI and ML pipelines?

The 2026 AI platform is substantially event-driven, and the log-versus-queue split maps cleanly onto its two traffic types. Log-shaped (Kafka territory): feature pipelines stream user behaviour events into online feature stores so models score on fresh signals; LLM interaction logs — every prompt, retrieved context, tool call, and response — flow into a retained stream feeding evaluation harnesses, fine-tuning dataset curation, and cost analytics, where replay is precisely the requirement ("re-run last month's conversations through the new evaluation" is a consumer reading old offsets); agent audit trails demand an immutable, ordered, retained record of every action an autonomous system took, which is a compacted Kafka topic wearing a compliance hat; and CDC streams database changes into search indexes, caches, and vector stores without dual-write fragility. Queue-shaped (SQS/RabbitMQ territory): the jobs those streams trigger — transcribe this recording, embed this document batch, enrich this record with an LLM call — are classic tasks wanting per-message retry, dead-lettering, and concurrency caps tuned to GPU or API-rate budgets. Real platforms run both, wired together: an event on the log triggers enqueueing of jobs onto queues, workers process and emit completion events back onto the log. Forcing either traffic type onto the other's substrate produces the familiar pathologies — unreplayable training data, or a distributed log babysitting a to-do list.

Q: Should I self-host Kafka or use a managed service?

Default to managed — MSK on AWS, Confluent Cloud, or Redpanda's managed offering — unless you have a platform team with streaming on its charter. Self-hosted Kafka is real distributed-systems operations: broker sizing and disk management (retention is a storage bill that grows silently), partition rebalancing behaviour under broker failure, consumer-group rebalance storms, security configuration (TLS, SASL, ACLs), version upgrades across a live cluster, and the KRaft migration story for older deployments. None of it is exotic to a team that runs distributed infrastructure professionally; all of it is a part-time job that ambushes teams who adopted Kafka for one integration. Managed offerings convert that to a bill — entry points in the low hundreds of dollars monthly — and the arithmetic almost always favours them below large scale: one engineer-day per month of cluster care exceeds the managed premium. Redpanda deserves mention as a Kafka-API-compatible reimplementation (C++, no JVM/ZooKeeper) that is meaningfully lighter to self-host if you must own the infrastructure. The decision inputs: existing platform-team capacity, data-residency constraints (managed means your events transit a vendor's control plane — usually fine, occasionally disqualifying), and honest volume projections. Under roughly a gigabyte per second of sustained throughput, the managed premium is noise next to the engineering time it frees.

Q: Can I replace RabbitMQ or Kafka with Redis or Postgres?

At modest scale, genuinely yes — and starting there is often right. Redis Streams provides consumer groups, acknowledgments, and replay-within-retention over a data store you likely already run; paired with its pub/sub for fire-and-forget fan-out, it covers a surprising share of messaging needs at tens of thousands of messages per second, with the caveats that durability depends on your persistence configuration and retention is memory-bounded. Postgres-backed queues (covered in depth in our task-queue guide) go further for job traffic: SELECT FOR UPDATE SKIP LOCKED gives correct concurrent claiming, and enqueueing in the same transaction as your business writes eliminates the dual-write problem entirely — an outbox for free. LISTEN/NOTIFY adds lightweight eventing. What these substitutes do not give you: Kafka-class retention and replay over terabytes, partition-parallel throughput in the hundreds of thousands per second, the Connect/Debezium CDC ecosystem, or RabbitMQ's routing sophistication and protocol breadth (AMQP, MQTT). The honest sequence for a growing system: Postgres queue while volume is a stream, Redis Streams when you need lighter-weight fan-out and higher throughput, a dedicated broker when the workload names itself — true event log with multiple independent readers and replay (Kafka), or routing-heavy task distribution at scale (RabbitMQ). Migrating a well-abstracted producer/consumer layer later is days, not months; the abstraction is the investment that matters early.