Mehdi Akiki
Published on

The 4 Real Reasons Kafka Is Fast

Authors
  • Mehdi Akiki avatar
    Name
    Mehdi Akiki
    Twitter

Reference · Through the layers

"Kafka is fast" is correct but too abstract. Here is the concrete version.

The 4 real reasons Kafka is fast

1. It mostly appends at the end of a file

Not "insert somewhere." Not "update rows." Not "rebalance a tree."

Just:

current file end -> write bytes -> new file end

So if the log file is:

[msg0][msg1][msg2]

and a producer sends 3 more messages, Kafka does:

[msg0][msg1][msg2][msg3][msg4][msg5]

That is cheap because the broker does not need to search for where to put the data.

2. It batches many messages together

This is a huge one.

Instead of doing:

write(msg1)
write(msg2)
write(msg3)
write(msg4)
...

Kafka tries to do more like:

write(big_chunk_of_many_messages)

Concrete example:

  • 1000 messages
  • each message = 1 KB

Bad path:

  • 1000 separate tiny writes

Better path:

  • 1 write of ~1 MB

Why is 1 big write faster?

Because every write has overhead:

  • syscall overhead
  • kernel work
  • bookkeeping
  • possible flush behavior
  • CPU overhead

So 1000 small writes are much more expensive than 1 big write.

That is one of the main reasons Kafka gets high throughput.

3. Consumers often read from RAM, not disk

This is the page cache part.

When Kafka writes data to a file, Linux usually keeps that data in memory too, inside the OS page cache.

So producer writes a message, then a consumer asks for it shortly after.

Very often the broker does not go to the physical disk again. It gets the bytes from memory.

So in practice the path is often:

producer -> broker -> OS page cache (RAM) -> consumer

not:

producer -> disk platter/SSD -> broker -> disk -> consumer

This is why people who say "Kafka is fast because disk is fast" are missing it.

The real answer is more like:

Kafka writes like a log, and the OS turns a lot of that into memory-speed reads.

4. It tracks almost no per-message state for consumers

This is the metadata point.

A slow system often does too much bookkeeping.

Kafka is lighter.

For a consumer group, the important thing is mostly:

partition 0 -> next offset = 120845
partition 1 -> next offset = 9981
partition 2 -> next offset = 551002

That is tiny.

Kafka does not need to mark every message as:

  • delivered
  • acknowledged
  • deleted
  • removed from the queue

Instead, the message stays in the log, and the consumer just says:

my next record to read is offset 120845

That is much cheaper.

The low-copy part, concretely

Without an optimized path, data movement can look like this:

disk/page cache -> kernel buffer -> user-space broker process -> kernel socket buffer -> NIC

That means more copying.

Kafka tries to avoid unnecessary copying, so the bytes can go more directly from the OS file cache to the network socket.

Less copying means:

  • less CPU
  • less memory bandwidth wasted
  • higher throughput

So "low-copy" really means:

the broker spends less time acting like a dumb middleman moving the same bytes around.

The easiest way to remember it

Kafka is fast because it does 4 cheap things:

  1. append
  2. batch
  3. read from cache
  4. store only offsets for consumers

That is the practical memory hook.

Very concrete comparison

Traditional DB-like mindset

"Insert message record, update indexes, maybe ack it later, maybe delete or mark processed."

That means more work:

  • index maintenance
  • row metadata
  • random access patterns
  • more writes than just the payload

Kafka mindset

"Put bytes at the end of the log. Consumer remembers a number."

That is much simpler.

One full example

Say 10,000 sensor events arrive.

Kafka-like path:

  1. broker collects many events into a batch
  2. appends batch to end of log file
  3. OS keeps recent file pages in RAM
  4. consumer fetches starting at offset 500000
  5. broker streams bytes from cached log pages
  6. consumer commits 510000

Notice what did not happen:

  • no delete per message
  • no row update per message
  • no complex query
  • no searching for "where should I insert this"
  • no heavy per-message ack bookkeeping on the broker

That is why it is fast.

Even shorter version

Kafka is fast because it treats messages like a video stream written to the end of a tape, not like rows constantly being inserted, updated, and deleted in a general-purpose database.