Do you need Kafka?

Rob Galanakis on March 9, 2023

kafkaesque maze

I'm not saying you can use WebhookDB instead of Kafka...

...but you (probably) can use WebhookDB instead of Kafka.

Giorgio Touskalos, the ancient aliens guy

Apache Kafka is a powerful technology that is all about ingesting and directing massive streams of events between nearly any type of data sources. It's flexible and powerful and, at large enterprises, almost everywhere (they claim 80% of Fortune 100 companies use Kafka).

If you squint, a whole lot of problems fall into this domain, which is one reason Kafka is so popular.

Unfortunately, Kafka is notoriously difficult to run, and despite the flexibility of what it can be used for, can't really be used by itself for very much.

That's sort of the point: Kafka connects together the parts of your stack that are actually valuable. In most cases that means the applications and services generating events, and the data warehouse that collects them.

Do you actually need Kafka?

Kafka is, for most use cases, purely added complexity. It's a bear to operate and integrate with and can be a big distraction from the work most application and product companies are doing.

If your Kafka pipeline is the common case — sending events from your applications to a warehouse — what if I told you commodity databases like PostgreSQL eliminate the need for Kafka?

If you're handling a sustained load of less than, say, 20,000 events per second, you can almost definitely run your pipeline through WebhookDB. It's not quite the same conceptually, since WebhookDB is not usually dealing with events (though it can). Instead, it's writing the events safely to Postgres (and soon, DynamoDB), and from there changes can be efficiently staged and sent to your data warehouse.

WebhookDB gives you nearly all the benefits of Kafka for the common use case, without the headaches and complexity.

If you're thinking of scratching an itch with Kafka, you should take a look at WebhookDB. Give WebhookDB a try or let us know what you think.

Recent Blog Posts

AI-generated image of balloons and a computer
WebhookDB is Open Source

March 11, 2024

We're aligning our business with our values and community and going Open Source,

Read More →
zoomed in artificial snowflakes, each unique
Every API is Unique

June 8, 2023

Just like people, every API is unique in its own special way.

Read More →
webhookdb hook logo wearing angel wings
WebhookDB Gives You Wings!

June 1, 2023

Answer any question instantaneously, instead of drowning in documentation and tools.

Read More →
programmers reading code behind two doors, one with more cursing than the other, correlating code quality and cursing
Why would they do that!

May 24, 2023

Or, how to stop worrying and learn to love every API.

Read More →