Do you need Kafka?

Rob Galanakis on March 9, 2023

kafkaesque maze

I'm not saying you can use WebhookDB instead of Kafka...

...but you (probably) can use WebhookDB instead of Kafka.

Giorgio Touskalos, the ancient aliens guy

Apache Kafka is a powerful technology that is all about ingesting and directing massive streams of events between nearly any type of data sources. It's flexible and powerful and, at large enterprises, almost everywhere (they claim 80% of Fortune 100 companies use Kafka).

If you squint, a whole lot of problems fall into this domain, which is one reason Kafka is so popular.

Unfortunately, Kafka is notoriously difficult to run, and despite the flexibility of what it can be used for, can't really be used by itself for very much.

That's sort of the point: Kafka connects together the parts of your stack that are actually valuable. In most cases that means the applications and services generating events, and the data warehouse that collects them.

Do you actually need Kafka?

Kafka is, for most use cases, purely added complexity. It's a bear to operate and integrate with and can be a big distraction from the work most application and product companies are doing.

If your Kafka pipeline is the common case — sending events from your applications to a warehouse — what if I told you commodity databases like PostgreSQL eliminate the need for Kafka?

If you're handling a sustained load of less than, say, 20,000 events per second, you can almost definitely run your pipeline through WebhookDB. It's not quite the same conceptually, since WebhookDB is not usually dealing with events (though it can). Instead, it's writing the events safely to Postgres (and soon, DynamoDB), and from there changes can be efficiently staged and sent to your data warehouse.

WebhookDB gives you nearly all the benefits of Kafka for the common use case, without the headaches and complexity.

If you're thinking of scratching an itch with Kafka, you should take a look at WebhookDB. Give WebhookDB a try or let us know what you think.

Recent Blog Posts

open lock hanging from a door
Securing API keys

May 11, 2023

Learn about securing API keys when using React or building any client software.

Read More →
checklist written in a notebook
API Integration Checklist

May 4, 2023

Ensure a smooth launch of your integrations with 3rd party APIs using our checklist.

Read More →
water falling into tiers of pools
New Pricing Structure

April 17, 2023

Learn about our new transparent and affordable pricing tiers.

Read More →
protesters in the occupy wall street movement
Building for the API 99%

April 10, 2023

Most APIs we use every day are not your Stripes or Shopifys. These are the API 99%.

Read More →