Building Event Driven Architectures
with Apache Kafka on Heroku

Technical session

Apache Kafka is the backbone for building architectures that deal with billions of events a day. Chris Castle, developer advocate, will show you where it might fit in your roadmap.

See a live technical demo showing you how to quickly get started with Kafka on Heroku.

Hear questions answered in a live Q&A with Rand Fitzpatrick from the Heroku product team.

Watch session

00:0000:00

Want to learn more about Apache Kafka on Heroku? Check out the docs.

Session slides

First slide of the presentation, titled Event Driven Architectures with Apache Kafka on Heroku
Kafka on Heroku
Kafka on Heroku is a durable, distributed messaging service for stream processing, optimized for developers

You’ll learn:

  • What Apache Kafka is and how to use it on Heroku
  • How Kafka enables you to model your data as immutable streams of events, introducing greater parallelism into your applications
  • How you can use it to solve scale problems across your stack such as managing high throughput inbound events and building data pipelines
  • Answers to questions posed by the audience (see some examples)

Next steps

Speakers
Rand Fitzpatrick

Rand Fitzpatrick

Director of Product

Heroku

Chris Castle

Chris Castle

Developer Advocate

Heroku

Session Q&A

Here’s a sample of the most common questions asked during the webinar along with their answers.

If you have any other questions, please get in touch.

Q: Apache Kafka new release version X has some great features. Does Heroku support those?
A: We aim to support new Kafka versions very soon after they are released. Additionally, once we support a new version, you can upgrade a Apache Kafka on Heroku cluster with no downtime using the CLI command heroku kafka:upgrade [CLUSTER] --version TARGET_VERSION. Please see Kafka Versions and Clients for more information about supported versions.

Q: Is the Apache Kafka on Heroku add-on sharable across apps? Many use cases would warrant multiple applications consuming and producing events.
A: Yes, please see the add-on page for more detail.

Q: What are the client libraries you recommend?
A: There are some recommended client libraries in the Language Support section of our Kafka documentation.

Q: How does Apache Kafka on Heroku handle authentication and traffic encryption for client connections?
A: All connections to Kafka support SSL encryption and authentication. Please see Connecting to a Kafka Cluster for more information.

Q: When would you advocate the use of a traditional message queue (e.g. RabbitMQ) over Apache Kafka?
A: We would advocate something like RabbitMQ over Kafka if you have a use case that is only queue-like in nature, is relatively low-throughput, and that does not require durability. Kafka provides for more options to structure your stream data, to persist the events as immutable records, and to hit higher throughput. Kafka is a simpler system, in many ways, and offers a greater level of flexibility and more robust scale.

Q: Are Apache Kafka and Enterprise Service Bus (ESB) similar concepts?
A: Kafka tends to be simpler, architecturally, than most ESBs. Also, it provides both durability (persistence and distribution) and high throughput in ways that most ESBs of prior epochs did not.

Q: How do I test out Apache Kafka on Heroku in a more affordable way?
A: We understand this is a critical need. We prioritized the dedicated cluster offering, to serve our customers' production use cases. Our teams are hard at work on non-dedicated cluster offerings which should more flexibly meet test and early development needs.

Q: How can I maintain Kafka clusters in two data centers (e.g. Europe and Asia)?
A: It depends on the specific use case. The simplest method is to write messages to both clusters. Another options is to run MirrorMaker on a Heroku dyno with the secondary cluster set as a target. This would automatically sync data from the primary cluster to the secondary one.