Building Event Driven Architectures
with Apache Kafka on Heroku

Technical Session

Apache Kafka is the backbone for building architectures that deal with billions of events a day. Chris Castle, developer advocate, will show you where it might fit in your roadmap.

See a live technical demo showing you how to quickly get started with Kafka on Heroku.

Hear questions answered in a live Q&A with Rand Fitzpatrick from the Heroku product team.

Watch session

Want to learn more about Apache Kafka on Heroku? Give us a shout.

Session slides

Kafka on Heroku
Kafka on Heroku is a durable, distributed messaging service for stream processing, optimized for developers

You’ll learn:

  • What Apache Kafka is and how to use it on Heroku
  • How Kafka enables you to model your data as immutable streams of events, introducing greater parallelism into your applications
  • How you can use it to solve scale problems across your stack such as managing high throughput inbound events and building data pipelines
  • Answers to questions posed by the audience (see some examples)

Next steps

Speakers
Rand fitzpatrick

Rand Fitzpatrick

Director of Product

Heroku

Chris castle

Chris Castle

Developer Advocate

Heroku

Session Q&A

Here’s a sample of the most common questions asked during the webinar along with their answers.

If you have any other questions, please get in touch.

Q: Apache Kafka new release version X has some great features. Does Heroku support those?
A: We aim to support new Kafka versions very soon after they are released. Additionally, once we support a new version, you can upgrade a Apache Kafka on Heroku cluster with no downtime using the CLI command heroku kafka:upgrade [CLUSTER] --version TARGET_VERSION. Please see Kafka Versions and Clients for more information about supported versions.

Q: Is the Apache Kafka on Heroku add-on sharable across apps? Many use cases would warrant multiple applications consuming and producing events.
A: Yes, please see the add-on page for more detail.

Q: What are the client libraries you recommend?
A: There are some recommended client libraries in the Language Support section of our Kafka documentation.

Q: How does Apache Kafka on Heroku handle authentication and traffic encryption for client connections?
A: All connections to Kafka support SSL encryption and authentication. Please see Connecting to a Kafka Cluster for more information.

Q: When would you advocate the use of a traditional message queue (e.g. RabbitMQ) over Apache Kafka?
A: We would advocate something like RabbitMQ over Kafka if you have a use case that is only queue-like in nature, is relatively low-throughput, and that does not require durability. Kafka provides for more options to structure your stream data, to persist the events as immutable records, and to hit higher throughput. Kafka is a simpler system, in many ways, and offers a greater level of flexibility and more robust scale.

Q: Are Apache Kafka and Enterprise Service Bus (ESB) similar concepts?
A: Kafka tends to be simpler, architecturally, than most ESBs. Also, it provides both durability (persistence and distribution) and high throughput in ways that most ESBs of prior epochs did not.

Q: How do I test out Apache Kafka on Heroku in a more affordable way?
A: We understand this is a critical need. We prioritized the dedicated cluster offering, to serve our customers' production use cases. Our teams are hard at work on non-dedicated cluster offerings which should more flexibly meet test and early development needs.

Q: How can I maintain Kafka clusters in two data centers (e.g. Europe and Asia)?
A: It depends on the specific use case. The simplest method is to write messages to both clusters. Another options is to run MirrorMaker on a Heroku dyno with the secondary cluster set as a target. This would automatically sync data from the primary cluster to the secondary one.