Building Event Driven Architectures with Apache Kafka on Heroku
Technical session
Apache Kafka is the backbone for building architectures that deal with billions of events a day. Chris Castle, developer advocate, will show you where it might fit in your roadmap.
See a live technical demo showing you how to quickly get started with Kafka on Heroku.
Hear questions answered in a live Q&A with Rand Fitzpatrick from the Heroku product team.
You’ll learn:
- What Apache Kafka is and how to use it on Heroku
- How Kafka enables you to model your data as immutable streams of events, introducing greater parallelism into your applications
- How you can use it to solve scale problems across your stack such as managing high throughput inbound events and building data pipelines
- Answers to questions posed by the audience (see some examples)
Kafka on Heroku is a durable, distributed messaging service for stream processing, optimized for developers
Next steps
- Learn more about the Kafka on Heroku product or read the documentation.
- Contact us if you’d like to get in touch regarding Apache Kafka on Heroku.
Speakers
Rand Fitzpatrick
Director of Product
Heroku
Christ Castle
Developer Advocate
Heroku
Session Q&A
Here’s a sample of the most common questions asked during the webinar along with their answers.
If you have any other questions, please get in touch.
Q: Apache Kafka new release version X has some great features. Does Heroku support those?
A: We aim to support new Kafka versions very soon after they are released. Additionally, once we support a new version, you can upgrade a Apache Kafka on Heroku cluster with no downtime using the CLI command heroku kafka:upgrade [CLUSTER] --version TARGET_VERSION. Please see Kafka Versions and Clients for more information about supported versions.
Q: Is the Apache Kafka on Heroku add-on sharable across apps? Many use cases would warrant multiple applications consuming and producing events.
A: Yes, please see the add-on page for more detail.
Q: What are the client libraries you recommend?
A: There are some recommended client libraries in the Language Support section of our Kafka documentation.
Q: How does Apache Kafka on Heroku handle authentication and traffic encryption for client connections?
A: All connections to Kafka support SSL encryption and authentication. Please see Connecting to a Kafka Cluster for more information.
Q: When would you advocate the use of a traditional message queue (e.g. RabbitMQ) over Apache Kafka?
A: We would advocate something like RabbitMQ over Kafka if you have a use case that is only queue-like in nature, is relatively low-throughput, and that does not require durability. Kafka provides for more options to structure your stream data, to persist the events as immutable records, and to hit higher throughput. Kafka is a simpler system, in many ways, and offers a greater level of flexibility and more robust scale.
Q: Are Apache Kafka and Enterprise Service Bus (ESB) similar concepts?
A: Kafka tends to be simpler, architecturally, than most ESBs. Also, it provides both durability (persistence and distribution) and high throughput in ways that most ESBs of prior epochs did not.
Q: How do I test out Apache Kafka on Heroku in a more affordable way?
A: We understand this is a critical need. We prioritized the dedicated cluster offering, to serve our customers’ production use cases. Our teams are hard at work on non-dedicated cluster offerings which should more flexibly meet test and early development needs.
Q: How can I maintain Kafka clusters in two data centers (e.g. Europe and Asia)?
A: It depends on the specific use case. The simplest method is to write messages to both clusters. Another options is to run MirrorMaker on a Heroku dyno with the secondary cluster set as a target. This would automatically sync data from the primary cluster to the secondary one.