Kafka high level diagram

What is Kafka?

Apache Kafka is a distributed commit log for fast, fault-tolerant communication between producers and consumers using message based topics. Kafka provides the messaging backbone for building a new generation of distributed applications capable of handling billions of events and millions of transactions.


Take Control of Your Events

Events are everywhere — user activity streams, log events, telemetry from embedded devices and mobile phones, and more. Kafka flips the script from push to pull, letting you take control of high volume event streams in your applications to transform the customer experience. With Kafka, you can accept inbound events at any scale with ease and route them to key-based partitions, providing a clear path to real-time stream processing for user activity tracking, ad tracking, IoT, mobile sync and messaging systems.

Build Modern Application Architectures

Application architectures like microservices require new approaches to coordination, scaling and orchestration. Kafka’s pull based communication model reduces backpressure on key services under load, letting you add and scale new services independently. Kafka enables moving from actor to channel centric app dev models, simplifying services discovery and reducing brittle RPC style and many-to-many coordination between services.

New Ways to Process Data and Time

Kafka lets you rethink the relationship between data, time and operations in your application. Kafka takes transactional data in tables and reduces it to a series of events, each representing a keyed record and operation at a point in time. This lets you create a record of all change events in your application for data recovery, replay, simulation and auditing. These same primitives let you build powerful data processing pipelines for analytics and transformation use cases, with consumers reading data from a set of topics, applying functions, and writing the output to a new set of topics.

Why Apache Kafka on Heroku?

World Class Operations

Now you can consume Kafka as a service with Heroku’s world-class orchestration and thoughtfully tuned configurations that keep Kafka fast and robust. We distribute Kafka resources across network zones for fault-tolerance, and ensure your Kafka cluster is always available and addressable.

Elegant Developer Experience

Easy to use CLI and web tooling make Kafka simple to provision, configure and operate. Add topics, create partitions, manage log compaction, and monitor key metrics from the comfort of the CLI or Heroku Dashboard.

Seamless Integration with Apps

Run producers and consumers as Heroku apps for simple vertical and horizontal scalability. Config vars make it easy to securely connect to your Kafka cluster, so you can focus on your core logic.

How it Works

Kafka provides a powerful set of primitives for connecting your distributed application: messages, topics, partitions, producers, consumers, and log compaction.


Messages

Kafka is a message passing system, messages are events and can have keys.

Brokers

A Kafka cluster is made up of brokers that run Kafka processes.

Topics

Topics are streams of messages of a particular category.

Partitions

Partitions are append only, ordered logs of a topic’s messages. Messages have offsets denoting position in the partition. Kafka replicates partitions across the cluster for fault tolerance and message durability.

Producers

Producers are client processes that send messages to a broker on a topic and partition. Producers can use a partitioning function on keys to control message distribution.

Consumers

Consumers read messages from topics' partitions on brokers, tracking the last offset read to coordinate and recover from failures. Consumers can be deployed in groups for scalability.

Log Compaction

Log compaction keeps the most recent value for every key so clients can restore state.

Go Deeper with Apache Kafka on Heroku

Take me to the docs
Tech sessions

Building Event Driven Architectures with Kafka on Heroku

Join us on November 3, 2016 at 9am PDT

Apache Kafka can be used to stream billions of events per day — but do you know where to use it in your app architecture? Find out at our technical session. Get a live demo and answers to your questions from Heroku product experts.