Search overlay panel for performing site-wide searches

Boost Performance & Scale with Postgres Advanced. Join Pilot Now!

Apache Kafka

Building a Service-oriented Architecture with Rails and Kafka

This blog post is adapted from a talk given by Stella Cotton at RailsConf 2018 titled “ So You’ve Got Yourself a Kafka .”

https://www.youtube.com/embed/Rzl4O1oaVy8

In recent years, designing software as a collection of services, rather than a single, monolithic codebase, has become a popular way to build applications. In this post, we’ll learn the basics of Kafka and how its event-driven process can be used to power your Rails services. We’ll also talk about practical considerations and operational challenges that your event-driven Rails services might face around monitoring and scaling.

What is Kafka?

…

Manage Real-time Event Streams and SQL Analytics with Apache Kafka on Heroku

Building a SaaS product, a system to handle sensor data from an internet-connected thermostat or car, or an e-commerce store often requires handling a large stream of product usage data, or events. Managing event streams lets you view, in near real-time, how users are interacting with your SaaS app or the products on your e-commerce store; this is interesting because it lets you spot anomalies and get immediate data-driven feedback on new features. While this type of stream visualization is useful to a point, pushing events into a data warehouse lets you ask deeper questions using SQL.

In this post, we’ll…

Deconstructing Monolithic Apps with SHIFT Commerce

Editor’s Note: One of the joys of building Heroku is hearing about the exciting applications our customers are crafting. SHIFT Commerce – a platform helping retailers optimize their e-commerce strategy – is a proud and active user of Heroku in building its technology stack. Today, we’re clearing the stage for Ryan Townsend, CTO of SHIFT , as he provides an overview of SHIFT’s journey into building microservices architecture with the support of Apache Kafka on Heroku. Software architecture has been a continual debate since software first came into existence. The latest iteration of this long-running discussion is between…

Kafka Streams on Heroku

Designing scalable, fault tolerant, and maintainable stream processing systems is not trivial. The Kafka Streams Java library paired with an Apache Kafka cluster simplifies the amount and complexity of the code you have to write for your stream processing system.

Unlike other stream processing systems, Kafka Streams frees you from having to worry about building and maintaining separate infrastructural dependencies alongside your Kafka clusters. However, you still need to worry about provisioning, orchestrating, and monitoring infrastructure for your Kafka Streams applications.

Heroku makes it easy for you to deploy, run, and scale your Kafka Streams applications by…

Kafka Everywhere: New Plans and Pricing for Apache Kafka on Heroku

Event-driven architectures are on the rise, in response to fast-moving data and constellations of inter-connected systems. In order to support this trend, last year we released Apache Kafka on Heroku – a gracefully integrated, fully managed, and carefully optimized element of Heroku's platform that is the culmination of years of experience of running many hundreds of Kafka clusters in production and contributing code to the Kafka ecosystem.

Today, we are excited to announce additional plans and pricing in our Kafka offering in order to make Apache Kafka more accessible, and to better support development, testing, and low volume…

Pulling the Thread on Kafka’s Compacted Topics

At Heroku, we're always working towards improving operational stability with the services we offer. As we recently launched Apache Kafka on Heroku , we've been increasingly focused on hardening Apache Kafka, as well as our automation around it. This particular improvement in stability concerns Kafka's compacted topics, which we haven't talked about before. Compacted topics are a powerful and important feature of Kafka, and as of 0.9, provide the capabilities supporting a number of important features.

Meet the Bug

The bug we had been seeing is that an internal thread that's used by Kafka to…

Data Pipelines with Kafka and Functional Reactive Programming with Node.js

Heroku recently released [a managed Apache Kafka][1] offering. As a Node.js developer, I wanted to demystify Kafka by sharing a simple yet practical use case with the many Node.js developers who are curious how this technology might be useful. At Heroku we use Kafka internally for a number of uses including data pipelines . I thought that would be a good place to start.

When it comes to actual examples, Java and Scala get all the love in the Kafka world. Of course, these are powerful languages, but I wanted to explore Kafka from the perspective of Node.js. While there are…

Apache Kafka on Heroku is Now Generally Available

Many of the compelling and engaging application experiences we enjoy every day are powered by event-based systems; requesting a ride and watching its progress, communicating with a friend or large group in real time, or connecting our increasingly intelligent devices to our phones and each other. Behind the scenes, similar architectures let developers connect separate services into single systems, or process huge data streams to generate real-time insights. Together, these event-driven architectures and systems are quickly becoming a powerful complement to the relational database and app server models that have been at the core of Internet applications for…

Dawn of the Dead Ends: Fixing a Memory Leak in Apache Kafka

At Heroku, we're always working towards increased operational stability with the services we offer. As we recently launched the beta of Apache Kafka on Heroku , we've been running a number of clusters on behalf of our beta customers.

Over the course of the beta, we have thoroughly exercised Kafka through a wide range of cases, which is an important part of bringing a fast-moving open-source project to market as a managed service. This breadth of exposure led us to the discovery of a memory leak in Kafka, having a bit of an adventure debugging it, and then…

Powering the Heroku API With Streams & Apache Kafka

We recently launched Apache Kafka on Heroku into beta. Just like we do with Heroku Postgres, our internal engineering teams have been using our Kafka service to power a number of our internal systems.

The Big Idea

The Heroku platform comprises a large number of independent services. Traditionally we’ve used HTTP calls to communicate between these services. While this approach is simple to implement and easy to reason about, it has a number of drawbacks. Synchronous calls mean that the top-level request time will be gated by the slowest backend component. Also, internal API calls create tight point-to-point…

Subscribe to the full-text RSS feed for Apache Kafka.