Search overlay panel for performing site-wide searches

Boost Performance & Scale with Postgres Advanced. Join Pilot Now!

Engineering

Scaling ipify to 30 Billion Requests and Beyond on Heroku

The following is the story of how Randall Degges created a simple API to solve the common problem of external IP address lookup and how he scaled it from zero to over 10 thousand requests per second (30B/month!) using Node.js and Go on Heroku. Several years ago I created a free web service, ipify . It is a highly scalable IP address lookup service. When you make a GET request against it, it returns your public-facing IP address. Try it out yourself!

 

I created ipify because, at the time, I was building complex infrastructure management software and needed…

Kafka Streams on Heroku

Designing scalable, fault tolerant, and maintainable stream processing systems is not trivial. The Kafka Streams Java library paired with an Apache Kafka cluster simplifies the amount and complexity of the code you have to write for your stream processing system.

Unlike other stream processing systems, Kafka Streams frees you from having to worry about building and maintaining separate infrastructural dependencies alongside your Kafka clusters. However, you still need to worry about provisioning, orchestrating, and monitoring infrastructure for your Kafka Streams applications.

Heroku makes it easy for you to deploy, run, and scale your Kafka Streams applications by…

Jekyll on Heroku

Jekyll , the static website generator written in Ruby and popularized by GitHub, is a great candidate for being run on Heroku. Originally built to run on GitHub Pages, running Jekyll on Heroku allows you to take advantage of Jekyll’s powerful plugin system to do more than convert Markdown to HTML. On my blog, I have plugins to download my Goodreads current and recently read books and to generate Open Graph images for posts. That said, it’s not straightforward to get up and running on Heroku without using jekyll serve to do the heavy lifting. jekyll serve uses Ruby’s…

In the Cloud, No One Can Hear Your OutOfMemoryError

Pushing an app to the cloud can feel like launching a probe into space. Once your project is thousands of miles away you can't bang on it with a hammer or replace broken parts when there's a problem. Your debugging efforts must rely on the instrumentation, telemetry, and remote controls included with the app when it was deployed. On Heroku, we've gladly done some of that prep work for you.

Two new Heroku features, Heroku Exec and Language Runtime Metrics, improve your production monitoring, inspecting, and debugging experience on the platform. With Heroku Exec, you can create secure…

Evolution of the Heroku CLI: 2008-2017

Over the past decade, millions of developers have interacted with the Heroku CLI . In those 10 years, the CLI has gone through many changes. We've changed languages several times; redesigned the plugin architecture; and improved test coverage and the test framework. What follows is the story of our team's journey to build and maintain the Heroku CLI from the early days of Heroku to today.

Ruby (CLI v1-v3) Go/Node (CLI v4) Go/Node (CLI v5) Pure Node (CLI v6) What's Next?

…

Hello RedBeat: A New Celery Beat Scheduler

The Heroku Connect team ran into problems with existing task-scheduling libraries. Because of that, we wrote RedBeat , a Celery scheduler that stores scheduled tasks and runtime metadata in Redis . We’ve also open-sourced it so others can use it. Here is the story of why and how we created RedBeat.

Why We Created the RedBeat Celery Scheduler

Heroku Connect makes heavy use of Celery to synchronize data between Salesforce and Heroku Postgres . Celery is an asynchronous task queue that lets us schedule and queue jobs for execution by a background worker process. Over time, our…

Latency Incident Report: Socket Bind Before Connect

Back on August 11, 2016, Heroku experienced increased routing latency in the EU region of the common runtime. While the official follow-up report describes what happened and what we’ve done to avoid this in the future, we found the root cause to be puzzling enough to require a deep dive into Linux networking.

The following is a write-up by SRE member Lex Neva ( what’s SRE? ) and routing engineer Fred Hebert (now Heroku alumni) of an interesting Linux networking “gotcha” they discovered while working on incident 930.

The Incident

Our monitoring systems paged us about a rise in…

How We Found and Fixed a Filesystem Corruption Bug

As part of our commitment to security and support, we periodically upgrade the stack image, so that we can install updated package versions, address security vulnerabilities, and add new packages to the stack. Recently we had an incident during which some applications running on the Cedar-14 stack image experienced higher than normal rates of segmentation faults and other “hard” crashes for about five hours . Our engineers tracked down the cause of the error to corrupted dyno filesystems caused by a failed stack upgrade. The sequence of events leading up to this failure, and the technical details of the…

Pulling the Thread on Kafka’s Compacted Topics

At Heroku, we're always working towards improving operational stability with the services we offer. As we recently launched Apache Kafka on Heroku , we've been increasingly focused on hardening Apache Kafka, as well as our automation around it. This particular improvement in stability concerns Kafka's compacted topics, which we haven't talked about before. Compacted topics are a powerful and important feature of Kafka, and as of 0.9, provide the capabilities supporting a number of important features.

Meet the Bug

The bug we had been seeing is that an internal thread that's used by Kafka to…

How We Sped up SNI TLS Handshakes by 5x

During the development of the recently released Heroku SSL feature, a lot of work was carried out to stabilize the system and improve its speed. In this post, I will explain how we managed to improve the speed of our TLS handshakes by 4-5x.

The initial reports of speed issues were sent our way by beta customers who were unhappy about the low level of performance. This was understandable since, after all, we were not greenfielding a solution for which nothing existed, but actively trying to provide an alternative to the SSL Endpoint add-on, which is provided by…

Subscribe to the full-text RSS feed for Engineering.