Search overlay panel for performing site-wide searches
Salesforce (Heroku) Named a Leader. Learn More!

kafka

Building an Event-Driven Architecture with Managed Data Services

Modern applications have an unceasing buzz of user activity and data flows. Users send a flurry of one-click reactions to social media posts. Wearable tech and other IoT sensors work nonstop to transmit event data from their environments. Meanwhile, customers on e-commerce sites perform shopping cart actions or product searches which can bring immediate impact to operations. Today’s software organizations need the ability to process and respond to this rich stream of real-time data. That’s why they adopt an event-driven architecture (EDA) for their applications. Long gone are the days of monolithic applications with components tightly coupled into a single,…

Heroku Streaming Data Connectors Are Now Generally Available

This summer, we announced the beta release of our new streaming data connectors between Heroku Postgres and Apache Kafka on Heroku. These connectors make Change Data Capture (CDC) possible on Heroku with minimal effort. Anyone with a Private or Shield Space, as well as a Postgres and an Apache Kafka add-on in that space, can use Streaming Data Connectors today at no additional charge. Customers use connectors to build streaming data pipelines between Salesforce and external stores like a Snowflake data lake or an AWS Kinesis queue for integration with other data sources. They also refactor monoliths into microservices, implement…

Introducing the Streaming Data Connectors Beta: Capture Heroku Postgres Changes in Apache Kafka on Heroku

Today we are announcing a beta release of our new streaming data connector between Heroku Postgres and Apache Kafka on Heroku. Heroku runs millions of Postgres services and tens of thousands of Apache Kafka services, and we increasingly see developers choosing to start with Apache Kafka as the foundation of their data architecture. But for those who are Postgres-first, it is challenging to adopt without a full app rewrite. Developers want a seamless integration between the two services, and we are delivering it today, at no additional charge, for Heroku Private Spaces and Shield Spaces customers. Moving beyond Postgres and…

Heroku Shield for Redis Is Now Generally Available

We are thrilled to announce that Heroku Shield for Redis is now generally available and certified for handling PHI, PII, and HIPAA-compliant data. Heroku Shield for Redis is the final missing data service for Heroku Shield, which is an integrated set of Heroku services with additional security features needed for building high compliance applications. All Heroku Managed Data Services — Heroku Connect, Heroku Data for Redis, Heroku Postgres, and Apache Kafka on Heroku — are now fully certified for handling PHI, PII, and HIPAA-compliant data as part of Heroku Shield. Security and compliance come standard with Heroku Shield, so developers…

Bring Your Own Key for Heroku Managed Data Services Is Now Generally Available

Security is always top of mind for Heroku customers; COVID-19 has further increased the urgency for enterprises and developers to deliver more mission-critical applications with sensitive and regulated data. Given the needs of our customers, including those in regulated industries like Health & Life Sciences and Financial Services, we are thrilled to announce that Heroku Private Spaces and Shield customers can now deploy a new Postgres, Redis, or Apache Kafka service with a key created and managed in their private AWS KMS account. With BYOK, enterprises gain full data custody and data access control without taking on the burden of…

Why Frequent Maintenances Are Essential for Secure Heroku Data Services

There are many reasons to choose Heroku Data services, but keeping the services you use secure and up-to-date rank near the top. This foundation of trust is the most important commitment we make to our customers, and frequent and timely maintenances are one way we deliver on this promise. We do everything we can to minimize downtime, which is typically between 10 – 60 seconds per maintenance. There are ways for you to minimize disruption too (see the tips and tricks below). The rest of the post explains how we think about Heroku Data maintenances, how we perform them, and…

How Heroku Operates its Multi-Tenant Apache Kafka Services

This blog post is adapted from a talk given by Ali Hamidi at Data Council SF ’19 titled “Operating Multi-Tenant Kafka Services for Developers on Heroku.” https://www.youtube.com/embed/-AtHKoTNR1k Thousands of developers use Heroku’s Apache Kafka service to process millions of transactions on our platform—and many of them do so through our multi-tenant Kafka service. Operating Kafka clusters at this scale requires careful planning to ensure capacity and uptime across a wide range of customer use cases. With significant automation and test suites, we’re able to do this without a massive operations team. In this post, we’re going to talk about how we’ve architected our infrastructure…

Building a Service-oriented Architecture with Rails and Kafka

This blog post is adapted from a talk given by Stella Cotton at RailsConf 2018 titled “So You’ve Got Yourself a Kafka.” https://www.youtube.com/embed/Rzl4O1oaVy8 In recent years, designing software as a collection of services, rather than a single, monolithic codebase, has become a popular way to build applications. In this post, we’ll learn the basics of Kafka and how its event-driven process can be used to power your Rails services. We’ll also talk about practical considerations and operational challenges that your event-driven Rails services might face around monitoring and scaling. What is Kafka? Suppose you want to know more information about…

Managing Real-time Event Streams and SQL Analytics with Apache Kafka on Heroku, Amazon Redshift, and Metabase

Building a SaaS product, a system to handle sensor data from an internet-connected thermostat or car, or an e-commerce store often requires handling a large stream of product usage data, or events. Managing event streams lets you view, in near real-time, how users are interacting with your SaaS app or the products on your e-commerce store; this is interesting because it lets you spot anomalies and get immediate data-driven feedback on new features. While this type of stream visualization is useful to a point, pushing events into a data warehouse lets you ask deeper questions using SQL. In this post,…

SHIFT Commerce’s Journey: Deconstructing Monolithic Applications into Services

Editor’s Note: One of the joys of building Heroku is hearing about the exciting applications our customers are crafting. SHIFT Commerce – a platform helping retailers optimize their e-commerce strategy – is a proud and active user of Heroku in building its technology stack. Today, we’re clearing the stage for Ryan Townsend, CTO of SHIFT, as he provides an overview of SHIFT’s journey into building microservices architecture with the support of Apache Kafka on Heroku. Software architecture has been a continual debate since software first came into existence. The latest iteration of this long-running discussion is between monoliths and microservices…

Subscribe to the full-text RSS feed for Jonathan Brown.