Cloud Database Workloads with Jon Daniel

SED 829 Transcript, EPISODE 829

[00:00:00] JM: Relational databases such as Postgres are often used for mission-critical
workloads such as user account data. To run a relational database service in the cloud requires
a cloud provider to set up a highly durable, highly available system.

Jon Daniel is an infrastructure engineer at Heroku. Jon joins the show to describe the
engineering and operations required to build a managed relational database service. Full
disclosure, Heroku is a sponsor of Software Engineering Daily.
<!– more –>
[00:02:04] JM: Jon Daniel, you are an infrastructure engineer at Heroku. Welcome to Software
Engineering Daily.

[00:02:08] JD: Thanks, Jeffrey. Thanks for having me here.

[00:02:11] JM: I'm excited to talk to you about running databases in the cloud, because that's a
complicated infrastructure problem. I want to start off by just talking at a high-level about
relational databases. Why do people use relational databases?

[00:02:28] JD: Yeah, this is a question that we get a lot, especially with things like CouchDB,
Mongo or whatever the latest NoSQL solution is these days. People pretty much use relational
databases because they are well-understood and their battle tested ways of storing, retrieving
and processing data.

[00:02:46] JM: And what kinds of queries perform better in relational databases versus the
other kinds of databases you mentioned, like document databases like MongoDB?

[00:02:56] JD: Yeah. So any query that kind of involves like an algebraic-like relationship, such
as associating like a group of people with a group of addresses and you have two different data
sources to work with. Also, these transactional natures can be – The transactional nature of
these databases, they can be really useful for things like financial systems or customer
relationship management tools.

[00:03:16] JM: So there are different relational databases that have different performance
characteristics. We have Postgres, we have MySQL, we have SQLite. We have the cornucopia
of “NewSQL databases”. How are the different databases that we encounter in the wild, how are
they making tradeoffs and what are those tradeoffs?

[00:03:40] JD: Right. So I used to work with MySQL pretty heavily before I made the jump to
Postgres about three years ago. So basically Postgres and MySQL for a while were sort of
neck and neck and what they can offer in feature set. Postgres these days seems to offer more
native, like built-in types that they have, like a standard, like JSON type. They have one for
siders. They have loads and loads and loads of others. Whereas I feel it MySQL has kind of
taken the let's take this project that we already have and keep improving the performance and
the stability of it, but not really kind of expand beyond that.

But MySQL has the advantage of it kind of being like the first big open source SQL database to
gain the sheer volume of traction that it did. So it's always going to have that built-in section of
users. Postgres on the other hand, I feel like it iterates far faster. It also made some design
choices that I think are a little better and that it relies on UNIX process types instead of like
handling threads and all of these other things. I find it easier to administer and easier to track
down issues when they occur.

Then we have things like SQLite, which Rails including SQLite by default was I think a mistake,
because running a web application on SQLite is just not what it was designed for. It was meant
for embedded SQL. It was meant for running things on a device where you don't necessarily
have network access. The fact that Apple included SQLite in the iOS for a lot of like the app
backend management, I think that is a more appropriate use of it and something I wish more
people would understand. But I think SQLite in general is an extremely, extremely powerful tool.

Way better than all of us building our own half-baked SQL implementations for our local apps
and such.

I'm not as familiar with some of the new SQL databases, but there are a few that I've seen that
look interesting. I know that like React kind of came about a while ago. I'm not entirely sure what
query language it uses, but a lot of people in the operations world seem to get very excited
about it. At a previous job, we used Datomic a bit, which sits on top of either Postgres or
MySQL, and it allows you to do some really interesting things, like going back kind of in time to
query differences and all of that stuff, which sounds extremely exciting.

[00:06:06] JM: So the subject of relational databases, we can go in a lot of different directions,
but we've covered relational databases and their usage in a lot of previous episodes. For this
episode I’d like to take the approach of how do you run a relational database in the cloud from
the point of view of a cloud provider.

For example, if I'm spinning up a web application where I'm using a SQL database as the
backing store, I want to be able to just one click instantiate that database and start to make
requests to it, and that feels very seamless and simple from the developer's point of view today.

But, of course, so much infrastructure work has gone on to allow that to be a seamless
experience. Could you just give us an outline of what are some of the challenges, the
fundamental challenges, from a cloud provider's point of view of offering a SQL database in the
cloud?

[00:07:11] JD: Oh boy! Where do I even start? So offering a SQL database in the cloud, it feels
like a very simple process coming from a kind of more traditional like IT systems administrator
background. You grab an operating system like Ubuntu or CentOS, you throw it on a server.

Yum or apt-get install Postgres and you’re ready to go.

But from a cloud provider standpoint, it's a much more involved process. First off, before Heroku
even existed, whenever we had to make these decisions about providers and do we run the
servers ourselves, do we use something like Amazon or DigitalOcean or Linode. First off, you
have to make that provider decision. You have to look at pricing. You have to look at what they
offer. You have to look at what kind of support is available, and especially automation. What
APIs do they expose? So you have to make that provider choice fairly early on, which is a big
decision.

Next up, you have to investigate what exactly do they offer from an instance perspective. In
older days of AWS, people would mistakenly provision a lot of services using their ephemeral
disks, not realizing that as soon as your EC2 is rebooted, all of your data is gone, and they had
EBS, which some people found to be rather complicated. But you have to make sure that your
data is persistent. So you have to have a server first. Then you have to provision disks. You
have to figure out what kind of disks do you need. EBS has like their gp2 drives, which are
general-purpose spinning disks. Then they have all their IO iterations that you can provision,
what are called prIOPS, which are priority IOPS. So how fast is this disk? How often can you
read and write to it?

So you have your server. You have your disk and you need to figure out how to connect it to
your greater environment. The old days of Heroku, we published everything open on the
internet, and it was behind an EC2 host name. That worked for the time, but these days we
actually run everything inside of a VPC. So it's a virtual private cloud, which you can think of it
as like almost like a VLAN of sorts, or a like mini-AWS instance or mini-AWS ecosystem that
you have access to.

So then you have to figure access controls. You have to figure out how do we take these
credentials from Postgres itself and vend it to all of our applications, and that's before you even
get into like installing and configuring Postgres. You have to think about instance types. What
are the different performance requirements that we need? What performance options do some
of these instances have?

For example, AWS offers instances that are called T2s, which burst CPU. They offer things that
burst disk. So, finally, you have to tune Postgres based on those kinds of instance types, and
that’s an extremely involved complicated lots of trial and error process. Once you get your
Postgres instance up, how do we handle failover? How do we handle database backups? What
do we do if the instance goes away because the hypervisor crashed? The list goes on and on,
and this could obviously take up an entire hour. So I'll plan to give you an idea of what goes on
there, and what I've said so far just scratches the surface.

[00:10:19] JM: That’s a great summer, and I'm looking forward to diving into some of those
topics. To take an even more naïve perspective, there are some people listening who have
probably never operated their own high-availability database, because they've always offloaded
that to the cloud. I think it's worth even discussing why are we doing this? Why did we get to this
point where we’re running a database in the cloud? Why is that preferable to the days of the 90s
when we had our own servers and we operated our own database and we served the traffic to
that database?

[00:10:56] JD: So it's all about risk, really. If you have your database running in your own server
and it's managed by your IT people, there is a lot more trust that you know exactly what's
happening. You can see the server. You can inspect the data center. You could do a lot of
things, but what happens if that server crashes? What happens if that server is damaged
beyond all repair? The CPU gets fried and you have to replace it? So now you have to
physically go out and buy another server.

There is a question a friend of mine always asks whenever he's touring a data center for like a
site evaluation, and that question is, “If I walk in here with chainsaws for hands and start cutting
every cable I see, how long until my website goes offline?” That’s something you have to
consider when choosing a data center.

We've had issues where a tractor-trailer loses its brakes and drives right into the power
management systems of the data center. Thankfully nobody was hurt the one time that
happened, but it does happen. If it is any physical data center that requires human access to
make changes, what happens when things go wrong?

Also, now there is an inherent cost to all of the underlying administration. We have to make sure
that the server’s operating system is up-to-date. We have to make sure patches are being
applied. We have to make sure we have intrusion detection systems, backend access in case of
emergency. If we can’t SSH into the server anymore, do we have helping hands? What do we
have? Putting it in the cloud kind of abstracts a lot of that away behind an API. So now you don't
have to worry about a physical server sitting somewhere and you don’t have to worry about the
BIOS getting out of date. You don’t have to worry about the server just getting old and dying.

You have a defined instance type. You tell Amazon or Google or whoever, “Here’s how much
memory I need. Here's how many CPUs I need. Here’s so much storage I need. I don't care
about the rest.” So it's just offloading things that I worry about.

But on the other hand, putting it in the cloud gives you may be a little bit less trust, because you
can't physically see the server or physically inspect the data center. But that's also what
contracts are for. So you have to change your perception of a few things. Frankly, personally,
after doing this for almost 11 years now, I trust Amazon's data center operations and their
security practices a lot more than myself running a server in a basement data center of an office
building.

[00:13:24] JM: Indeed, and we have been able to build on that trust further by using the
abstractions that we have been offered by AWS and building other services that have been built
on AWS, and people are developing trust in those services. You have core infrastructure that
has been built on top of AWS and people trust those providers that have built on top of AWS.

So we have this chain of trust, and trust can be articulated to some extent by the term
availability. Availability is an abstract term, but it basically means can you interface with the
software that you're trying to interface with? From a database standpoint, the database that is
highly available, you will be able to interface with that database from the point of view of writing
to that database, reading from that database, deleting data from that database. I guess that's
write, basically. But there are circumstances that can change over time, where maybe a
database is highly available, but it's only highly available from the read point of view. You might
have the write point of view be degraded in quality.

Could you talk about the term availability and explain how that applies to cloud databases?

[00:14:55] JD: Yeah. So availability, like you’ve said, has a lot of different terms. There’re a lot
of thought processes behind availability, and it depends on your workload. I don't want to name
names, but previous job I work for, it was an e-commerce provider, and their workload was
predominantly read- heavy. Any writes were to our payment providers mostly. You bought a
thing and now we have to make sure we can take your credit card.

So we defined availability in if you go to our.com and can see all the things we have for sale,
that's available, and that came down to, in this case, it was MySQL. Can MySQL return data?

So we had a lot of various read replicas that our developers were able to use. If we had some
backlog in writes or anything like that, we weren’t happy about it, but we were okay with that.

That didn't necessarily constitute like a P0 incident.

But a lot of customers, especially working with things like Salesforce CRM, which is what a
number of Heroku’s customers integrate with, at least at the higher level, write is extremely
important, because they're trying to like sync data between their CRM, which is their source of
record and their Postgres database so that their Heroku apps can perform whatever work they
need to do.

So in that case, write availability is far more important, and that puts us in kind of an interesting
position, because if we can spend up all of these database – We call them followers instead of
the old terminology, so leader and follower. If we can spin up all these followers to make reads
highly available, but writes fail, that's not helping the customer at all. S

So we kind of take a very sort of conservative approach to how we consider database available
in all cases. If we see a backlog of queries, or too many locks open, or even health checks
failing because they're failing to get a process available even if there are other processes
currently returning data. We do consider that database to be unavailable. We run some
automated remediation, and if that fails, we page an operator.

I’d say a lot of times it comes down to bad queries or things that are indexed incorrectly, or
things that are suboptimal. But there are a few cases that we've seen where EBS, for example,
will give out. They have had issues in the past where writes get queued and they can't hit their
sand fast enough. We do consider that be an unavailability situation, and if it's a highly available
database, we will trigger a failover hopefully getting it on to an instance that has a healthy EBS
disk, which considering that we use separate availability zones for leaders and followers, that
almost never happens. It's almost always safe. But it really depends on your application. When
you're a kind of general-purpose cloud provider like Heroku, you have to sort of assume that
read and write are both just as important.

[00:19:40] JM: Another term we should discuss here is consistency. When you talk about
databases, consistency often means if you have different replicas or different materialized views
of the database, to what degree are those views of the database the same?
Could you just talk about why consistency matters and the different circumstances where
inconsistency can develop and whether inconsistency is a problem? When it's a problem?

[00:20:15] JD: Yeah. I’d say the vast majority of cases, if there is inconsistency, that is a
problem. If you were to go to amazon.com and go to buy a new videogame, and I think the last
game I bought on Amazon was Spiderman for the PS4, and one minute it’s saying it has 25
copies available. You reload the page. It goes to a different database follower, and now it says it
has zero copies available. You're going to be a little unhappy. That's going to be a confusing
experience.

So consistency especially in Postgres, when you have like ACI-compliant databases, is
incredibly important. At Heroku, we use just native standard Postgres replication, the thing
that's built into the database engine that anybody can use, and it works most of the time almost
always, I would say. There are certainly instances where it can fall behind. If you do a lot of
writes, like really, really intensive database writes on your leader, you will start to see
inconsistency problems, because replication just cannot keep up.

I'm not going to name names, because this happens, but we did have a customer one time that
was essentially row-by-row deleting every record in their database and recreating it every single
day, and that caused their replication to fall behind so badly that we did the math, it would've
taken 20 some thousand years for it to catch up from a Postgres streaming replication
standpoint.

So these are things you have to kind of take into consideration when it comes to consistency,
that every time you run one of these queries, that same query is going to have to pass over a
network that may or may not be working at that present time, because the internet's a Wild
West, and it will have to replay on a different server. That different server could have different
qualities to it. It might be the same instance type, but it might be on a more noisy region or a
more noisy availability zone. It might have more disk load because something else is happening
on.

We've had cases where customers will use different instance types entirely for their leaders and
followers, and that's also not great. We have some last-ditch efforts that we do in case of
serious inconsistency. For example, if streaming replication fails so badly, we actually revert to
restoring a base backup on the follower to get it active again, and then we replay all of its writeahead
logs to try to get it closed enough for streaming replication takeover again. But you have
to just be mindful of every time you write data, that data doesn't – And just to that disk, there is a
whole power of things that happen behind-the-scenes.

[00:22:57] JM: What you're describing here is the fact that when we start using databases in
our developer 101 experiences, whether we’re hacking on a laptop, taking some coding school,
or we’re in computer science courses at the University. Generally, our experience with the
database is this thing that's sitting on my computer and it's just a single node instance and I
write to it and I read from it and everything feels pretty simple. But in a production deployment,
what you actually want from your database is replicas. You want a replicated database. You
want a situation where when I write to my database, I'm actually writing to multiple instances of
the same database, because if a meteor impacts the data center, I want to know that my data
has been replicated to another data center and I can continue humming along and my users can
expect availability from my application, because the data center that was not hit by a meteor is
still intact.

So maybe you could talk a little bit about this idea of replication and how replication fits in with
the other two terms we’ve discussed; availability and inconsistency.

[00:24:19] JD: Yeah. So replication is an easy concept in theory and very difficult in practice.
The rather naïve view of replication is I run an insert statement on database A. Something
happens, and that insert statement is magically run on database B.
In reality, it's essentially the same, but the implementation is a little bit different. But it's the basic
idea that data on one instance will always be replicated or reproduced on that other instance,
but there a lot of asterisks by that.

So if you have to databases physically sitting next to each other with a 10 gigabit per second
fiber-optic cable connecting the two, you're probably going to be fine. Nothing bad will most
likely happen. But when you're talking about like in the context of AWS, you have availability
zones, which in some cases AWS is a little hush-hush on the underlying implementation. But in
some cases they’re physically separate buildings on opposite sides of a neighborhood. There
can be – Whether it’d be copper or fiber leased lines running between these buildings, and that
is essentially entering a much more high-speed version of the internet. It's entering a MAN or a
metropolitan area network.

So you are going to have all sorts of things that could potentially go wrong. That data that you
write on database A that is in the same physical availability zone as your app servers might
have a 50 to 100 millisecond lag before it gets to database B, and that can cause your
replicants to fall out of sync. There’s an SLA involved. It's usually going to be in the order of
milliseconds, but as you write more and more and it takes longer and longer to reach the other
database and this replication gets backed up, you're going to get more and more behind.
So there's no magical answer to any of this. It's all about monitoring. It's all about operations.

You need to make sure that you have the appropriate alarms in place to know when databases
are falling out of sync and your operations team, whether it’d be someone like a service provider
like Heroku or GCP that you’re paying, or if it's your internal operations team your company.
They need to be aware of how do I graph that replication lag? How do I discover how far back
it's falling? How can I tell when this is just a momentary blip and it will catch up, because we had
a big heavy write operation? How do we tell if this is something more serious? Is there a
degraded network performance between two physical data centers? Is it one of the instances is
failing? There are so, so, so many questions, but it boils down to like most things in tech. It's not
magic.

[00:27:11] JM: There's an abstraction within databases such as Postgres that I'd like to discuss
a little bit, and this abstraction is the write-ahead log. I believe in Postgres it's called the writeahead log.

[00:27:23] JD: Yup.

[00:27:24] JM: And this is an abstraction that we need to achieve reliable replication. It's
basically the cure for the lack of this – You could just copy the insert statement from one
database to another, and most people who are interacting with the database never have to think
about a write-ahead log. They never have to interface with something like this. Could you
explain what a write-ahead log is and explain what we are doing with the write-ahead log in
order to have replication?

[00:27:55] JD: Right. So you can kind of to think of write-ahead log in its basic sense as a file
that contains all of the almost block on the hard drive level, but it's a little higher level than that.

But block level DIFs that happen between point A and point B. You can configure Postgres to
automatically put out a write-ahead log every so many seconds or whenever it reaches a certain
size. So like one megabyte, or whatever. But these are like the physical disk DIFs of point A and
point B.

So this is kind of more powerful than using just copying the “insert statement”, because in that
case, what happens if one of the insert statements fail on database B, but they succeed on
database A? There can be inconsistency there. But replaying the write-ahead log from database
A to database B ensures that at a like file system level these two databases are exactly the
same.

Your primary or your leader Postgres database every so often will spit out one of these writeahead
logs and it pipes it out to what you defined as your archive command in the Postgres
configuration file. So in a lot of the like mastering Postgres administration books that you'll see,
they'll talk about, “Okay, you can use rsync, or you can use copy, or FTP, or whatever to get this
elsewhere,” and what we’re doing at Heroku is essentially the same.

We use a tool called Wally, which will connect to S3, and every time a wall segment is pushed,
it'll actually push that into S3 to make sure that if the worst were to happen to your instance, you
are safe. The reason we use S3 over other options is S3 does guarantee high-availability. We
don't run things on like reduced availability. We can have all kinds of access controls in place to
make sure that nobody can read anybody else's wall, and it allows us to potentially give that
wall to customers if they want to do some kind of review or inspection or anything. That's not
something we do very often though. So it has a lot of advantages, and it's a lot better than
pgdump rsync to another database.

[00:30:10] JM: So that model of just writing the wall to an S3 bucket and you get replication that
way, does that mean that you actually only need one running Postgres instance because you
don't need to have actual replication in other database nodes?
[00:30:32] JD: This is how we’re getting back – Or this is something that goes back to
availability. In theory, yes. In practice, no. Because restoring a Postgres is not ever instant. If
you are building a basically Rails or Django app on your laptop and you’re trying to learn new
things, your databases is going to be a few megabytes and it will be really fast. But if we’re
talking about insert large e-commerce provider here, their database is going to be terabytes in
size, and performing a base backup restore and then Wally – Or then wall recreation from S3 on
that database will never be fast.

So it's about time. If you're willing to wait the hours or days to get your database back, which I
doubt any company is willing to do, then yes, you can rely purely on S3. But having that like
separate database ready to go, it’s the difference between an almost hot and cold standby. The
hot standby, the minute the leader has a problem and has to go offline, that hot standby is ready
to go. It can just start taking requests immediately, and relying on S3 is more of a cold standby.

Your data is safe. You didn't lose anything, but getting it back is going to take a little while.

[00:31:44] JM: I’d like to start getting into more of these discussions about what it's like to
actually implement a database service as a cloud provider. So if I log on to Heroku, I've got
service I want to deploy. It requires a Postgres database. What goes on in the backend when I
click provision, or deploy, or whatever I’m clicking to spin up a database. What's happening on
your infrastructure?

[00:32:16] JD: A lot. So Heroku at one point made this decision to kind of break away what we
call runtime, which are your actual application processes and what we consider add-ons, which
an add-on is something outside of a process running your rails app, for example. One of those
add-ons, for example, would be a database.

So if you run Heroku add-ons create standard zero, which is our lowest here and cheapest paid
plan, what happens is that API request that’s sent to platform API, which then sends a request
to the provider add-on, which in this case is Heroku itself. When we receive that request, we
look up the plan type. We look up your account. We make sure credentials are in order. We
make sure that you have a credit card on file that we can charge. But at that point, we grab an
EC2 instance. Often times we have them ready to go. We have them – What we call Slack
instances. So we don't want to spin up an entirely new one. We grab that instance. We do some
configuration on it. Pretty much, we configure Postgres to be the plan type you asked for. We
make sure that appropriate discs are mounted. We make sure appropriate IP addresses are
configured. We configure Postgres based on the version that you requested.

Actually, every supported version of Postgres that Heroku offers is installed on every single one
of our instances, and that's just to make the general provisioning faster and upgrading your
database later a little easier.

So once we have network, once we have disk, everything like that, we drop configuration files.

We create you a database user. We create that database. We install extensions that are
necessary, and at that point our control plane checks to see when this database is healthy,
when a consent select one, for example, to the database and get an appropriate response back.

Once that's done, we take those credentials and we export them through the platform API and
then it appears in your Heroku config listing, if you run the Heroku config, like show all
command, then you'll see it. At that point it's available.

[00:34:19] JM: Tell me a little bit more about the operational side of things. So you've got people
spinning up these databases. What's required on your end to make sure that those things stay
up?

[00:34:34] JD: Yeah. So from that perspective, we have a control plane that kind of acts as a
giant like clock or like scheduler, and we have these different process types. Some check
services specifically or Postgres services specifically. Some check what we call a server
abstraction, which is all things that make up a server. Others check specifically the instance,
specifically the disks, specifically the network, and we do a series of tests there to make sure
like, “Okay, TCP connections work. We can SSH into the box. We can run select one,” or
numerous other health check queries that you can see in your logs if you look closely enough.
We kind of evaluate all that to determine if an instance is healthy. We also do some preventative
checks to see when is this instance going to potentially stop being healthy.

One of them is disk size. We advertise plan limits. So a particular instance will have, say, one
terabyte cap. So what happens if you go to 1.1 terabyte? You have 1 terabyte provisioned, but
does that .1 that mean that you databases is dead? No. We look at the instances and we figure
out how fast is this growing, and if it looks like it's going to surpass that, we do provision extra
disk space, but let you know that this is going poorly.

Another thing we’ll do is we have a bit of a wall cache, because the network backs up
sometimes, and we can't just hold up your entire Postgres database because we can't write to a
wall directory, or wall S3 bucket. So we have the wall cache on the disk, and if that disk starts
getting too full, then we get alerted and we can take preventative action to prevent these kinds
of things. So there's more to the health of a database than just being able to select star from
people or insert into blah.

[00:36:24] JM: Has this stuff gotten easier to do in the last – I don’t know, three or four years?
Because I've spoken to a lot of companies who have, for example, migrated to Kubernetes, or
migrated to a managed container service, and their life has become a lot easier. They don't get
paged as much, because the way that the infrastructure gets architected, if it's built on top of
these container services, sometimes these managed services, can make things a lot easier. But
I'm not sure if that applies to something like a managed database, because that's kind of a
different beast than like an e-commerce company that's got some microservices and got a
single database and it’s much simpler architecture. What you’re doing as a little bit more
technically intense. So to what degree has running a database cloud service become easier in
the past few years due to some advances in fundamental infrastructure?

[00:37:25] JD: AWS has been a huge advantage, and I don't mean that the fact that AWS
exists, but more of AWS's commitment to enhance monitoring, enhanced operations, enhanced
API management. I haven't looked in Heroku, like my team at Heroku’s AWS console in
probably a year, because I don't have to, because if you can do it in the console, you can do it
via an API call, which is pretty amazing. A lot of other enhanced monitoring has really helped.

The fact that they can let us know ahead of time if a hypervisor is going to have problems.

That's when you get the like your database requires maintenance emails that everybody loves
from Heroku. Those are often imposed on us by Amazon.
But some of things that have made things easier, honestly, is the improvement of container
technology. So the fact that – We actually don't use Kubernetes or Docker in the data team at
Heroku, but we use LXC directly. The nice thing about that is being able to have fairly standard,
almost single tenant configuration set up on that instance, and the configuration changes
themselves as to like what version of Postgres or what plan type is really based on mounting.
So we know that everything is going to act very similarly from one to the other and there's not a
lot of one-off configuration happening in there.

Also, advancements around performance. Really, the fact that Amazon is able to offer us kind of
like enhanced provision priority networking, provisioned input-output operations on disks, faster
provisioning times. All these things have made working with these services easier and also
much, much faster to recover.

Prior to the introduction of VPC, whenever you had EC2 Classic or what they call EC2 Classic, it
was a total Wild West. If somebody was port scanning you from – If you had a noisy neighbor or
someone who’s port scanning even the same AZ, there wasn't much you could do aside from
complain to Amazon. But giving us this like secure isolated mini-AWS or mini-AWS ecosystem
has really help from a managing standpoint, because we can have a lot more trust that the only
things happening in our ecosystem are things that we know about.

[00:40:52] JM: We’ve had a number of shows recently about these second layer cloud
providers, which are cloud providers built on top of AWS, or cloud, or other fundamental cloud
infrastructure, and Heroku was one of the first to start to develop in that market. But there’s all
these interesting problems for the second layer cloud provider industry. One that I think is
interesting is the cost management issue.

If you're a software architect or a software engineer who's familiar with cloud services, it can
look like an all-you-can-eat buffet of things that you want to work with. But if you're building a
cloud provider on top of another cloud provider, your fundamental economics are really, really,
really important, because you need to be able to offer a somewhat commodity feeling service to
the user, your customer base. But you want your cost structure to remain low enough for that
kind of commodity service to be at a defensible price. At the same time you’re looking at all
these like new fancy Amazon services and these are the building blocks for your infrastructure.
You don't want to overspend on them. To what extent is cost management and issue the you're
thinking about?

[00:42:17] JD: Right. So one of the first things to think about when you're a kind of cloud
provider on top of another cloud provider is there's an episode of Seinfeld I think where Kramer
talks about nobody pays retail, and that's very much true once you get to a certain size in
Amazon. You have reserved instance types. You have prepay. They’ll do like volume discounts if
you're big enough, and having just your sheer size and like the weight of your organization
behind you can allow you to make really good deals with Amazon to get things like lower-priced
instances or you pay upfront for them or even just better support so you spend less time
mucking about, so to speak.

The other thing is you kind of have to take into account of almost the opportunity cost of what
you're providing to customers. I've seen numerous blog posts about like how I ditched Heroku
and moved to DigitalOcean. DigitalOcean, they offer some really great products. But the kinds
of things that they're going to offer with their Postgres databases may not be the same. We are
essentially taking on the role of your operations team, we’re doing a lot of preventative
maintenance. We’re doing the a lot of just things that you would never think about behind-the
scenes, and that’s sort of what we bake into our costs.
But there is a fine line. We need to make sure that our costs are competitive enough that we
won't really fall into, “Oh! It's just cheaper for me to hire an operations person to make that more
appealing,” but also not sell ourselves so short that we’re not making a lot of money. It's always
a fine balance, and more often than not, it's your bigger customers that sort of dictate the
direction that you go with a lot of your like higher-tiered plans. Because most customers at a
smaller stage, if you’re like standard zero to standard four, you're really looking for a fairly
general-purpose database, and those are more like we’re building cheaper cars here as
opposed to like building a Ferrari, and it's when you get into the larger customer scale that
you’re kind of building that like more customized solution. Your cost is becoming less of an issue
and it's more about the features you can provide.

[00:44:23] JM: You’ve shown in this conversation is a willingness to talk about other cloud
providers, which I appreciate. So I’d love to get kind of high-level perspective for this like layer 2
cloud provider, layer 1 cloud provider dynamic that has developed. As a software engineer, it
makes me really excited. Then kind of as a business analyst, it makes me really intrigued,
because who would've thought that there would've been this variety of second layer cloud
solutions that are built on top of other cloud providers so that the second layer kind of cloud
resellers or however you want to call them, they have a really like dynamic flexible business,
because they don’t have to invest in this core server infrastructure and their life is more about
like developer experience. It's like a design problem.

What kind of developer niche can I cater to? I mean, I've used Heroku for quite a long time
because I am more interested in kind of the product level discussions and decisions and maybe
like the frontend decisions. I don't really want to have to think about like did my write-ahead log
get backed up today. So like Heroku is kind of a good fit for that kind of thing. Other people
really enjoy getting their cost structures really, really low and don't really mind managing their
own cloud infrastructure. So it's like, “Okay. Maybe you want to go with AWS or DigitalOcean.” I
mean, even the layer one cloud providers have their own subjective decisions, design decisions
that they're making. So I'm just so optimistic about this environment, but maybe you could just
give me your feedback from being a developer for a while and obviously working at layer two
cloud solution. To some extent, I think the Salesforce has its own servers to some extent and
there are some Heroku involvement there. But just tell me what your perspective is on this
emergent dynamic of the layer 1 and layer 2 clouds.

[00:46:31] JD: I actually came from a primarily software development background before I
made the move to Heroku, which is a little more infrastructure-driven. I’m seeing a lot of
similarities between the evolution of a lot of – From machine code, to programming languages,
to these like application frameworks. So I kind of think of running your own data center as being
a C developer. You're working in the weeds.

It is full of a lot of very well understood problems and there's a wealth of knowledge the you can
kind of pull from and build on the back of, but it's hard and it's easy to shoot yourself in the foot
and it's easy to make decisions that you feel are the best ones at the time, but are very, very
difficult to undo later if you find a more optimal solution.
I kind of look at things like AWS as being like the abstraction that they built on top of their data
centers, AWS itself, to be like the Ruby programming language. Ruby was written in C. It
utilizes a lot of the functions that C can provide, but it gives you a happier kind of interface to
that. That was like a whole – Like Ruby is a programming language to make developers happy,
and Amazon is sort of an abstraction to make system operators happy.

Most people don't like being in data centers. They’re loud. They’re hot. They're just generally
uncomfortable. So the thought of, “I never have to go to a data center again. I click a button and
I get my server,” is really appealing. Then on top of that, Heroku is more like Rails.
So, yes, there’s still Ruby involved. There’s still the concept of a server and of CPUs and
memory and these things you have to take into account, but I don't have to think about session
management now. I don't have to think about, “Oh no! How am I allocating objects in the most
effective way?” I don't have to think about how do I build an abstraction layer around my
database. These things are available to you and its batteries included. So building a Rails app, I
don't have to think about building a web framework. I can just focus on building an application
that provides value to my business, and using Heroku is similar. You have to think about all of
the things that come with spinning up a server. You just focus on building your apps.

[00:48:35] JM: Do you have any war stories from your experience of helping to build out this
large-scale managed database service? Like any outages that come to mind, or maybe not
outages, maybe just difficult technical problems that were really hard to sort out.
[00:48:55] JD: I have some. I wish I could go into detail about a few, but I can't unfortunately. I
always assume everybody's doing the best job that they can, and that is a really, really difficult
thing to keep thinking of when you have some of these problems that turn into war stories.
I was not at Heroku when the great U.S East One outage happened a while ago, but I was using
a different cloud provider, Engine Yard at the time, and that felt like the world had ended. We
had a customer issue a while ago that I can’t get into details on, but it involved their database
being down for a nontrivial number of days, and there wasn't much we could do, because they
had backed themselves into such a corner.

Those are always a struggle having to sit in front of a customer and tell them that this is not
going to be fixed anytime soon and we're doing everything we can is always tricky. Also, it's
important to remember that software is dynamic. We see a lot of people coming to us needing
help, and the first thing they always say is we didn't change anything, and that is a certainly
useful data point. But it's not really vindication either. If I don't put gas in my car for two weeks
and I start driving to a different city, my car is still going to be broken down. I haven't changed
anything, but something bad still happened, and it's just a data point. Sometimes customers
kind of make this assumption that like, “I run my database on Heroku and I never have to think
about it again, and I can just throw whatever I want at it and it'll just deal with it,” and that's really
not true. These are just servers in the background.

So there's a little bit of back-and-forth that kind of has to happen, and it's important to
remember, when these things happen, we as Heroku, we want you to be successful. Nobody's
out to get you. Nobody's trying to be a jerk support engineer or jerk engineer, and sometimes
when you don't hear a response back right away, it's because we’re just as confused as you
are. We’re not like absolute database masters over here. There are certainly some people that
know more than others, but we’re all just developers trying to do our best.

As far as large-scale incidents too, it happens. We've had full platform API outages before that
sometimes redo the mistakes we've made sometimes due to higher-level issues, and you just
have to do your best and you have to be calm. If you're a service provider like Heroku, we’re
upset when this happens, but you just have to take a deep breath and deal with it.
If you’re a user, if you're running your apps on Heroku and your business is down, we feel your
pain just as much as you do and we want it to be back so badly, but also be calm and know that
we’re trying our best.

[00:51:40] JM: S3 goes down sometimes. S3 has gone down. S3, the entire service, that
underlies so much of the internet at this point. Every time that kind of thing happens, it scares
the crap out of me. It's terrifying, and nobody wants that to happen. As you have already spoken
of, Amazon is the most – Basically, the most reliable cloud provider at least in terms of data that
we have probabilistically speaking, and it still goes down sometimes, because these are really,
really hard problems. They’re getting solved over time and there's a lot of cause for optimism.
But what you said about like sometimes the customer can shoot themselves in the foot and
backed themselves into a corner, like if I do – I can totally do something with Postgres. If I do
something like I try every entry to the write-ahead log. I decide to wipe out my database and
then reconstruct it from the write-ahead log. If I just have some trigger that's trying to do that, I’m
probably going to overwhelm – You can do these infinite loops, infinite recursive, overwhelm the
database very quickly loops.

[00:52:53] JD: You’ll have a bad time.

[00:52:55] JM: You’ll have a very bad time. For you, as Heroku, in general, you want to prevent
people from shooting themselves in the foot, but you can't account for every kind of user
shooting themselves in the foot behavior. To what extent do you want to design the database
service to account for people who may be shooting themselves in the foot?

[00:53:18] JD: Yeah. We have some preventative monitoring in place and we will let customers
know to the best of our ability when we detect bad things. Some customers may have seen
these very handcrafted artisanal emails usually from a support engineer at three in the morning
that are, “Hey, we noticed you’re doing this really weird thing and you're write-ahead log disk is
full. We can’t push it to S3 fast enough.” You’re going to have a bad time if you don't stop doing
whatever you're doing or work with us a little bit here.

So those are tricky. We have some improvements that are coming down the pipeline that are
going to kind of give customers a little bit of a heads up on other data corruption or potential
backlog of archiving takes place. Also, we do make tools available to customers. So if you've
ever run the Heroku help PG or Heroku PGPSQL commands, there are actually a whole lot of
assistive tools in there as well.

One of my favorite that I actually use all the time is PG diagnose, and that will actually run a
series of basic test against your database and give you a really nice sort of ASCII printout of
everything that it detects that could be a problem. So your most expensive queries, indexes that
are rarely used but written too frequently. It'll even give you like index cache hit rate so you can
kind of re-architecture queries a little bit better.

We would like to eventually make this a little more preventative and let you know when we
detect these kinds of differences, but even if after a deploy, maybe 15 to 20 minutes after one,
you just run PG diagnose on your instance. That'll give you a good bit of information about what
could potentially be going wrong and what could be improved.

[00:55:07] JM: Now, you don't work on Kafka or Redis, but you have colleagues within Heroku
who are working on these managed services. These are, to some extent, similar to Postgres,
because it's a manage storage service that you have to maintain high-availability for. You have
to maintain a simplified surface area to allow the developer to operate their own storage service
for all the difficulties of operating your own replicated Postgres cluster. I think Kafka probably
puts it to shame with how difficult it is to operate. Do you have thoughts for what it's like to
operate those other services to manage a distributed system of Kafka or Redis in contrast to
something like Postgres?

[00:55:59] JD: Yeah, they have their ups and their downs. I’ve worked on both of them a little
bit. Redis hasn't really been getting a lot of attention lately unfortunately, but a lot of those
operations are very similar to Postgres with the biggest advantage being that restoring a Redis
backup tends to be a lot faster, because it doesn't have all of the bells and whistles in place of
Postgres, because Redis is really more of a cache or a temporary place to put your data.
So basic day-to-day operations on Redis are almost exactly the same. We even have a bit of a
joke that the tool that we use for Postgres real-time database backups is called Wally, and the
tool that we use for Redis is called EV. So it’s from the Disney movie.

Kafka, that's an interesting beast. So people who haven't ever run Kafka themselves may be
surprised to learn that Kafka itself isn't actually just a single service. It’s actually two. There’s
Kafka on the frontend, which is your message broker, and then there's Zookeeper on the
backend, which handles a lot of the distributed locks and the orchestration and the kind of raw
storage of your data.

So when you spin up a Kafka instance in Postgres, you don’t have to think about how many
Zookeepers do I need or things like that. We handle that for you, but that really means that
spinning up a Kafka instance means we’re managing two different services, and each of those
services are running on a non-one number of servers.

So a lot of our smaller Kafka plans run on three Kafka brokers and then five Zookeeper
instances, I think, which is part of the reason why Kafka plans tend to be more expensive. Then
you get all the way up to like eight or nine Kafka brokers and five zookeeper instances. It can be
pretty intense. One advantage to Kafka though is it's more distributed nature. If you have a
Postgres primary that goes down, you're in trouble. Even having failover kickoff, it's not instant.
It takes 30 to 45 seconds, give or take. But if you have a Kafka broker go down, you can fix that
pretty quickly. Request just stop going to that broker and the other ones pick it up. If the broker
that went down is a controller, then there is a leader election to pick a new controller, which is
the kind of leader Kafka instance. The general disaster recovery operations of Kafka are much
easier, but the overall orchestration and moment-to-moment management of Kafka is a bit
trickier. So it's all about tradeoffs.

[00:58:33] JM: Well, it’s been a great conversation. Let's just wrap up. Are there any elements
of working on large scale cloud provider infrastructure that have surprised you that are
memorable?

[00:58:46] JD: Yeah. I was a Heroku user for years before I joined the company, and back in
2011 when I deployed my first Rails app to Heroku, it seemed like magic to me. As I got to peer
behind the curtain of Heroku a bit, it's extremely impressive that it all works and it still feels like
magic sometimes. But these are all just servers and networking and ultimately people that are
no dumber or smarter than you. Many of whom come from the same background as you who
are just doing their best, and that knowledge that we have just as many rough edges we make
just as many mistakes as you. Sometimes we are just as confused as you and bad things
happen. That was extremely surprising to me, especially coming from someone who works in
like smaller northeastern rustbelt tech companies where we think of like the Silicon Valley as
this like shining hill and that's where Heroku is based out of. But just seeing the general human
element of everything was really surprising and very delighting to me.

[00:59:54] JM: Okay. Well, Jon it’s been great talking. Thanks for coming on the show, and I
remain a big fan of Heroku. I continue to use it to run many of my application. So, nice work.

[01:00:03] JD: Thank you. Thanks for having me.

© 2019 Software Engineering Daily