Getting to the Heart of Twelve-Factor Apps

Narrator
Hello and welcome to Code[ish], an exploration of the lives of modern developers. Join us as we dive into topics like languages and frameworks, data and event-driven architectures, artificial intelligence, and individual and team productivity, tailored to developers and engineering leaders. This episode is part of our Tools and Tips series.

Jon
Hello everyone! My name is John Dodson and I’m an engineer working for Heroku. I’m a big Heroku super fan and excited to talk to you about what awesome stuff is happening at Heroku. And today friends, today is a special day because we’re here to talk about something that’s so uniquely Heroku. That’s right. We’re going to talk about 12-factor apps. Also, I’m joined by someone who’s intricately involved in the heart of 12 factor, Vish Abrams. Hello, Vish.

Vish
Hi, Jon.

Jon
Hey, so Vish, we’re just going to kick things right off. So, if you can tell everyone a bit about your software journey and what led you to your current role.

Vish
Sure. So currently I am a distinguished engineer working at Google on Google Distributed Cloud, but I think about my software journey as really beginning in earnest when I moved out to the Bay area to work at the NASA Ames Research Center on a private cloud for NASA, which was a very interesting journey, to say the least. One thing that might be particularly relevant to you is we were essentially trying to rebuild Heroku at NASA.

Jon
Nice. So, you worked for NASA, and while you were there, you worked on the OpenStack project to build a new compute cloud for them, as you said. So, what are some of the problems you had working for NASA that you apply to software stacks now?

Vish
So, one of the most challenging bits of working in government is the security controls and the bureaucracy.

Jon
Right.

Vish
And that is an area that takes some particularly difficult effort to work through, I think the best way to put it. So basically, you’re, you’re sitting there writing documentation, I remember one thing we had to put together that was for verification and validation it was called, and we had to spend essentially a month with the whole team working on this project that produced a 60-page document that I think ended up being read by one person. And I think my takeaway from that whole experience is to really think about where you’re investing your time and energy, and make sure that the processes that you are working on to keep yourself safe or secure are actually producing value, because it’s very easy to sort of get distracted by a lot of busy work and overhead that doesn’t actually give you value in terms of making you more secure or making you more available or things like that.

Jon
Yeah, that makes a lot of sense. Hopefully that one person that read that really enjoyed that, so yeah. For our careers in software development, it seems like change is happening at a much more rapid pace than I’ve ever seen. And I don’t even know if it’s entirely true exactly, but often it can feel that way when you see an absolutely astounding AI demo where it seems like code is written with magic. I’m wondering what you think folks involved in writing software need to think about and to prepare for to be successful in the industry over the next 10 years?

Vish
Yeah, I mean, this is a watershed moment for the industry for sure. I mean, we’re seeing it reflected in the way that businesses are operating, how they’re hiring, what they’re looking, even looking for. And I think we’re all still coming to grips with exactly what the sort of AI revolution, as people have termed it, is really going to approach. But I think the most important thing that we haven’t figured out yet, is AI is really good at producing the first version of things.

Jon
Yeah.

Vish
And there is this expectation across the industry that, oh, well, you know, if I make my developers 50% more productive, then that’s going to translate into 50% more productive business. And I think the reality of things is that it is never the case. We’ve had many times in industry, you know, when you go back to the manufacturing revolution where we’ve optimized certain things incredibly and it takes a considerable amount of time for the actual business value to catch up, because it’s actually a classic business process modeling problem. If you optimize heavily one area of the business, you might not actually increase the overall throughput. And there’s some really great examples with traffic organization where they’ve discovered that adding a new road can sometimes slow down traffic as a whole. Right? So you have a new shortcut, everybody tries to take it and it pushes the … there are these knock-on effects that cause the other roads to slow down to compensate, and actually can make things worse. There was this study that was done in New York about this, where they estimated that if they took out one of the main roads, it would actually increase the speed of traffic across the city. And I think in software we’re going to see the same thing. So, we’re going to optimize certain aspects of software development are going to come far, far faster, but other aspects which before were incredibly slow are going to become the bottleneck, and we’re going to have to figure out how to optimize those other areas to really get the full value.

Jon
I think that makes sense when I’m thinking about AI and sort of the future of the industry as well, I think that we’ll need to master the details of our domain. So at every sort of aspect of where your business is, and so I think that’s one of the things that I’m trying to think about the most as a developer, which is do I understand all of the important stuff to be able to use AI to make sure that it’s what we really need? When I’m building a feature, or when we’re thinking about something for a customer. And I think as long as we’re like, try to think about for me, and as long as I’m doing that and understanding the details and understanding that I can get AI to do the right thing, then I think, you know, I’m in a safe spot. I also think AI has been really helpful for my journey in so much as understanding a project. You can use it to understand a project so much faster than you could otherwise. And so I think that’s a huge benefit.

Vish
I completely agree. I mean, one really great example is traditionally building software was the slowest part of the process.

Jon
Yeah.

Vish
And it seems like now that’s getting faster. But now the question is going to be evaluating the software that you build. You know, it used to be you had an idea you could produce maybe one or two versions of that idea and decide between them. Now you can produce 50 or 100 versions of that idea. How do you actually optimize the decision-making process on whether you’re building the right thing? I think we’re going to see the industry really transform around what can be built, and I foresee us having a lot more businesses that maybe were not software enabled in the past that are going to become software enabled, because building the simple first version of something becomes so easy. But then the bottleneck is going to be in operating and managing that software over time. Like think about the software explosion which businesses are going to see when…

Jon
Yeah.

Vish
…you know, hundreds of thousands of new developers that previously were marketing people or salespeople or in the other areas of the business, suddenly have access to building software. What’s that going to mean for software proliferation, and how are we going to manage that over time? I think that’s the where the big questions are coming in.

Jon
Yeah, I think that’s a really great point. So, in a blog article you wrote, “From OpenStack’s Ashes: Building a Stronger CNCF,” you know that the Cloud Native Computing Foundation needs to remain flexible to address the risks of technological changes. You also talk about OpenStack and how it lost developer excitement and focus to Kubernetes. Do you think Kubernetes as a technology is set up to handle future technological changes, or at some point, do you think it could be replaced by something simpler? Because you did speak about simplicity in that as well.

Vish
Nothing is ever going to live forever in software.

Jon
Right.

Vish
The pieces of software that we thought were going to be the way it is for all time, clearly it’s not the case when it comes to operating systems. That said, on the other hand, we do find that even these old pieces of software tend to persist in small ways a lot longer than we think they will. I keep hearing stories about people going and writing COBOL. You know, it’s still around in the banking sector.

Jon
For banks or whatever, yeah.

Vish
Yeah. So, we do know that these things will continue to exist.

Jon
Yeah.

Vish
But when it comes to open-source projects, you know, there definitely is a hype cycle, and we experienced a huge one with OpenStack when we put it together at NASA where we had, we went from six developers working on a project, you know, in someone’s house in, in the Mission in San Francisco to over 500 developers across 10 or 20 very large companies within six months, like it just exploded in terms of popularity. And that hype went away just as quickly.

Jon
Yeah.

Vish
I think in general, the CNCF and Kubernetes is in a much more stable spot than OpenStack was. It’s persisted a lot longer, you know, the hype cycle for OpenStack was only really three or four years.

Jon
Yeah.

Vish
And Kubernetes just hit a decade. So, I think it’s got more staying power for sure, and they’ve managed to do a great job of pulling in new open-source projects. This AI change is going to be a big test of that question, right? Is the community going to be able to adapt and sort of subsume all of the AI hype and interest, or are things going to start moving away from that and moving towards building on AI as the next big thing?

Jon
Yeah, I think we’ll see. So 12 factor, can you explain what the 12-factor app methodology is and why it’s still relevant today?

Vish
Sure. So, we might need to start a little bit with the history.

Jon
Sure.

Vish
So for people that don’t know about where 12 factor came from, so it was originally written by Adam Wiggins who was one of the co-founders of Heroku, actually.

Jon
Yeah.

Vish
When Heroku was founded, there was a big transition happening in software. Previously, creating software was sort of an art, especially distributed web systems, which were very new at the time. You needed a very broad range of skills in order to successfully deploy an application. At the same time you had to be a software developer, you also had to be kind of a systems engineer. You had to understand how to keep your operating system up to date, secure it, run your, you know, it was usually a lamp stack or something like that. So, there’s all these pieces and layers that you had to be familiar with in order to produce a distributed system. And then even the knowledge about how to build a system that scaled well was kind of isolated to a few people who had had experience doing it. And at the time, there was this project, Ruby on Rails, that started taking, I think, the software industry a bit by storm in that it simplified the process of getting a piece of web software out there to a great degree. So, it made it very simple for new people to jump in and say, oh, I could do this software development thing. And it did that by sort of obfuscating or hiding some of the underlying layers that you needed to worry about. And Heroku kind of took that a step further and said, well, now that you can build software really quickly using Ruby on Rails, wouldn’t it be great if you could deploy the software really easily onto the web without having to really think about anything but your source code? And in order to make that actually real, there are some principles you have to follow about how you build your source code. And those principles were really being learned at the time and encoded into the Heroku product. And the 12-factor app manifesto was written up as a written codification of what those principles were. So, here’s the things you need to think about as a developer, if you want to build an app that you can deploy easily, that you can maintain over time, that you can scale up and scale down to meet demand, and it really kind of made it clear how to build an application that way. And most of software development, web software development has kind of adopted these principles as general best practices. And so that’s why it’s still relevant, is because that process of writing some code and deploying it to production and maintaining it over time is really similar to how it was even then, you know, over a decade ago. In fact, we still have pieces of software running that were written back then on the Heroku platform.

Jon
Yep, we do. Now that you’re speaking of, you know, Heroku’s early days, I remember being a developer and having to set up Linux and Debian at the time as what company that I was working for is using, and I remember hearing about Heroku and I was like, you just have to push up your code and it takes care of everything else that can’t possibly be real. Like, and I remember working for a startup and then we used it and I was absolutely blown away with what you could do. It really did save us a lot of time, and yeah, it was absolutely incredible, you know, converting everything over and it was great. It’s one of the reasons why I’m here because, you know, Heroku sort of changed my developer journey in a lot of ways. You know, and then also realizing that, you know, I could reach out to a person and they could get back to me and they could talk to me about support in a useful way. I’d never seen anything like it. And so it was really magical in that way, so.

Vish
Just to illustrate sort of the impact that Heroku had, even though at the time it was a relatively small company, I said earlier that when I went to NASA, we were trying to rebuild Heroku.

Jon
Yeah.

Vish
It’s really interesting thing about how that happened. So basically, at NASA, there were 3,000+ websites and they were hosted on all sorts of different systems. Some were on servers under people’s desk somewhere and shared hosting provider, some were managed by third-party contractors, it was just a mess. And the whole goal was like, hey, we could actually make this much more efficient by giving everyone a common platform, and we modeled that platform on Heroku because Heroku just offered that experience so well. Of course, we quickly discovered that Heroku had AWS underneath it, so it was easy to sort of build a platform as a service when you had an underlying infrastructure. At the time, GovCloud didn’t exist, so the whole genesis of the OpenStack project was actually because we needed to build an infrastructure underneath the platform in order to build Heroku again, which is kind of a funny, roundabout way of of backing into building a cloud, which is what we ended up doing as part of creating what’s called the Nebula Project and that eventually morphed into OpenStack over time.

Jon
Fun. So, the 12-factor app manifesto is now open source. Can you speak to why it was open sourced and if someone wanted to get involved, how would you suggest they go about it?

Vish
That’s a great question. So basically, we looked at the 12-factor manifesto and Heroku was maintaining it for a long time. But what we started to see was that while the principles still were largely relevant, there were a few references that were to kind of outdated technology that didn’t really exist before. We were worried that new developers, as they came across the manifesto, would kind of go, oh, this isn’t really relevant to me, this was clearly written a long time ago, and it’s probably not still useful. And so we felt like we needed to give it a bit of a refresh. And the intention from the beginning was never to completely rewrite it or change a lot. It’s to keep the principles, update some of the examples, and then bring in a few pieces of new sort of processes that have come up in the past decade that have become standard, that just really weren’t well represented. I think a great example of that is the 12-factor manifesto talks about logs and sending logs to standard out. But these days, it’s not just logs that you need to worry about, you need to think about telemetry and you think about exception tracing, you need to think about execution tracing and using something like open telemetry is kind of become the standard, and so the intention is to add those types of things where new practices have come out that really complement the existing practices in there that should be mentioned.

Jon
So, keeping with 12 factor now being open sourced, how has thinking about a larger community and different companies’ needs affected 12 factor?

Vish
So, we’ve gotten some great sort of collaboration and contribution from various folks, including, you know, Intuit has been involved, AWS has been involved.

Jon
Right.

Vish
And the great news is there’s pretty strong consensus that the principles are correct.

Jon
Right.

Vish
There’s a little bit of sort of disagreement about the optimal way to implement those principles, but what’s that that’s forced us to do is really focus on the basics and not get into way into the weeds details on what we’re trying to accomplish with 12 factor. So, we’re not trying to come out with a hard and fast set of requirements of your app, must do this, your app must do that in order to be a 12-factor compatible. We’re more focusing on here are the principles, and here’s the different ways that you can follow those principles and still be a 12-factor app. And so that keeps it kind of broadly applicable and relevant across many different industries, many different companies regardless of your approach to software.

Jon
Right. So, Vish this is super important question, just wanted to set the stage with that. So, in the film Back to the Future 2, Marty travels in the future with Doc Brown to change a few things relating to Marty’s kids. During that part of the movie, we see a futuristic 2015 filled with flying cars, hoverboards, and self-drying jackets. Now, obviously, much of this didn’t happen in 2015, however, one fan theory seems to speculate as to why. According to one fan theory, the reason why we have never seen these future tech items in our universe is because Marty and Doc never altered our past. Because if you remember in the original Back to the Future, after Marty changes the past in 1955, his father, George McFly, becomes a science-fiction author. This fan theory surmises that because our universe didn’t have a George McFly authoring science fiction, our scientists were not inspired to create all the future tech such as flying cars and self-drying jackets, because we don’t have that. So, you know, I’m not sure there’s really a question here. I just love this fan theory, it was very meta and also hilariously strange, and I suppose if there is a question here, has a book, movie, or song inspired any software, anything you’ve ever created? Just curious

Vish
In general, yes. So, I’m a huge fantasy nerd.

Jon
Right.

Vish
So, I read like Tolkien and then George R.R. Martin.

Jon
Oh yeah.

Vish
Jordan, Brendan Sanderson, like I just really get in there. But I also went through a phase right when I was getting into computers and going more into the sci-fi side.

Jon
Okay.

Vish
And I read a bunch of Asimov’s stuff, particularly the Foundation series.

Jon
Oh yeah.

Vish
And the I, Robot, I can’t remember the name of the overall series that that was called and the follow on. And I love the way that he tied everything together. So, this sort of really interesting thing about that, I mean, there’s a bunch of really interesting future thinking around AI and…

Jon
Yeah.

Vish
…controlling the future and preparing for disaster and trying to do the right thing. But I think the most interesting bit for me was, you know, there’s these expansive trilogies that are seemed like they’re totally disconnected and at some point, he manages to tie them together, and you realize suddenly that this is the same universe and vastly separated in time. And I wouldn’t say that that has particularly inspired a piece of software, but it has encouraged me to always look back at all the varied things that I’ve done.

Jon
Yeah.

Vish
And to try and bring them into the value of the things that I’m building now. And I think I’m lucky to have worked in many very sort of distinct areas over my career where, you know, I was working in cloud and then I was working in containerization, and then I was working in AI and platform and being able to sort of reach back into those previous projects and pull pieces of value and apply them to the current project is something that I feel like I learned from that brilliant tie-in approach of sci-fi writing.

Jon
For me, I’ve been super inspired by sci fi as well, especially in a particular show like Star Trek: The Next Generation with how they view the future and, you know, humankind being sort of benevolent and solving a lot of social problems and sort of moving forward in this really sort of rightminded approach, which I find really inspiring. I think that was sort of a lot of my North Star when I was thinking about how people could move forward and stuff like that. I think, you know, you’re talking about AI and you’re talking about, you know, how culture has talked about AI for so long, I think culture is really good at freaking people out about AI. You know, killer robots, that it could be some of the most dystopian stuff. I think I’m sort of, for lack of a better word, done with all that and trying to think about how it could possibly be helpful, because I think that it really can be.

Vish
100%. I mean, the capabilities, if we’re just starting to discover this, right?

Jon
Right.

Vish
I’m thinking about the way that I use assistance with AI is still not like, within the next couple of years, I can imagine us having sort of a persistent assistant that you’re working with that’s just, you know, you go in and you talk to ChatGPT or one of these services, and in many times it feels like you go in with a particular goal and you’re kind of starting over the conversation every time.

Jon
Right, right.

Vish
But I think we’re not very far off from having something that can keep the context of previous conversations and work with you where you say, you know, oh, my wife needs me to organize dinner. It will know who your wife is and be able to help you figure out where to go or if you if you’re doing research on, I use deep research type approaches all the time to learn about new developments. I was just actually doing one yesterday when there’s this old paper that I read about how they were worried that the deployment of 5G was going to make our weather prediction get much worse. Some places estimated like NOAA estimated it was going to be 30% worse because the 5G signal leakage actually prevents the weather satellites from detecting the moisture in the air properly.

Jon
Huh, interesting.

Vish
And I had been thinking about this for a long time, and every time, you know, you go out and the weather prediction’s wrong, I would go, yep. See? It’s happening. And then, I mean, I needed to take a look and see if that was actually the case. And I did some deep research, and it turns out that there haven’t been any follow ups to actually prove that it affected it. So, they apparently worked around some of the issues and it managed to minimize the impact, so, I can’t blame bad weather predictions on 5G. Unfortunately, I’m sure my conspiracy theorist friends will be very disappointed.

Jon
There were a lot of conspiracy theories around 5G. I hadn’t heard that one, that’s interesting. So, we’re talking about 12 factor, and I’m curious, are there any new factors or principles being discussed, or is the work primarily focusing on the old factors?

Vish
Yeah, so there are a couple. We have a sort of North Star of keeping the number of factors to 12, not because there’s anything particularly special about that number, but because that’s how everyone knows it. And it would be confusing to people to go to the 12-factor site and see 13 or 14 factors.

Jon
Right.

Vish
And so we focused on one area where we think we can combine a couple of the factors to make room for another one.

Jon
Okay.

Vish
And there’s been a proposal that I’ve been working on around including identity and particularly workload identity in the factors, because I think it’s a very common problem that systems have to deal with that we could make much better if there was sort of a consistent way of doing it. In many ways, historically, the 12 factors, while they also exemplified practices that were in place, they were also used as a way to sort of push different communities, including framework builders and software engineers, to sort of collaborate on creating a shared way of doing things, like we did this with the port environment variable in the early days where we got the concept of being able to listen on its port specified by the platform into the 12-factor manifesto and into, like the Ruby libraries, so that customers wouldn’t have to think about it, it was just sort of built in. And so the identity factor is one way that we’re thinking of can we push everyone towards a consistent view of how to do workload identity? That one’s in process. The rest of the updates that we’ve talked about have more been including additional things in the factors, like I mentioned before, observability being added into the logging factor and then cleaning up and updating just some of the examples. This is the main areas that we’ve been looking.

Jon
Yeah, that makes sense. I think, you know, just thinking about this for five seconds, obviously this is going to be an amazing solution, but you could just make the 12 factor, it can be 12.1, 12.2, 12.3.

Vish
Yeah.

Jon
And there you go, Vish you’ll never end. You’ll be, it’ll be awesome.

Vish
The hundred factors in 12 categories.

Jon
Yeah exactly. Exactly. Isn’t that how software engineering works? So, some developers may have applications that don’t quite hit every factor of the 12-factor app as these developers work to migrate over to these best practices. Have you seen developers fall in any traps or common problems when trying to do this that they should keep in mind while moving forward?

Vish
I think one of the biggest challenges, and this happened actually, so 12 factors predated containerization or at least generally understood container containerization through Docker and Kubernetes. And one area that Docker and Kubernetes specifically moved away from the 12-factor principle, is they tried to enable stateful applications as well as stateless applications. And there are good reasons to put stateful applications in containers and to make them more consistent. But what I’ve seen happen as a result of that over time is that people don’t carefully think about how to separate the stateful parts from the stateless part of their applications. So, 12 factor is really about the principles for building stateless apps, you know, and the idea is that stateful apps are handled separately and they’re handled via sort of some sort of add on mechanism or what we call backing services in the 12 factors, and those backing services can also be stateless, but in many cases the backing services are where you’re storing state. And a lot of the value that you get from 12 factor is being very clear about separating the stateful from the stateless parts, so that you can scale the stateless parts up and down very quickly, and you can take advantage of disposability and things like that. And where I think people get stuck a lot is not really committing to that approach and kind of mixing the stateful and stateless parts of your app, and that is where you end up with scaling issues and recovery issues over time, because you haven’t made a clean distinction there.

Jon
Great. So, you’ve written and spoken about updating 12 factor for a few years now, and one thing you’ve noted is updating it for the next decade. How do you do that when you don’t know how the future is going to turn out?

Vish
So, there’s something that we’ve really started investigating recently, which I think is key to that particular approach, which is a bunch of people have asked, well, is there a 12 factor for AI? Right?

Jon
Right.

Vish
And there’s been some attempts to write something like what a 12-factor app that has AI capabilities would look like. And surprisingly, or maybe not so surprisingly, it really doesn’t look all that different from a non-AI application.

Jon
Right.

Vish
I mean, there are some things you need to think about with state in AI applications as well, but the same principles seem to apply, like whether you have AI capabilities in the application or not, there’s still scaling rules that you want to think about. But I think a much more interesting area for the next 10 years of 12 factor is at the time when 12 factor was created, it was about a new world of developers coming into software development through Ruby on Rails. We’re in a similar space now. There’s a whole new world of developers coming into software development through AI coding tools, and the principles of 12 factor were designed to help those new developers or people that haven’t built distributed web applications understand the principles that they need to follow in order to make their system work over time. And we have a whole new group of developers reaching those same challenges, having to think about the same things. So, we need to think about how we can get that 12-factor knowledge to that group, and I think the way to do it is actually to encode it in the AI generations themselves. So, really we need to think about how to instruct agents that are helping build these applications to follow the 12-factor principles, because if you ask a coding agent to build you a snake game or something along those lines, it will produce a snake game that actually does what you want it to do very well. You’ll be able to move the snake around and capture… eat things and get longer. All that will work. So, the functional requirements of your ask will be met very easily. But if you can imagine the entire space of all the snake games that could be built, say, you know, these models are not deterministic, right? They follow probabilities. So, you can get a thousand different versions of that snake game. All of them would meet the functional requirements. But of those thousand, maybe only three are built in a way that are easy to maintain or over time, right, that are easy to scale up or you know, that are…

Jon
Right.

Vish
…embedding those 12-factor principles in how they were built, and so really, what we need is to congeal the principles into a form that agents can use. And the good news is agents are really good at human language. So having one that’s good for humans is also good for agents by default. But it does need to be customized a little bit so that those rules or guidelines are encoded into sort of the rules that the agent is following, and that’s an area that we’re looking at, and I think that will more than anything, impact the next decade, because we’re going to see more and more software built using these tools.

Jon
I agree. So, Vish, we’re finally at our last question here before I wrap things up. So, speaking of the future, imagine a world where the 12-factor app refresh is largely complete. What do you hope it signals and inspires in a larger software industry?

Vish
Most of all, I hope that people, when it’s complete, still look at it and go, yes, this is super relevant.

Jon
Yeah.

Vish
And that it really, really exemplifies what people think about when they’re building applications. And then more so I hope that it helps combat that software proliferation problem that we talked about.

Jon
Yeah.

Vish
Right? That it gives some consistency to the incredible new wave of software that’s being built. That’s going to be probably ten x or 100 x the previous wave of software development. If it can help create that consistency, then I think it’s been successful.

Jon
Awesome. Well, I hope it is. So, thank you very much, Vish, for talking to me today.

Vish
You’re welcome. It was great to talk.

Narrator
Thanks for joining us for this episode of the Code[ish] podcast. Code[ish] is produced by Heroku, the easiest way to deploy, manage, and scale your applications in the cloud. If you’d like to learn more about Code[ish] or any of Heroku’s podcasts, please visit Heroku.com/podcasts.