Looking for more podcasts? Tune in to the Salesforce Developer podcast to hear short and insightful stories for developers, from developers.
46. Go at Heroku
Hosted by Johnny Boursiquot, with guests Edward Muller and Rishabh Wason.
Many organizations and teams have adopted Go for its focus on concurrency and efficiency, and Heroku is no different. Although it's no longer a "new" language, diving into Go can be intimidating, whether you're a seasoned programmer or a new hire. Johnny Boursiquot, Ed Muller, and Rishabh Wason discuss the joys and challenges of learning Go, applying it to projects at Heroku, and teaching colleagues how to write idiomatic code.
Johnny Boursiquot is an SRE at Heroku, and joining him on this episode are Ed Muller (one-time Go language owner at Heroku), and Rishabh Wason, an engineer fresh out of university. Ed initiates the conversation by talking about how Heroku rolls out buildpack updates to users that are concurrent with Go releases.
Heroku is a polyglot organization, and Go is being used as one of its four primary languages. It's finding its way into backends, microservices, and services which communicate with each other. Since many engineers at Heroku have experience in multiple languages, it's become essential for Go experts to teach others how to write idiomatic Go code. Part of this is done through the Go Design Guide, a living document that details the pros and cons of various ways to write Go logic. Ed in particular finds that pairing with other engineers on their issues has helped him understand a beginner's mindset, which he can then use to update internal documentation.
Rishab also shares his strategies for how he learned Go, coming at it with prior experience in Java and Python. He provides a list of online resources which helped him. He also talks about some of the discrepancies he found in different code bases--for example, in their different dependency management styles. For him, the way in which he grew his understanding of the language was to ask targeted questions to reviewers in his pull request.
Links from this episode
Johnny: Hello, and welcome to Code[ish]. My name is Johnny Boursiquot and I will be your host for this episode of Code[ish]. Today, we will be talking about Go at Heroku. I am pleased to be joined by two of my colleagues, Ed Muller and Rishabh Wason, who will share their experiences with the language, and add some color to how they've seen it used within the organization. So, let's begin with you Ed. How long have you been at Heroku, and in what capacity?
Ed: Let's see, I've been at Heroku now for eight years, and during that time I've been on well, pretty much all the teams. I started on our database team, and transitioned through some infrastructure teams that worked on AWS, and eventually metrics and logging. After that I wanted a little bit of a challenge, and became the Go language owner, and eventually though found my way back to metrics, and now into SRE land, where I'm working on a Go tooling, some open telemetry stuff, and generally observability.
Johnny: But Rishabh I know even less about you, and what you've been up to at Heroku. So how long have you been on Heroku?
Johnny: Very cool. So as for me, I'm relatively new into Heroku still. I'd say at least relative to the two of you. It's been what? Maybe my six... I think I'm entering my sixth month at Heroku? I joined as an SRE, and Ed and I are also part of the SRE org within Heroku. And this has been an awesome journey for me.
Johnny: So Ed, I know you've been at Heroku the longest, so I know internally right now that you are definitely one of the people who looks after the buildpack. So, whenever there's a new version of Go that comes out, you are one of the first people the org looks to, to make sure that we're running the latest and greatest. We're patching, and we're basically keeping up with the releases of Go.
Johnny: Can you give us a little bit of insight into what the process is like? Is it something that's difficult to do, hard to manage? Do you have automation around it? How easy is it to keep the buildpack fresh, and serving our customers?
Ed: Keeping Go up to date on the buildpack itself is relatively straightforward. We have a little script inside the buildpack repo itself, that we can use to do the work, to pump versions. And then things like CI and CD for the most part validate that the existing tests and everything are good.
Ed: There are challenges supporting every new major release, so like Go 1.12 to Go 1.13 et cetera, et cetera. For instance, the defaults is currently still one of the Go 1.12 lines, because there are a whole bunch of implications with turning to modules being the default on... And yeah, we're not yet sure we understand all of the implications of it from the standpoint of the buildpack anyway. And how that affects all the users of the buildpack. But the general process of bumping for point versions, and things like that is really simple.
Johnny: Pretty much since I've been in Heroku every new project, it sounds like it's being written in Go. But we still have a lot of projects that had been written in Ruby. Some new projects are either Ruby, and we have some Elixir as well.
Johnny: So we're truly a polyglot sort of organization. We have a set of go-to languages that we use, but Go is increasingly being one of the first ones we look at. Especially for backend kind of systems, and services that needs to talk to each other, that kind of thing. So how we do metrics, and how we do... when folks hit the API at heroku.com, most of that stuff is in Go. Private Spaces are run that most of that stuff is controlled in Go. So we have a lot of Go.
Johnny: So does that mean we have a lot of Go expertise in-house? And along those lines perhaps Rishabh, you can sort of chime into this one. How easy has it been for you, right, coming out of school and this being your first job? Were you doing Go at school, or did you learn Go on the job? How easy has it been for you to sort of learn Go, and be productive with the language at Heroku?
Rishabh: At college actually I never even heard of Go. The main languages that I used was actually Java and Python. And coming to Heroku, and starting my first job, it was quite a bit of a surprise. Just hearing that I would be working in Go and Ruby, and learning a new language.
Rishabh: So specifically there were like a couple of resources that really helped me gain a bit of an understanding about the Go ecosystem, and learn how I can actually start writing idiomatic Go. For example, the runtime team, specifically we have a program called Runtime University.
Rishabh: And it has a huge section on Go, which it also references Go by Example, and other resources where an engineer who is relatively new to Go, or wants a refresher, can actually go in, and do some practical exercises and learn. For example, syntax about the language, or how to actually write good Go.
Rishabh: Also, for myself, one of the biggest things that has really helped me learn Go at a good rate, is reading a lot of Go code. So, taking an existing microservice that we have, and then reading it end-to-end really helped me understand how to build a service in Go.
Rishabh: And then also when I'm making changes to an existing service, reading that whole code base was very helpful. So, I think overall as a new engineer, we have some patterns at Heroku such as Runtime University that really support a new engineer to learn some Go.
Johnny: So one of the things that you, as a Go developer start to hear as a new Go developer, you start to hear more and more of this concept of idiomatic Go, right? So you could write Go that looks kind of like Java a little bit, or some other language that you're familiar with. But you wouldn't be doing it the way most Go developers do. Right?
Johnny: And this is something that basically is more subjective than it is a specific set of... If you don't write it a certain way, then your code will come back kind of thing. But it's more along the lines of this is the expectation right out in the Go community. When you look at a Go code base, and Ed, I know you've been the leading person internally behind our Go Design Guide. Maybe you can add some flavor there, that says basically how much of the community's concept of idiomatic Go... How much of that basically is impacting, or influencing how we approach Go development internally?
Ed: I like to think of our Go Design Guide. First of all, it's a living doc, right? So, I expect it to evolve over time. It's an attempt to basically try to get all the engineers on the same page about how to structure projects, and how to write Heroku's version of idiomatic Go.
Ed: I do feel that a lot of things captured in that Guide are inspired by the things you see in the community at large. For instance, the Guide has many call-outs to everything from Dave Cheney's blog, to Jada's blog, to the Go Wiki, and things like that.
Ed: And usually has texts for purposes of clarification, or to better relate the contents that are linked out, to the context of Heroku, and the types of services and tools that we need to write.
Ed: And like I said, some of the main motivation for me, starting at several years ago was, as the Go language owner then, I would read a lot of Go code internally, and different projects were trying different things, which is something we want to continue to encourage.
Ed: But as the code bases go from experimentation phase into production, and things like that, in order to have that code base easily maintainable and modifiable by a large swath of engineers, my opinion is that it's better if we do Thing X in the same way across as much of our code Go basis, as possible.
Ed: And the attempts for the design guide is to state what those ways are, and the reason and motivation behind why we feel that is the right way for us to do it. And to provide examples, and all that context for posterity.
Ed: And again, since it's a living doc though, it doesn't mean it's going to all, like if we say do this for these reasons, it doesn't mean in six months from now, it might not give different advice for the same thing. Because we've learned something in the interim, and we have a process in place through an internal RFC, or request for comment, process to modify that doc.
Johnny: Mm-hmm (affirmative). Rishabh, I know you want to add some flavor to that.
Rishabh: Yeah, so this is actually something I forgot to mention earlier. The Go Design Guide for myself, has actually been a fantastic resource as a new engineer. I think, for example, when I start working in a new code base with Go, learning the language as a beginner, you're always looking and asking around for patterns or questions around, is the way that I'm writing this correct?
Rishabh: And does it make sense? And the Go design guide being there as an artifact that you can look at, and see patterns and standards, which to most people make sense, provides a sense of proof, or like prior art to say, "Hey this code makes sense because the Go Design Guide references this pattern. So, I'm going to implement it this way."
Rishabh: And also, when doing code reviews for example, I also referenced the Go Design Guide as a reason as to why certain code should be written in a certain way. So it provides a lot of examples, and provides a lot of explanation as to writing a good standard of Go.
Johnny: So, as someone who say, sort of learned Go on the job, I do want to sort of ask though, as a newcomer to go, what have you found to be the hardest thing to learn about the language? What would have been your stumbling blocks? What things you keep coming back to, and over and over, that it has taken some time to sort of sink in for you?
Rishabh: Yeah. So I think there were a few things that I found difficult along the way when learning Go. One of them, was actually dependency management. So different code bases, used different tools to manage their dependencies. Some code bases that we have, use dep. Others, use Go mod, or modules. So initially when I was starting, I had no idea how dependency management worked in Go. So knowing when to use either, was a big issue.
Rishabh: Also, a difficulty along the way that I had, was knowing that the code that I was writing was idiomatic. So I would look to other resources, such as the Go Design Guide, or Dave Cheney's blog, to really validate whether or not the code that I was writing, was idiomatic.
Rishabh: And another issue, or another difficulty along the way was knowing if the pattern that I was using to solve a problem was appropriate at the time. And it's very similar to writing idiomatic Go, but knowing if the solution that I had was appropriate.
Johnny: And have you found basically team members to be a go-to resource as well? So how much of a face-to-face have you had to do, right? And how much of it do you think you spend, in pull requests, going back and forth? Did you find... Have you thought... I guess what I'm asking, what has the experience overall been like for you? For somebody who didn't know Go before, and has sort of been learning on the job?
Rishabh: Yeah, definitely. So I think being very new to Go, I'm just asking a lot of questions, and always questioning why something was done, the way it was done. So just being open to asking a lot of questions, especially to senior engineers who've been writing Go for a multitude of years, and specifically in pull review or pull requests, asking targeted questions to the reviewers, and saying, "Hey, does this section of code make sense, based on this pattern that I found here?"
Rishabh: And really soliciting specific feedback. So, yeah. I also definitely took advantage of pairing, while doing code reviews. So, we would hop on a call together, look at the code together, and I would explain, and do some kind of a walk-through, explaining the reasons why I took, when writing a certain piece of code. And really got a chance to add some more flavor to the code review process.
Johnny: So Ed, I'm curious. Do you think your journey, I mean you've been doing Go for many, many years. Where do you see yourself at this point in your career, in terms of your expertise with Go? Do you still think the language has a lot to offer you? Do you still think there's a lot to learn? Has the way you've written Go along those lines, has the way you write Go today, has that changed over the years?
Ed: Oh, it's totally changed. Go's not my first language. Maybe my seventh, eighth, sixth? Something like that, language, professionally. I did what everybody does, or at least I assume what everybody does, when they first learn a new language.
Ed: And that is mix in styles picked up, and/or used in the language that they used previously most. Previous to Ruby for instance, I had used Python a lot. So when I first came to Ruby, I wrote my Ruby like Python, and was quickly yelled at by a bunch of Rubyists. So when I came to Go, I wrote what I currently called Guby, the mish-mash of Go and Ruby, which I've seen plenty of. I've also seen plenty of Gython, I've seen plenty Gava.
Ed: I've seen all of these, and I'm not, I don't say that to cast derision on the people doing that, but I believe it's also important to recognize that's part of the process of transitioning, and learning a new language. As far as a new people coming into Go though, I think the best thing we can do for new engineers, is pair with them, and pair with them on specific tasks.
Ed: I have offered, and continue to offer pairing with any engineer, on any project, and those who have taken me up on it, the sessions have been most productive. I believe for myself as well, experiencing the beginner mindset, which I'm so removed from now.
Ed: So experiencing that firsthand. And then I hope for them as well, because through that process of pairing, we can explore the design philosophies, that I think are idiomatic Go, where some early attempts at this people I was pairing with, attempted to use me as a Go encyclopedia. And I feel like neither one of us, neither party, came away from that experience feeling they had gotten good use of that time.
Ed: So that's something else I recommend for more senior Go engineers. When you offer something, or do pairing with somebody, try to do it on a... Like have a specific focus, and a goal to accomplish. Refactor a new, implement something new. Both of those, whatever. Not just like, "Hey look at this code, and tell me how it could be better." Because there's no absolute right answer to that. It involves too many contexts that are specific to the project. How the deployment, the engineer's involved, et cetera, et cetera.
Johnny: And where do we see Go going at Heroku? Obviously we've already mentioned that a lot of our control playing software is written in Go. And everywhere I turn, I think... And you know within the organization, I see more and more Go. Do you think that's the language, the bulk of the engineering teams at Heroku are going to be writing software for your foreseeable future, in your mind?
Ed: I'd certainly like that. I suspect a lot of Rubyists might be more immediately attracted to something like Elixir. That's a theory. Not something I know. But I think Elixir can give the Rubyists, some of the advantages that you would get from Go, while keeping some of the things that they like about Ruby, and those are the same.
Ed: A lot of those things are the same things that kind of drive me crazy about Ruby, and Ruby and Go are different. With that said, I do think Go is embedded into our infrastructure, to the extent that it will likely continue to be one of one to four primary languages, that Heroku ends up using. Especially while things like Docker and Kubernetes, and stuff like that, continue to dominate mindshare in the industry.
Johnny: So Rishabh, what are you most looking forward to on your end with the gloss of working with Go? Any project you get to work on, that involves Go you're happy with? Or do you have a specific kind of project that you find the most fun to write in Go? Where's are your head at?
Rishabh: So, currently I have the pleasure of actually writing a new service, that actually deploys to our Kubernetes clusters in a secure, and scalable fashion. And that service was done in Go. So, it's been a journey for myself in the sense that, it's a foundational project, where I get to actually bootstrap a new service in Go, and really figure out for myself, good design patterns, and writing good Go code. It gives me the space, and the time to actually experiment and learn, which I'm pretty happy about.
Johnny: Well, I am glad we're able to have this chat. And I personally learned a bit more about how Heroku uses Go, and knowing that there are folks who are brand new to Go in your organization as well, that are learning how to write Go, and how to write idiomatic Go. And folks like Rishabh, and also folks like Ed, who have been doing it for a while, providing some guidance.
Johnny: And me, having joined the organization, and knowing that Go was one of the determining factors for me joining the organization. I'm glad to see that there's a lot more to the org, a lot more to the services, and everything else that we're doing that's written in Go, and going that makes me happy. So, I'm grateful that we're able to get them to have this chat, and thank you for being a guest on our show today.
Rishabh: Yeah. Thank you. Thanks for hosting.
Ed: Thank you Johnny, for having me.
A podcast brought to you by the developer advocate team at Heroku, exploring code, technology, tools, tips, and the life of the developer.
Site Reliability Engineer, Heroku
Johnny is an SRE at Heroku, runs the Baltimore Go user group, leads the Baltimore chapter of GoBridge, and teaches regularly at conferences + meetups.
Principal Member of the Technical Staff, Salesforce/Heroku
Edward is a Principal Software Engineer @ Salesforce's Heroku. He is involved in Open Telemetry, Golang, Gobridge, Logrus, and other OSS projects.
Software Engineer, Heroku
Rishabh is a software engineer on the Heroku Runtime team.
More episodes from Code[ish]
Robert Blumen and Marcus Blankenship
How can developers learn from catastrophic errors such as airline disasters? Learn how understanding the root causes of failure in complex systems can prevent their recurrence. →
Karan Gupta and Marcus Blankenship
How can applying the right technology choices at the right time impact your coding and business choices? Karan Gupta explains how practicing “pragmatic engineering” can have an oversized impact on business and business efficiency. →
The episode focuses on managing a certificate authority (CA) within an enterprise. The internal CA is compared on many points to PKI on the public internet. →