Looking for more podcasts? Tune in to the Salesforce Developer podcast to hear short and insightful stories for developers, from developers.
Special Episode: When Giving Back Saves 1000s of Jobs
Hosted by Greg Nokes, with guests Garrett Thornburg and Brett Allred.
In order to support small businesses affected by the COVID-19 shutdowns, the government launched a loaning program to disperse funds to help businesses keep their workforce employed. This program turned out to be extraordinarily popular, crashing the government website and using up all of the funds. MX Technologies was tasked to come up with a more automated solution, and over one long weekend, got an app running on Heroku to process applicants faster. Greg Nokes, a Master Technical Architect at Salesforce, interviews two members from MX, Garrett Thornburg, an engineering team lead, and Brett Allred, its CPO, on how they pulled this off.
Greg Nokes is a Master Technical Architect at Salesforce. His focus for this interview is on two members from MX Technologies, a fintech startup that helps financial institutions: Garrett Thornburg, an engineering team lead, and Brett Allred, its CPO. A consequence of COVID has been an increase in unemployment. In response to this, the U.S. Government came out with a paycheck protection program, or PPP, to lend money to small businesses to pay their staff. The program turned out to be incredibly popular, as banks and credit unions were inundated with applications to apply for these PPP funds. Many of these loans were processed by hand, and banks turned to MX Technologies to help process these faster.
The technical challenges were daunting, and due to the urgency of the situation, the team at MX Technologies worked on a solution over three days to try and get an app out there. Choosing Heroku as a basis for the tech stack was not a difficult decision. They were able to code something up using Rails, and an automatically scalable Redis and database system. The only concern was that because they were working with financial data, they needed to ensure that whatever system they used met rigorous security guidelines. Naturally, Heroku did, and it even passed third-party penetration tests, instilling confidence in their use by the government.
The MX team decided to open source their app and the Terraform scripts used to generate it because they want to make sure everyone is able to benefit from this essential program. Through their efforts, they've been able to help distribute $5 billion dollars, helping to save thousands of jobs and businesses.
Links from this episode
- MX Technologies helps banks and customers manage their finances
- The open-sourced SBA Load Processing Portal powered by MX.
- The U.S. Small Business Administration is responsible for distributing loans
Greg: Welcome to Code[ish]. This is Greg Nokes, master technical architect for Salesforce. I've been with Salesforce for about nine years, working on Heroku the entire time. Focused on customer facing architecture, helping folks use the platform. So I'm always super interested to talk to people about how they're using Heroku and the platform in general. Today we have Garrett Thornburg and Brett Allred from MX Technologies. We're going to talk about what they're doing on the platform. So Garrett and Brett, could you introduce yourselves please?
Garrett: My name is Garrett Thornburg. I'm a team lead on the engineering team at MX.
Brett: And I'm Brett Allred, the chief product officer at MX.
Greg: So tell me a little bit about MX Technologies and what problems you guys are addressing in the marketplace.
Brett: Yeah, sure. So MX Technologies is a Utah based company. We primarily have served financial institutions. So we've been around for about 10 years and we got into the market building tools for banks. So we build software that helps financial institutions, ultimately with a mission to empower the world to be financially strong. So even pre-COVID and everything that's happened as a result economically, we've spent the last 10 years as a company trying to find ways to help individuals strengthen their finances to become financially stronger, to manage their money. But the approach that we've done is by building software for banks and for credit unions and then the banks and the credit unions then offer those tools to their members.
Brett: As COVID happened, as unemployment rates started going up and we were looking at what the Fed was doing with interest rates, and they acted very quickly to take them down to zero. And we were looking at what the economic impact of this situation was going to be. The government came out and said, "Look, we really want to try to keep people employed and we know that as businesses get shut down because of this virus and the situation, we want to do something to help those businesses retain their employees. We want to have somewhat of a safety net for these small business owners because they're really the backbone of the economy." And so they decided to do this paycheck protection program with the idea that they want to protect paychecks. And so as we got into the situation, we started hearing about the rollout of this paycheck protection program from the financial institutions that we work with.
Brett: And because of the popularity of the program, banks and credit unions were, they're getting inundated with applications to apply for this small business loan, these PPP funds. And really I think most people were applying as you know now because there was a part of this PPP loan that was going to be forgivable and maybe even all of it forgivable. And so, assuming that you were to use the funds primarily for payroll to protect paychecks like was the intent, you could also use it for rent and some utilities, but it was really there to keep your people working and keep them having a paycheck.
Brett: And so we saw the problem that was existing in the market and in the United States, then we started hearing about it from our financial institutions and we thought, "You know what? This is a good opportunity for us to really fulfill our mission as a company to empower the world to be financially strong. Having a paycheck and protecting paychecks I think is really important. And then also helping those small business owners." So it met the mission and then two, our clients are primarily banks and credit unions, so we could also help them at the same time.
Garrett: Yeah. We have a little over 2000 banks and credit unions on our platform today and anything we can do to help them, we are definitely game to jump in on. And it was interesting because when the SBA first launched the PPP program, it was all manual data entry. There's no sense of automation. And that's funny when you think about the millions of Americans who needed immediate relief for their businesses which have typically at most 20 days of cash in the bank. Of our pool of institutions who are asking us questions about if we're going to do anything, they said, "Our initial problem is that we have tens of thousands of applications for our customers and we have three employees at the bank who can submit these loans for approval."
Garrett: And so it just became this overburden on their loan team to get these approved in a timely manner. And on top of that, the SBA was constantly changing the applications. So what was on paper was not what actually reflected on the website which meant a lot of phone calls at strange hours calling these people who are applying, these applicants for these loans saying, "Hey, we need this other additional information. How fast can you get it to me?" And so it was 30 minutes plus to get this loan from paper on the desk of a loan officer to approved by the SBA, it was very painful. So they reached out to us to see what we could do to help automate that process.
Greg: How did you guys choose Heroku for this and what value do you think it brought to the table?
Brett: Early on when we decided we were looking at the problem, it was actually a Friday afternoon when we got on the phone with a particular financial institution that was talking about this problem. And they have this backlog of applications that they needed to submit. And the word in the back room if you will, was that these PPP funds were going to run out extremely quick.
Greg: Mm-hmm (affirmative).
Brett: So we're talking to this financial institution, we understand the magnitude of the problem. And we know we basically have a weekend to get a solution up. There's a handful of us on a Friday afternoon and we're like, "Okay, well, we're working all night for the weekend and we need to get something up Monday morning so we can start processing these loans in some type of automated fashion and really help these people before the funds are gone." And so when you start looking at what you need to do to innovate rapidly, and we start choosing technology stacks, both on the software side and the hardware side, you can build a Rails app, push it to Heroku in a matter of minutes, and you can scale it the way that you need to scale it pretty quickly.
Brett: And with the whole marketplace of add-ons, it became really easy when we need to do asynchronous processing. It's just like we can add a Redis server, right? We can scale up databases and there's a lot of stuff there that ultimately when we decided, "Hey, do we need to do this from roll our own or if we could look to a platform, this platform as a service concept that Heroku has." It was really for quick iteration, quick innovation and getting something to market very quickly that is absolutely critical, seemed like a no brainer decision to me.
Garrett: Yeah. At MX, we don't typically use a cloud solution because we'll have PII data and we have to essentially make our entire production environment a PCI zone. So in order to push new features, it has to meet all these security requirements. And so we were used to looking at products, anything we're going to use outside of our ecosystem with those security eyeballs. And so that was one of the first concerns with looking for a place to push it. We're like, "Okay, we know we can probably make AWS work because there's plenty of startups who've been able to achieve that." And then honestly, we looked at doing due diligence. We saw all of the compliance that Heroku had and we said, "Is this enough of a security story to get this product live?" And we found out it was and were able to conduct audits.
Garrett: Some things like encryption at rest with Postgres was very valuable. And then on top of that, any engineer who has to work with Terraform plus AWS knows you're going to have 15+ files of boilerplate just to get a hello world across a load balancer. And for us just to say, "Hey, what's that? We're doing a Rails app. Also, we need ImageMagick and we need a Chrome Webdriver. Go."
Greg: So you guys are using Terraform plus Heroku. So you're using Terraform to manage Heroku?
Garrett: Yeah. So we're actually using Terraform to manage Heroku as we bring institutions on. It's just really easy to add another plugin on Heroku and share the code that provisions the environment. And it's only a matter of minutes to take a client from phone calls to live in production.
Brett: And I think that's important because you think of again, taking something to market quickly. And if we had to build this, there's a couple challenges that exist. One, the multi-tenant nature of the solution and having to build that in and manage that. And then we have isolation of data. And so we have one institution that doesn't want to share, mingle their data in a database with another institution. And so we were able to actually build it in a very simple way that kept everything... Each institution isolated with their own application servers and web servers as well as database servers, but using Terraform with Heroku, with Garret's work that he did. It was really amazing. A new institution wanted to come on, we added a couple configuration settings into Terraform and they were spun up and running extremely quick. And I don't think we could have done it as fast as we did and responded as quick as we did if we didn't have the solution that we chose with Terraform and Heroku and some of these other technologies.
Greg: Tell me about what Heroku products are you guys using? I'm assuming you're probably using Shield, Private Spaces, that sort of thing. Could you let me know what you're using? Is that safe to talk about?
Garrett: Yeah, so we were initially looking at Shield rolling that out, especially because you can get some added safety blankets with Shield. We ended up not needing it. The way we rolled this product out was we tried to treat it like a startup and what is the minimum path you need to get to production? So we spun up just on the standard Heroku plan and then we had a third-party security auditor come in and conduct a series of penetration tests against our instances to make sure we weren't doing anything silly. Once we got those out of spec, we presented them to our customers we're working with. And they passed it by their security and their legal team and they said, "This passes. This is exactly what we need to get up and running." That's what we went with. I think if we were doing this long-term, we would definitely evaluate something like Shield, but the nature of this product is spin it up quickly, be very effective. And then, when we're done and this COVID situation is resolving itself, shut it down.
Greg: Tell me a little bit about your open source contributions. I see here that you guys built out the Terraform to connect the SBA API directly then you open sourced it?
Garrett: Yeah. So one of the big things with this project was we didn't want to be a gateway where pay us money to get loans approved. That's not what we wanted to do at all. And so we said, "Look, the software is free and you can pay for hosting and support." And I thought that was such a brilliant way to build this, right? Because there are so many partner vendors who've reached out to us saying, "Look, we need something. We might not even deploy this code, but we need an example of how to work with the SBA's website." We said, "Great, look at our code. Please use it."
Garrett: And we've had individuals who said, "Look, Heroku, even though with something like Shield, it's just not going to pass our legal team where we still don't believe in cloud. That scares us, but we do have infrastructure. Can we please run it on our own?" We said, "Great, here's the source code." But we update it all the time. And that became a really common thread. So anyone who wanted access to our software has total free access to it. It's a private repo due to the nature of SBA documentation, but it's source that's available for anybody who wants a copy.
Brett: And I think that's important because you look at the problem and honestly before us, there were some companies, I probably can't name them, but they were going out and really taking advantage of the opportunity and saying, "Well, hey look, we're going to build something and we're going to charge you enterprise type pricing for this." In my opinion were gouging but if you look at it from a technology history, all of the wonderful technology that we get to use because people decided to open source the technology, right? And everything from the servers that it's running on, the databases that this is running, the web application stack. We used Rails and the Ruby programming language.
Brett: There's this whole beautiful history of open source software and so when we look at what we can do to help out in some way, we thought at a minimum, we should open source our solution and just try to help out with all these small businesses that need assistance. And really the banks are the primary people that need assistance with this. And just even having the technology to spin this up really quick, have them up and running in 30 minutes to an hour. If you were to look at the whole technology stack, we did a fraction of what it took. And so it's a huge thanks to the whole open source community for deciding to open source technology. And we're a huge believer in being able to do that. So it's been pretty special.
Greg: Same boat as you. Absolutely love the open source community. And I'm so thankful for all the work of all the people that do. And I love to hear people that give back to it because I mean, in my mind, that's really what open source is. It's not just using Linux and all of that. It's giving back to that community, it's giving back to the world itself and making the world a better place than you left it.
Garrett: One thing that goes along with open sources is we're often just giving our own free time to make software better and to enable companies stand on our shoulders and propel themselves forward. And that's a philosophy we adopt for this project because the first institutions that stepped forward and asked for help, we said, "Look, let's just do this together. We're not going to focus on... We'll do it for the sake of partnership and for the sake of... We are nerds and for the first time ever nerds get to be superheroes for a brief moment. So let's put on our capes and just knock this thing out." So we were working Friday night, Saturday night, Sunday night, just through the weekend long hours, trying to bust this thing out. And to the point where we're putting so many hours in that I don't care what the sales commission could ever be on a product full enterprise grade, it's just not worth it.
Garrett: What we're doing it for is those people who would need a paycheck to pay their bills and that's constantly what we're reminding ourselves of. Every meeting we said, "Look, why are we doing this? We're spending these long hours to help those who don't have right now, because this is the skill set that we as a nation need in order to lift one another up." That's the whole open source philosophy applied to this project. Why this project was free for anyone who wanted to use it? There's no product manager who can say, "Please." Worked that much time on this. We just wanted to do it for those who need it. And we wanted to make it available for those who need it.
Greg: If you had to estimate, how long do you think it took from idea to production?
Brett: Yeah. So I would say we started on a Friday afternoon, worked all weekend. The first cut of it, we put up on a Monday morning and as anyone who's written software knows you put up the first cut and then the work begins. We put that up and we started submitting these loans into the SBA system. And the first approach we actually had to take was using some robotic process automation. So we're logging into the website, we're imitating a user, we're filling out these forms, et cetera.
Greg: Mm-hmm (affirmative).
Brett: That first weekend was really critical. And then from there, it was just iterating on making sure that the software is submitting right. And then later we started working with the SBA. They heard about what we were doing, they saw the traffic that we were sending, the volume that were doing and they reached out and said, "Hey look, we were working on an API. We need to work with some partners like you that are technology-focused that can innovate rapidly, that can make changes. As we make changes, we see how fast you're able to get something up and running with this RPA style system." And so we started working with them on the API and got API access really quick. When we heard about it... You did the API integration so maybe you can talk about that.
Garrett: Yeah. Well, before we... Let's come back to that for a second because what was crazy is the people who don't really know, the PPP came in two waves. The first traunch of money ran out in about a little less than two weeks.
Greg: Mm-hmm (affirmative).
Garrett: And so you can imagine the traunch opens that week and then Friday, we're just hanging out and then it's like, "Boom, let's start building." We start building Monday, we release maybe by Wednesday or Thursday, worked very consistently submitting applications to the SBA. You can tell other people had a similar mindset to using a Chrome Webdriver to try submitting these applications. We start seeing the website more consistently fail, right? I mean, you hit these timeouts, these Oracle-
Greg: The website being the SBA's website-
Garrett: The SBA's website fail as more people started trying to write these web drivers and then the money ran out and they closed the doors in the middle of the night. And so we thought we didn't know what was going to happen next, but we took that time to just offer a little bit of the software and start asking phone calls with the SBA, with partners ask, "Hey, how many loans were you able to clear? What's left for you?" And we heard a number of them say, "Hey, we didn't even get our forms out to people until the day they closed."
Garrett: Right? I think we've had this week of we hear there's some rumble that there might be a second wave of finances coming and it's in this week that we were able to finally get on the phone with the SBA. And that's where they said, "Hey look, we want to green light you as fast as possible. We know that you've integrated with a number of financial institutions and you can help push their loans through." So they were amazing. They had some partners working with them. We met them on a Thursday. And if you know banking or you know government, a fast turnaround typically never happens.
Garrett: And here we have a meeting on a Thursday where we're honestly expecting, "Okay, we'll put you in the queue in three weeks. We'll send you some auto reports and we'll get you this paperwork to fill out and come back to us. That very slow turn is how these deals typically build. They said, "Okay. Unfortunately, our testing website's down, but we might be able to create users for you on our side. And later tonight, it's hopefully going to be back in working order. And tomorrow let's get on the phone again to try give you a test request to show you how to submit the loans through our API."
Garrett: "And then we'll green light you. We'll give you production access." We're like, "Are you serious?" That's how fast they were. And because their test site was down and they had just opened up a second traunch, Congress had just approved it, we knew we had to act quickly. So we had a bunch of us just trying to reverse engineer the test site to figure out their SOAP API to get these loans submitted. And then we emailed them back in the middle of the night and said, "Hey, we think we got this figured out." And expecting maybe a little bit of like, "Hey, please don't do that." But also like, "Okay, we'll talk about it tomorrow." They just CC'd us directly with the SBA people said, "They're good for production. Thanks guys." They were amazing.
Garrett: But that gave us access to their test API and 12 hours later, people started low testing on their test environment. So things are going to be crazy when the floodgates open Monday morning. But as a result of the load testing, they shut off the whole API. So we had access, we have some documentation and then we had a weekend with no test environment. So that was pretty stressful, but we're able to code to it the best we could. And then Monday morning we thought, "Hey, we still have this webdriver that works really well. So we'll go live with that Monday."
Garrett: And then 10:00 AM Monday rolls around and the site is completely dead. I mean, everyone's hitting it as hard as they can. This is when the second traunch is finally live. Their site is completely unresponsive. And so we start switching back to our direct connect with the SBA and we're able to get maybe 1 in 1000 thousand loans through. It's just so slow. And then we found bugs and we're iterating on it, but it was an amazing experience to just work with the team and help iterate slowly and fix one bug, put up a pull request, merge it, push it up, and then hit the... The Heroku auto deploy runs out and get the next batch of errors popping up and stomp them. And then eventually the SBA said, "Hey, we see that a bunch of the traffic is coming from webdrivers and we're going to ban all webdrivers."
Garrett: But fortunately we had our direct connect from a couple of days prior, and we were able to stay live and continue processing loans. And that's when we really started kicking into gear and we were able to process thousands and thousands of loans for these institutions. And it was staying up until 5:00 AM, sleeping two hours and getting back up and going until you die, but that's what we needed to do to get it done and we did.
Greg: So what do you think the impact has been? Do you have any metrics you can share?
Brett: Yeah, so we're looking at the total volume. We look at the amount of money that's been able to get distributed out to the small businesses across the United States. And so we're approaching right at the $5 billion mark on what's been processed through our system and then that's just us directly. The open source portion of this has continued to extend out and other institutions have put this and they've rolled out their... They're doing their own version of our software and processing loans that way so we don't have a lot of insight into those numbers because they're private. But from what we've done, write up about $5 billion of loans processed and distributed out to small businesses.
Greg: That's amazing. That is really amazing. I'm sure that there were lots of jobs saved and businesses that were able to continue to stay afloat because of the hard work you guys put in.
Garrett: Yeah. And we can't really talk too much about the number of jobs but it was definitely in the 10,000 to 100,000 range of jobs. Again, that's the most satisfying thing to take away from my perspective. I'm just coding on this project and working on a Rails app. I was like, "Everyone at Hacker News says I could do that in a weekend." Well, here we are doing it in a weekend. And the impact is not just people, but it's people where it matters, right? It's in their paycheck, it's their lights in their home and it's their family food on the table. That is the most meaningful thing that I, as a software engineer, can give back to people. But it's funny to me still how just a little bit of Ruby code turns into a normal life for somebody. Maybe not totally normal in this strange time, but close enough.
Greg: Anything you would like to add before we close up?
Brett: There's one thing I've been thinking about as we've been talking and it's an idea that I originally heard from Steve Jobs and I'm going to paraphrase what he said. But it was this idea that your whole worldview changes as soon as you realize that the world was created by people that are no smarter than you are. They just went out and they decided to do something and they made it happen. And everything we see around us as a result of people who just believed in themselves and went out and made it happen. And I've been in situations where you hear stories or maybe for those that are listening to the podcast are really like, "Man, that's a great story and it's really cool how these people were able to help out or whatnot."
Brett: But really, I think as Garrett was saying early on in the podcast is that when you have the ability to create, we have the ability to as technologists altogether in this and you can write code and you can come up with ideas and you can create solutions that solve problems that exist in the world, it's really amazing. But I think a lot of it is just believing in yourself and finding when that opportunity comes, taking advantage of it.
Brett: You talk to anybody that worked on this project, I think you can hear it from Gary. It was a lot of hard work, but it's worth it and the team did an amazing job. The motivation the team had, there's just the intrinsic reward that the team had from doing all this was awesome. And so I think we can all in whatever way we can, pay it forward. And that can be a pull request on an open source project, right? It can be whatever, but just do something to give back. And that's just what was going through my head as we were thinking about this that I think everybody has the opportunity or can find the opportunity or look for an opportunity to help someone out and pay it forward.
Greg: Thank you so much for your time. Thank you for sharing this amazing story, Garrett and Brett, and thank you for what you guys have done to help out the world as a whole and our country and our community and the open source movement. We're super excited that you're able to build this on Heroku And I'm glad that we were able to have some small part in helping you get this project across the line.
Brett: Awesome. Thank you. Appreciate you having us on.
A podcast brought to you by the developer advocate team at Heroku, exploring code, technology, tools, tips, and the life of the developer.
Director of Product Management, Heroku Data, Heroku
Greg is a lifelong technologist, learner and geek.
Lead Software Engineer, MX Technologies, LLC.
Garrett has been working in the fintech space for over 6 years. He currently leads the Platform team at MX.
Chief Product Officer, MX Technologies, LLC.
Brett is the former CTO of NUVI and a passionate technologist who has spent his career at the intersection of software and business.
More episodes from Code[ish]
Laura Fletcher, Wesley Beary, and Ian Varley
In this episode, Ian, Laura, and Wesley talk about the importance of communication skills, specifically writing, for people in technical roles. Ian calls writing the single most important meta skill you can have. And the good news is that...
Jim Jagielski and Alyssa Arvin
Jim Jagielski is the newest member of Salesforce’s Open Source Program Office, but he’s no newbie to open source. In this episode, he talks with Alyssa Arvin, Senior Program Manager for Open Source about his early explorations into open...
Lisa Marshall and Greg Nokes
This episode of Codeish includes Greg Nokes, distinguished technical architect with Salesforce Heroku, and Lisa Marshall, Senior Vice President of TMP Innovation & Learning at Salesforce. Lisa manages a team within technology and product...