Shifting Left with Harmony: Finding Balance Between Security and Developer Efficiency

Hi everyone. Thanks for coming. My name is Hannah Sutor and today I'm going to be talking about shifting left with Harmony and finding balance between security and developer efficiency, things that often conflict with each other. Now, I have to admit, I was super excited hearing I was presenting in Florida on the rooftop and I thought, 'This talk is going to be really fun.' And I wanted to make it fun and I felt like the title I submitted the proposal with was not fun. And I was doing the outline and really struggling to make it fun. So, I hope you don't mind, I axed that title and instead, I'm going to be talking about 'Everything I Needed to Know about Securing a DevOps Platform, I Learned from RVing.' It's the same content but slightly reconfigured to be a little bit more interesting and include a bit of information about me and my journey.

Okay, so about me. As I mentioned, my name is Hannah. I work in product at GitLab, specifically in authentication, authorization, dealing a lot with identity. And I'm originally from Pittsburgh, Pennsylvania. I lived there for a long time into my adulthood and recently... well not recently anymore, I moved to Colorado about five years ago now. I have three cats so I thought that little graphic was perfect. I'm a mom to a six-year-old and this other symbol I put in here to represent my love of nature. So whether it's the beach, the mountains, or the desert, I sort of find my peace and clear my head in getting outside. And I picked up a COVID habit of going for a walk in the dark every night because it's so nice for my eyes to not be taking in all of the light. So even when it gets dark not until nine o'clock in the summer, I wait until it's dark before I go for my walks.

Why are we here? I wanted to talk about something that sounds simple, and I think gets thrown around a lot. So that's 'secure your developer's tools,' right? Like, simple as that. Just do it. But I think it's a lot more difficult and a lot more nuanced than that sort of directive would let on. I, as a product manager, interviewing a lot of CIS admins who are responsible for security, and then also interacting a lot with developers who are sort of the downstream consumers of the different security protocols put into place, I get to understand where both parties are coming from and maybe understand where we can meet in the middle of it. And I think that security and developer efficiency doesn't have to conflict and I think that it's getting better and better all the time. So, I want to share a little bit about that.

I want to be clear about the focus for this presentation. So, in the middle here, I put GitLab or DevSecOps platform but this could be any developer tool you use. This is not GitLab-specific whatsoever. But there are a lot of things we can do to build secure software, right? We can run SAST and DAST scans, we can do code approvals, and things like that. But my focus is much more on the outside. So protecting the platform from being entered in the first place. So this is not about creating secure software as much as it's about securing the platform itself.

You may be wondering, 'What's RVing got to do with it?' Would love to share. Let me give you a little bit of background here. So, in 2016, my life was hit by an earthquake. Now, like this beautiful little baby, me, how is this an earthquake? In 2016, I became a parent for the first time and I also became completely miserable. I'm a very independent person. Prior to kids, I would go to work, I'd go to the gym, I'd go to a networking event, then I'd come home late at night after being gone for 15 hours, wake up, and do it all again the next day. So I love being out and about. So being trapped on the couch with an infant who cried all the time seriously broke me, broke my soul in a way I can't describe. But what it did is it forced me to reevaluate my life and decide, is this what I want out of it? I was born and raised and lived all in the same place, I felt like I was lacking experiences, and decided, 'Hey, maybe this being broken down to this level is a chance to rebuild a new life with things that are important to me.' So my partner and I decided a few of the things we wanted in the next revision of our lives was more time together, we wanted new experiences, and we wanted less stuff. So a decade or so of being grown adults, we had a house full of stuff that was starting to feel suffocating. Here, you can see this is my house in Pennsylvania with a dumpster in the driveway. So, of course, we tried to give away, donate what we could, but there's some stuff that just had to be trashed. This was all in preparation for selling our house. And then this other picture here is for donated tons and tons of stuff to Goodwill and every time I grab a tax receipt, and this is just from clearing out the house. We sold our cars except for the one that we kept to tow behind the RV, and we set off for our new life. We moved into our RV, Trudy. She is a 38-foot gas motorhome. We bought her used off of a retired couple in Ohio. And she had a kitchen, a sofa, a bedroom in the back, and a bathroom. She was rather outdated when we bought her, think solid brown. So we painted the inside of her white, we got some new furniture and ripped out the old, and we set off for our RV adventure not knowing if we would come back to Pennsylvania, when we would come back to Pennsylvania, or how long we were going to be on the road, or anything like that.

Okay, so what I want to do is take some lessons I learned on the road and apply them to platform security. So for each of the next examples, I'll go through the lesson, I'll go through how it showed up in RV life, and then I'll go through how it shows up in security.

So the number one lesson is that you need to understand the motivation why you're doing what you do. In the case of RVing, I shared we wanted to have less stuff, more new experiences, and spend time together. I was born and raised on the East Coast. We didn't travel much growing up. I'd never seen a cactus, so getting out west and seeing a cactus for me was very meaningful, and now I'm always noticing cacti, and I live somewhere where there's cacti now which is nice. And I think one of the things RVing really did for us that we only realized in retrospect was it got us out of the routine of every day and it is that routine that makes time fly. Those are what makes the years go by and you don't know what happened, but being in the RV every day, I would wake up, a lot of the times I wouldn't remember where I was, so I'd have to think like, 'Oh yeah, you're in Virginia' as soon as I'd wake up. Going to a new grocery store all the time and not being able to easily grab stuff but you have to think about everything. It's stressful, but it also kind of gets you out of that routine.

Finding the 'why' in platform security, I put up here as I'm talking to stakeholders and customers, some of the common reasons I hear. I think ones like 'implement zero trust' or any kind of framework for example, is not exactly a great example of getting behind the 'why'. So I would challenge you to question that, like what are we hoping to achieve by implementing zero trust principles, and that can help you get closer to the 'why'. I think fear of being the next headline is probably the number one. No one wants their company name in the news for being that company that had a big breach, so that's probably the biggest motivation. And no matter what your 'why', I would challenge you to articulate the trade-offs that come with increased security because there are some rights, and that's not often thought about by sometimes the higher levels at a company. We can have trade-offs in developer workflows. We can impact how quickly things can turn around and get done if we're imposing a ton of security checks. We also have trade-offs in admin overhead. We're humans, we're humans using computers, we make human mistakes and we often need humans to, for example, unlock our account right if we get locked out. So there's a trade-off over there. You need to have the admin capacity to handle the increased security, and you also need to have the admin tooling in place that allows people to do things quickly like unlock accounts.

Alright, next lesson. So what's cutting edge now will be table stakes soon. I thought I'd share a little bit about how we stayed connected in the RV. This is Trudy. We worked while we were on the road. This is the desk that my partner and I, we each worked from one side here, a little Ikea table that you could fold down. We had a device called a Mobley, which they don't sell anymore, but it was a truly unlimited Wi-Fi hotspot through AT&T, which was five years ago now, so they don't, I'm pretty sure they don't offer those anymore without paying a large sum of money. And it was only designed to plug into a cigarette lighter in your car. My partner is pretty handy, he welded it or soldered it all with the new cord so we could plug it into the wall so we had a good hotspot while we were on the road. But there was still the frustration of constantly having to look at coverage maps or understand from other people we met on the road like, 'Hey, does the Grand Canyon National Park, are you able to get connected from there?' So it was stressful and we also bought this range extender here. You can see we mounted it on the back of Trudy. This is about $800 and did absolutely nothing. So I think the lesson here is that connectivity moved really quickly. So now I'm still a member of these RVing groups and everyone is like, 'Oh, Starlink, you know all you have to do is get out of the way of a tree', you know, like my gosh, that would have been so nice. So something moving really quickly, here's an example at work. Something I saw move really quickly that deals with security is WebAuthn. It's a FIDO and W3C standard, think of it like passwordless authentication. So you can use your fingerprint to authenticate, you can use your face scan, or you can use a hardware key like a YubiKey. I went to my first Postcode conference about a year ago in Silicon Valley and I heard rumblings like, 'WebAuthn is going to be a thing', and I trusted so much and took away so much of this that I came back to work and told my team, 'Hey, I think we should implement this'. So we did in April and then in May, Microsoft, Apple, and Google announced their support for it, which is really what was needed to create that critical mass to get it to adoption. All of last year there was so many reports of even OTP no longer being secure because it can be fished, it can be brute forced. So at GitLab we decided to completely move to WebAuthn, not for our primary factor. So admittedly we still do use username and password, but everyone is forced into having a second factor, and that second factor now is WebAuthn only, meaning no more OTP. So no more typing in the code, you have to either face scan, use a hardware key, or use your fingerprint in order to authenticate. So for me, hearing this was going to be a thing to us not only putting it in our product but fully adopting it, it was about seven months.

Another lesson from the road: when things are no longer working, change it up.

Here's Trudy, and here's her towing my car which I just wanted to put in here because I think it's cool. You don't see this much anymore. You can tow certain cars four wheels down, and I just so happen to have one of them with this older Ford Escape. So, it's what we towed behind Trudy. That way, when we parked somewhere, if you wanted to run to the grocery store or something like that, you didn't have to drive your RV there. It's very helpful to have a tow car.

But in the back of Trudy, it was this space next to the bed. Here, you can see there were originally two little recliners here. It was kind of like a little seating area. We thought, 'Oh, that's perfect. We can put our daughter's crib back there.' Because it's hard to find an RV with multiple bedrooms and we didn't want something huge. Well, that little sleepy one-year-old got older while we were on the road, and suddenly she thought, 'Hey, my mom and dad are only like two feet away from me. I'm going to scream until they get me. I know they're right there.' So, we needed a separation between the child and ourselves. We tried to put a curtain up and all she did was rip it down. So, we had to let her have this bedroom where we could shut the door. And we bought a twin mattress, and every night we would drag the twin out, drop it on the living room floor of Trudy. It took up pretty much all the width that was there. One would sleep on the twin, the other one would sleep on the loveseat. We're both tall people. You can imagine that we lasted like this for about a month before we were like, 'We either need to come off the road, or we need to buy an RV that has a separate bedroom.' So, we weren't ready to come off the road. We thought we still had more to learn, and what we did was, we bought this fifth wheel trailer. Its make is the Fox Mountain, so we called it the Fox. And we had Trudy, and for a time, we had them both. This was somewhere in Arizona. We had to buy this diesel truck to tow the fifth wheel. We bought the Fox second-hand, so we bought it used, and we also sold Trudy ourselves. So, this was a lot of juggling, but we spent some time and moved out of Trudy and into the Fox.

And here is the Fox being towed by the truck and our daughter's bunk room. You could fold this flat, and pretty much it was a closet. Like, once you fold that bed down, there's no room to walk. But it had a door, and that's all we cared about.

Okay, how this applies in security. I'll give you another example from my everyday life. I always do better with actual examples rather than some abstract concept. So, at GitLab today, we support non-expiring access tokens. This is bad security practice. We don't have the greatest scopes, narrow scopes for our tokens. So, this is potentially a widely scoped token that has no expiration and can, in theory, get leaked, right? So, bad. But our customers also use these tokens for automation and integrations. So, we can't just enforce an expiration on them and expect our customers to be okay with that. So, what we did is, we published our eventual deprecation of non-expiring access tokens. We got feedback from our customers that, 'Hey, if you're going to do that, you need to provide us an easier way to rotate them. We don't have a great way for rotation right now.' So, what we're doing is, we are going to end of life non-expiring access tokens this summer, but we are going to provide a rotation API, and we're going to automatically, if our customers don't set an expiration on their own, we're going to extend it for a year. So, I'm hoping that by giving people a year, lots will make lots of noise about it and provide an easy way to rotate. I'm hoping that this is one way of doing better by our customers by helping increase security and also making things a little bit easier and taking into consideration their automation before we go ahead and enforce this change.

Lesson Four: Don't solve the edge case while ignoring the obvious.

For those who aren't familiar, an RV has tanks in them. So, if you're not hooked up to the sewer or water, you can carry your own water, and you can have your own dirty water that goes into the tanks under the RV once you use it. There's gray, which is like your sink water and your shower water. There's black, which is your human waste. And then there's a fresh water tank for the clean water. And you can constantly check the levels by rocking these buttons here. If you rock them, the lights light up and show you how full the tanks are. The one you have to worry about is the gray water. It fills very quickly, especially if you're doing dishes. So, we were always worried about our gray water. It's like another source of stress that you don't normally have. We didn't like to be hooked up to hookups; we liked being more out in nature and not at a campground, so we used our tanks a lot. But you really have to keep an eye because if you overflow them, well, it backs up and overflows into your RV.

Well, what do you know? We were always great on our tanks until one day we left, and we left the sink on. We ignored an obvious thing like accidentally leaving the sink on, and we instead were so focused on never overflowing our tanks. But of course, it happened. We came in, opened the door, and immediately saw water trickling down. Oh my gosh! So, we had to rip up the flooring, dry things out, and it was a big disaster day for our RVing adventures.

So, when it comes to platform security, I would encourage you to focus on likely risk factors. I've listed a couple of them on the left here, and then in the smaller text, I've listed some compensating controls that you can put into to mitigate some of these common risks. I put this picture here because I have one customer who loves to challenge on calls that, 'You know, I'm just so worried that one of my developers is going to be coding at Starbucks. They're going to leave their computer unlocked. They're going to go to the bathroom, and somebody's going to come and grab their computer, run out, and download all of our code in that time, and steal their computer.' So, it just kind of makes me laugh a little bit because, statistically, that is much more unlikely than some of these other risks.

Yet this particular customer is not seemingly worried about any risks other than the physical theft while unlocked risk.

Risk... Okay, final lesson: It's all a balance. Now Trudy, this is a physical balance issue we had with Trudy. If you're not familiar, RVs are often on uneven surfaces and they need to be leveled. This is a jack, you press a button and these come down from underneath the RV in four places and they're supposed to sense and level the RV front to back or side to side. They don't work a lot, so they're definitely an imperfect mechanism. And here we were in West Virginia and we didn't realize how uneven this site was and the leveling jacks could literally not level Trudy. So they tried a level but she was still, I don't know, probably two feet in the air at this point. And trying to walk around the RV while only the jacks are down is very bad for the jacks. They're not meant to take that kind of weight; you really need the weight of the tires as well. So, we could not get Trudy balanced in this scenario.

It's also just a balance in terms of the good and the bad, right? We had an Instagram and a blog and we had a lot of people reaching out to us and like saying, "You're living my dream, it looks so amazing!" And I was always very candid and honest too that like there's a lot of bad that came along with the RVing thing. We were ended up being on the road for 15 months, so a little bit over a year. I put this photo here, we were actually down in the Tampa area in Trudy five years ago. So, Fort De Soto is like 30 minutes away. And then, you also kind of have to live your life around the dump station and the tank situation. And here's my partner, literally the only thing separating you from your own waste is a thin PVC pipe, and if by chance it's not connected properly or... you might get sprayed. These things happen. So, it's a very dirty job as well, whenever you're emptying the tanks.

Here's one of my favorite pictures of us in Utah with the fox, one of our favorite states in general to go RVing in. And then, here's a not so good: we were in Kansas going 80 miles an hour on whatever major interstate runs through there when we had one tire go... Very harrowing to pull over and try to get off of the interstate pool of trucks while you have a flat tire. We did it and then we ended up really continuing our journey only to find 20 minutes later another tire blew on us. So, we had at one point two flat tires and that's what I'm showing here.

Okay, so to bring it to security, this is just my take on things. It's not, you know, it's not written down as gospel anywhere here. But basically what I'm trying to articulate is that whenever you're doing or having your engineer do a risky action like pushing code to production or changing some administrator setting that could have wide implications, right? Maybe you want to apply more friction at that point. I put low risk MFA because I truly think anyone entering your developer platform needs to at least go through that. But then once they're doing certain activities in the platform, maybe it makes sense to apply more friction. Or if they are an admin, adding friction or a very narrow scope, like having their credential be time bound is something you can consider. So, just to reiterate that balance again of like let's not introduce too much friction because people will find a way around it, or it will severely impede their workflow. But it makes sense to apply it when you're doing a risky action, or one of your users is doing a risky action.

So, just to summarize, here's all of the lessons that I shared that were from the road and applied them to platform security. And I just wanted to say thanks. And here's Trudy going over one of the Bay Area... Oh, it's not showing up, I don't know why. Okay, well it was a nice photo of Trudy going over one of the bridges in the Tampa Bay. So, thank you all.

[Applause]

I don't know if I have time for questions or not... Question or two, yeah?

So, specifically about rolling WebAuthn out... My company tried to do it in 2019 and we were trying to do it with YubiKeys. And we had about 60,000 users of our platform and we never successfully rolled it out, mainly because people didn't understand how the keys worked. So, how did you do your user education when you rolled out like, "Well, now your second factor is no longer a text message, it's this little teeny thing you have to plug into your computer." Like, how did you train people on that? So, I think we have a little bit of an advantage in being a DevOps platform company. So, I think even our, you know, marketing and sales people are a lot more tech-savvy, I think, than maybe some companies. And I think also, there we did FAQs, we did office hours so, you know, ask questions about this change that's coming. Actually, most people are not. We let people choose whether they wanted to use a key or a biometric. And I think the vast majority of people are using what's already built into their iPhone in terms of the fingerprint scan or the face scan or the fingerprint scanner on their Mac. So maybe, if you can find a middle ground there where you're not forcing a hardware key, or maybe you're only forcing that on people who have admin rights or something like that, maybe that might help. Yeah, I don't work there anymore, but that's good advice. Thank you! Also, they gave people no choice. It was like, "You are... We are doing this within a week." So, I think kind of... Yeah, it was like, and if you aren't gonna do it, you better have a good excuse. And, you know, they really made it difficult people opt out of it. It was like, "You... We are doing this!" Though.

Anyone else? Yeah.

At my former place I worked, we created several communities of practice to give people who are in charge of things, using it for example developers, the power to decide also on their own how is it with security. Do you think security should take away from this typical security people, you know, living in their own stuff, and say okay give it to this community for example, or form a new community where people can discuss like he said, finding a way which fits the maturity and cultural level of the company. What do you think about this? Yeah, I think that's definitely the way to do it. I think sometimes it can be hard for security to take the open collaboration because they feel such a strong sense of responsibility. I know best, right? So sometimes it's hard, I think, for them to open up a bit. And I'm actually one of the few product people where I interface a lot with our security team and I think it is a unique challenge in seeing that they're, in my opinion, a bit closed off to wanting feedback and collaboration. It's not the same way as we treat our product customer feedback where we're so grateful for it, right? And we really take it into consideration. So, I think it's a mentality, I think it's a mentality shift but a very good thing to incorporate. And that would be a low friction way to start, right? Like, this is we're interested, this is just suggestions, it's not trying to tell them what to do.

All right, thank you.