ClickOps over GitOps

Alright but then, let's get started. It's 10 o'clock and we have plenty of material to go through. So, the title is 'ClickOps over GitOps'. Scary, huh? Alright, so I'm gonna talk about Click-Ups. What I mean by that, I'm going to talk about platform engineering which is a large trend going on right now in the DevOps space. And, I'm gonna show you use cases with demos with the ClickOps idea. Alright, just go and sit down. It's okay.

Good morning everyone. I am Laszlo Fogas, the founder of a little bootstrap company called Gimlet.io. Prior to that, I was an independent DevOps consultant working in the Nordic space. The story there was usually 'Hey, we are in a data center next to Copenhagen, let's go to the cloud, let's do Docker, and hey, what is Kubernetes?' That's what I was doing. And, of course, that's pretty much your role as well. I think in your companies, you go out and find all those open source tools and you try to put them together to be comprehensible for developers to use. Prior to that, I was part of a Danish startup story, and I like to start things up wherever I go. I started up Meetup groups: Copenhagen Cloud Native, Copenhagen. And, right now I'm based in Hungary, so I'm the founder of Cloud Native Second.

Alright, so 'ClickOps', the scary term. If you go on Google and search about 'ClickOps', it's the scary thing that you go on the AWS console, you click around and that's it, basically. That's not good practice because obviously there are going to be inconsistencies if two or three people are just clicking around. Suddenly, you're just gonna miss something and then there's going to be an outage, which of course means configuration drift and it's not manageable. But you all know that.

So, what do I mean by 'ClickOps'? I'm reinterpreting the term and trying to come up with this practice. I try to make dashboards cool again. Basically, 'ClickOps' is doing cloud operations by clicking on a dashboard. Got that so far? That generates a stream of infrastructure as code changes based on your actions. So, suddenly if you think of 'ClickOps' like this, it's not as scary anymore. Dashboards are bad, but they have very good features. They make you be productive. Common tasks, you're going to be able to perform faster. You're going to be able to discover new features more easily. Even I, a seasoned DevOps engineer, when I need to launch a Google Cloud Kubernetes engine, I always go to the dashboard first because they introduce new features all the time. If I go to the CLI, exploring all those switches, it's just not the practical way of discovering toggles and stuff. So, dashboards have good features.

But you know, the bad things about dashboards are that if you click on the dashboard, it's just not productive and inconsistent that is going to show up in a few months' time. But with this approach, because if you click, and there's going to be a Terraform diff or a Terraform patch or a GitOps file put into your GitOps repositories, suddenly you get rid of the configuration drift. And if you use a tool that also allows people to actually access the underlying infrastructure's codebase, it could coexist with people who use the dashboard and people who want to do advanced stuff like writing those Terraform scripts or CRDs, because they know how to. If these two worlds could live together, I like that vision very much, and that's what I try to make a reality with my company as well.

So, why ClickOps? You know, it's shameful to click on the dashboard. But you know what's worse? Spending three weeks achieving the same thing that you could have done in two hours on the dashboard. That's just the fact that infrastructure as code is sometimes tedious, hard, and sometimes it's just not even possible what you want to do, and you bang your head into the wall. But then there's the other take: you could not write infrastructure as code for three weeks, but then you're gonna spend maybe three months or more. Many times, my job was to say, 'Hey, we need to do this Terraform thing now because we are large enough,' and then I have to go through all these little things that people are not even with the company anymore, and I have to put into Terraform, and it's really painful. So, there's a reason we're doing infrastructure as code; it's just that sometimes it's tedious.

Is ClickOps a new thing? In this interpretation, I would say so. It was only a year ago when I saw this tweet from Corey Quinn, a hilarious person on Twitter who makes fun of AWS and all the hyperscalers whenever they do something bad, expensive, or shenanigans. He calls those things out. This one time, he was serious. This was just a meme: 'Yeah, at first you do the console.' I won't explain memes here, but this time he was serious, and he put down this vision which resonated with me. He said, 'I can still use the console to generate a stream of changes in infrastructure as code.' Then, this line: 'Suddenly, using the console stops being a shameful thing.' We're a little bit elitist, all of us here. We're engineers, we want to code and test, and whoever uses the dashboard is just not an engineer, whatever. So, is it a new thing? Yes, I think in this interpretation. My company, Gimlet, surely talks about this. I've also seen Ambassador Labs and CloudBees using ClickOps I think in a similar sense.

There is something going on right now in the industry that's called platform engineering. There are many hundreds of teams building platforms. If you are building a platform for your developer team, then you know what I'm talking about. In some way or another, you try to achieve a very similar thing. You try to make the many open source tools digestible for developers, like easing the cognitive load. So maybe having a UI is your goal, maybe not. But you're certainly simplifying the open source space. Those internal efforts may or may not lead into a UI, but having a UI is a lot of work.

So, platform engineering, I'm just going to quickly go through that because I think you are familiar with the term. Is there anyone who has not heard it? Alright, or just shy, cool. It's the golden pass, the paved path, or the internal Dev platform over the cloud native landscape. Because there are great tools out there, they can give you great capabilities. But when you try to plug them together and you try to make just one nice user experience to your custom needs or to your team, it's just painful. It's a lot of work.

I love Kelsey. It was more than five years ago that he brought that everybody wants a PaaS. This just has to be built by them. So that's why platform engineering is going on right now. Then there is this was like almost four years ago and he was so right on the money, so early on. So I just love this guy, how clever he can be.

Yes, so why are we doing this? There are many tools and we are shifting all things left. You know this term like there is software writing, coding, then there's QA, security, compliance, and all that. And you try to shift everything to the left. Poor developers now have to know everything. And honestly, like DevOps is early intake. You wrote it, you run it. It's kind of a total order if you don't have the right tooling. I can be mindful of security if I have the right tools. And there is like Snyk and other tools which give me, you know, like every morning I get those pull requests and I'm so happy about those. But without this tooling, it's just an impossible work for engineers to be on top of everything.

Plus, DevOps mindset skill is just seriously not evenly distributed. Some are very good at it, some want to be good at it and some just couldn't care less. They want to be in UX and other things. And that's, I guess it's alright. But you just have to be aware of this reality that the DevOps ethos is just, you know, you need some tooling to get there. To have a good baseline in your team.

I love this slide. This is not my slide, this is from Daniel Bryan from KubeCon last year. This shows how technology evolved over time from 2000 to 2020. And you know, first you had just a monolithic application, then SOA, and microservices. And then you suddenly see the cognitive load of developers. How many things they have to keep in mind to deliver their software to production. This red curve was basically the complexity, or the load on developers. They have to understand so many tools. And it started out nicely. You know, I have one Mona Lisa, upload the jar file, and I'm good to go. And then it became a bit more complicated with ESBs and whatnot. And there was this very nice moment in 2010.

I'm not sure if you can guess what those tools were that made this nice moment. That's the Heroku moment, and also like New Relic and other things. You had those two nodes, you paid a shitload to New Relic but you had observability. You didn't need a Prometheus cluster and all that. And then things went out of control, like microservices and all the great capabilities because we got a lot. We got many good things which just takes time and effort to put them together. And if you don't help with tooling, developers just gonna tell you, 'Just go home, man, I don't care.'

So I think platform engineering is a natural progression of DevOps, or it's still part of DevOps. It's not like 'kill DevOps' or anything like that. But you have to build those tools to help developers to manage this complexity. Yes, it's a new thing. This is just a Google Trends report I made, so it's only like a year-old trend. I think in 2021's Puppet report, self-service platforms were a differentiator. Teams who had self-service, they had better metrics on, I don't know, mean time to recovery and all those cool DevOps metrics. Also, Gartner recognized this as a trend and we are on the hype cycle right now. Like this is platform engineering with a two to five-year timeframe until it's gonna reach the plateau of productivity.

So yeah, we have a conference and it's happening. We are all platform engineers now, not DevOps anymore, or something like that. So if you go home and you tell your boss that you're a platform engineer, you might get a raise. So that's your ROI of this conference. You're welcome. All right, and that, you know, so ClickOps leads into platform engineering and goes back and forth, and I'd like to show you some use cases with demos, like how we achieved some of those things in Gimlet.

So, first of all, you know, when a developer needs to deploy, he needs to write those Kubernetes manifests one way or another or another. Maybe you have your company helm chart, maybe you just write a role, Kubernetes manifests, and basically, and then you have to GitOps flow to deploy everything. I think GitOps is now the de facto standard to deliver YAML onto our cluster. With ClickOps, it was actually a very easy thing for us to do because the file format existed, the Kubernetes resources existed, so it's basically just a templating question like I need a deployment with this name and the other templating. Then obviously it's a harm chart, so, so we made a general-purpose time chart for, web services and Cronjobs. So basically, it exposes the 20, 30, 40 most common use cases, and you basically just edit a value style, and you get the larger Kubernetes manifest. And we also made an open-source React component on top of it because there's, it's a little less known fact that Helm charts can have schemas, and if you have a schema, you can generate a UI on top of it. So we made a React component that actually we use and open source, and everything on this blog post, you can read the rest. Basically, you take a helm chart, you throw this component on top of it, and then you can basically render the Hub chart, and you can configure it on the UI. I want to show you this, just the open-source part if you are cool with that, just one minute.

Yes.

All right, so I need to cheat. So I have this little command manifest configure and with a YAML file, like what I'm editing, it brings up a UI, and it's basically just a value style with a little bit of a schema on top of it plus some UI helpers like how to render stuff, which widgets to use. So if I change it to five replicas and if you, if I add an environment variable called Dummy 2 - and then value 2, and if I just close this, this is just a little proof, proof of concept here, then it actually wrote this Demi ammo you can see the values here, replica five, wars hello, and value two. And if you want to get the Kubernetes Manifest out of it, you can render this, and you will see that it generated the configmap, with the two values, and then deployment, service and Ingress, and all that.

It's basically like her template comment, so pound has some very nice things and, of course, people sometimes bash him for certain reasons, but if you, if you use it for the right thing, I think this, it is still a great tool. So this was just a short demo on, on this first use case, like how to put a UI on top of a Helm charts. It's simple, basically, because everything was out there, you just, I just needed to write this React component.

Use case 2, DNS, so that is also, you know, in the old world, DNS was this heavy thing, like you needed to request through Geo tickets and other teams and stuff to days to, to make, and they often messed up, so you have to start the process over. It's, it is also reduced to just manifest all during, so, like, there is the Kubernetes resource Ingress, it is great and with some more tooling, it's actually, again, this problem is just editing a YAML file, but you all know that. But if you're not, you know, you make the, the Kubernetes Ingress resource, there is a, an Ingress controller behind it, and then the actual DNS entries could be done, either you use like a wildcard DNS and you are done with the problem, you just have a shortcut there, or you use an open-source tool called External DNS, which whenever there's a new Ingress showing up in your cluster with a new DNS name, it will make the change in Cloudflare or some other DNS provider. It's very handy, it works super swift. I'm gonna have a workshop this afternoon, and if you join, and you're going to make an Ingress, it's gonna go into the Cloudflare DNS infrastructure using this little controller. Oh, yes, and there is Cert Manager, another project of this ecosystem, which is just magic, and it works, and Let's Encrypt is behind it, so all your domains will have certificates, which is like, oh my God, like, still magic even after five years of using it.

All right, secret handling could be also reduced to just manifest ordering. Like manifest editing. I just have to use the Kubernetes. It doesn't do secret management very well or actually it doesn't really solve it. It has the resource, but it leaves many questions open. There's huge businesses built on top of it, so that's the million or billion-dollar question, but for us to make it ClickOps, so, like, editing and, uh, and making it manifest, we use an approach called Sealed Secrets. Basically, you encrypt your secret, and you can put it into git, and only your cluster can decrypt it. So it is a safe way to actually, one very cool thing is that suddenly your application and your secrets are delivered with the same mechanism. If you don't use this approach, this encrypted GitOps based secret handling, then, you know, you deploy your application with your Github's workflow, and somehow, some other workflow gonna inject your secrets into the cluster, and those things are a little bit like a broken workflow and GitOps pulls this nicely together. So I like that. And with this approach, it's again a YAML, it's again a custom resource in Kubernetes. We can template it, we can basically click on a dashboard and suddenly you can have everything backed up with a Git commit. So yes, it is ClickOps.

The fourth use case, we need a database. Now things get more serious. This is not my play, this is Victor Farsik's play, but it's basically you can even have an RDS or a civil database provisioned with Kubernetes custom resources today, which is super cool because, again, this was very ticket based. You have to ask your friend, or friend, or colleague who had access to the Terraform or you had to pick up the Terraform knowledge and yes, I'm gonna provision that RDS instance. But today, a bit Crossplane and Flux also has like a Terraform controller component. Suddenly everything is, again, fits to the GitOps paradigm which is amazing if it fits the paradigm, if there's a CRD, we can put the UI on top very easily. So you're gonna click on the dashboard, hey, I want the database. It's going to create that CRD file, put it into the Git repository, and the controller is going to provision it. And again, you are clicking on a dashboard but you are actually, you know, like generating the stream of infrastructure as code changes that your colleague would do, who, who doesn't use a dashboard. So that the two worlds play about together here.

Actually, back in the Ingress, I wanted to show you something. I think, oh, I have only five minutes, so yeah, I'm gonna do it.

So, this is actually a Gimlet, so this is your GitHub, your Git repositories, these are the applications you developed and stuff. And this is my favorite one, called Demo App, and you can see why it is my favorite. Down here, you can see your Git comets and so on. Up here, there is an environment with different releases and it is running now on the Demo App Five dot gamert.io domain. So if I use the same Helm component, a React component as before, and I go into the Ingress part, and if I change the URL to Demo App 6, suddenly you can see a little diff here that's going to go into the Git repository. It can open a pull request for you. So it's not like you are commenting directly into main, you can still review like your usual code review process. And if you merge it, CI is going to build an old app and once the build is done, we can deploy this with Gimlet. Alrighty, let's go back to here. You can see that CI is running. There is just gonna be one hiccup with this demo. I have to hit Refresh on this page because one of the web hooks is not configured. That's just, that's just how it is. When CI is done, I'm gonna do the refresh and I'm going to do the deploy. So, just one more minute.

All right, image is pushed, everything is fine. I'm gonna hit the refresh I was talking about. And if I, as you can see, this is the Git commit that was deployed on The Meta environment and I'm going to deploy this new version onto this environment. If I do this, there's going to be a GitHub's commit made and then Flux is going to synchronize it to the cluster. As you can see, Flux is now trailing. So it's, you know, just taking to, to synchronize and then you're gonna see that this Demo App five is going to change to six. I'm gonna click it and by that time, Cert Manager did the work, External DNS did the work. So it's all automated and just based on clicks. Oh, it's already Demo F6, you've probably seen it's flashing and stuff. So if I click this one, it is HTTPS Demo F6, and there is a stream of Git changes. So if I look at this one, one minute ago, I released this version. It's actually made the change in the Ingress. So this is the Ingress and the host was changed. So if people would not go to the dashboard, just go to the source code, everything is there. So that's basically how the two worlds come together.

So I have three minutes left, so there is another use case with the Infrastructure, Infrastructure Component Marketplace. There are many great Community hand charts here, uh, we put it on a UI and because of Flux's Helm Release CRD, again a CRD, everything is reduced to just editing YAML files. We can put a UI on top of it and honestly, like, all marketplaces are good for you when you start up but it doesn't allow you to manage that configuration and or how to update later on. So it's like a little bit of Half Baked, like most marketplaces are.

And then here's the most important thing. It's if you are clicking the dashboard, don't break the work of people who are not using the dashboard. The two worlds must coexist and to do that, the UI should be robust, being able to handle, like outside changes. And you don't have to, like, seriously, like, don't lose at its mates that are made outside of ClickOps because then suddenly, like, expert users are gonna say that is crap software. And what we do is basically, since we are doing just text file edits we can just do the same as Git. We do a three-way merge. So basically, this was the Baseline, this was the UI person, this was the Advanced person, and suddenly you can merge things together. And if there is a conflict, there are conflicts obviously when you edit stuff, then you can resolve those and they're going to surface nicely.

Alright, so I have one minute left. It's open source. So if you go there, give us a star, that would help us with VCS and all that. And so, thank you very much. We have a Source Early Access because even though we have an installer, an open source installer, we do a SaaS version just to make it more easy, easy to, to get started because it lands so much nicely into this idea that you click, click, and off you go. We are on the Civil Marketplace. This is me and I'm going to have a workshop at four o'clock if you want to do just pure open source GitOps with me. So, thank you very much.