Virtual Kubernetes Clusters - Tips and Tricks

This is a session about virtual Kubernetes clusters tips and tricks. When I came up with the title for this talk, I really wanted to get accepted, so I thought I'd make it kind of clickbaity. So, I put the "tips and tricks" thing in, and here we are. It's the end of the day. I'm sure that people are starting to get tired. So, I want to make it even more clickbaity than this and see if we can kind of pump you all up about virtual Kubernetes clusters. How about "10 Virtual Cluster Tips and Tricks That Will Surprise You?" Okay, I think we're getting there. I think it's a little more clickbaity, but I feel like we can top this still. "10vcluster Tricks That Will Make You a Cloud Native Rock Star." Okay, yeah, we're gonna go with that.

Hello, my name is Rich. I'm the creator and host of a podcast you might have heard of called Kube Cuddle," where I interview people from the Kubernetes community. Before I got into developer relations, I did Ops for like 20 some years, so I have seen some things.

And I'm Carl. I'm a Dev here at Loft. I'm unreasonably upset that I didn't bring flip flops because this weather is like the hottest summer day in Seattle. And I'm here to help out with some demo points.

Awesome. So, things we're going to cover today: We're going to talk about Kubernetes multi-tenancy challenges, what are virtual clusters, and some vcluster tips and tricks. So, the why and what of virtual clusters.

Kubernetes multi-tenancy is really hard. There have been mainly two Kubernetes multi-tenancy models that people use: namespace-based isolation and cluster-based isolation.

Let's talk about namespace first. That is where tenants are restricted to one or more namespaces using things like RBAC and Network Policies. Some of the pros of this approach: it cuts down on cluster sprawl, which is great. It's really hard to manage a whole bunch of clusters. There's less wasted resources probably because we're really packing the workloads in our clusters. Some of the cons: users can't manage global objects like CRDs.

Now, that's not something that you necessarily want them to be able to do in production, right? But if you're doing dev environments or people are just wanting to spin up clusters for POC kind of stuff, you want people to have a lot more control probably. And users may need multiple namespaces, like maybe they want to run services that are supposed to run in different namespaces and talk to each other. Then you end up getting exceptions to your network policies, and these environments can get really complex when you've got a whole bunch of people running on a big shared cluster doing all kinds of different things.

Cluster-based isolation is where everybody gets a cluster. Some of the pros of this: there's better isolation because we've just got one team or person operating a cluster, and there's less complexity in each cluster. But some of the cons: it's difficult to manage all these clusters. Some organizations have tens of thousands of Kubernetes clusters. There's wasted resources because, unlike the other method, we're not packing a bunch of workloads in, and the cost can be a problem too.

One thing I want to shout out really quick: I really love this talk by Holly Cummins that she gave at KubeCon a couple of years ago. She used this phrase that I really love. She called them "zombie clusters," which are clusters that are running, and they've got workloads in them, and nobody's using the workloads, and maybe nobody even knows that the cluster exists anymore. This is a problem because these things are using up resources, they're using up power, and that indirectly has an impact on the environment. If this is something that interests you, I'd recommend looking up that talk.

So, which of these buttons are you going to push? There's another option now. There's what's called virtual Kubernetes clusters. A virtual cluster runs inside of a host cluster, but it appears to the user as if it's a standalone dedicated cluster. And we're going to be talking today a lot about this thing called vcluster. vcluster is an open-source project that was launched in April of 2021. So, almost the two-year birthday, that's exciting. It's currently the most popular implementation of virtual clusters, and I think by quite a bit. It is a certified Kubernetes distribution, so it passes the CNCF conformance tests, and it is really fast and easy to use.

So, how does vcluster work? The virtual cluster runs in a namespace on a host cluster, and there's a vcluster pod that runs in there. It contains a Kubernetes API server and some other tools, and it saves its state in a database inside of that namespace. That's SQLite by default, but you can point it at etcd or even PostgreSQL. And the vcluster CLI that's used to interact with it is a really small Go binary. Let's take a quick look at this architecture.

So, here's my host cluster. It's got the Kubernetes API server in it. It's got some other things. As an admin, I control the context of that host cluster and I connect to its API server to do that. Now, I can make a namespace in there and have a vcluster pod inside of it, and that has some things in it too. So, it's got the API server in it. It also has that data store that we talked about. It has a controller manager and it has this thing called the vcluster Sinker. That's actually kind of the magic of vcluster and we'll talk about that more. As a tenant using that virtual cluster, I'm going to connect to its API server and I manage that context of the virtual cluster. I can create namespaces inside the virtual cluster and I can have deployments, I can have pods, all kinds of stuff inside of there. Now, one thing that you'll notice that's missing on that orange line for the virtual cluster is that there's not a scheduler. That's where the Sinker comes into play. So, when I fire up a pod in the virtual cluster, the Sinker tells the underlying cluster scheduler to schedule it and that's where it runs. There's this bidirectional communication because it's got to keep the state updated. So, if the state changes on either side, that gets synced up by the Sinker. That's the basics of how it works. I'm going to turn it over to Carl now who's going to show you a little bit about this. Is that big enough text we think for everybody?

Alright, cool.

So, for this demo here, we're just going to start with a really simple K3s cluster running Docker with K3D. To start out, we can see it's a five-node cluster, nothing too exciting. We can also check the pods and we see, hey, nothing interesting going on, just our C DNS running. Now, we can use the vcluster CLI or the vcluster CTL, whatever you want to call it, that Rich mentioned. This is that tiny go binary that is how we deploy vcluster or one of the ways we deploy vcluster. We're going to go ahead and tell it to create a vcluster, give it a name "my vcluster", nice and easy. We're giving it some values file here where we pass some other options. We'll talk a little bit more about those later. We're saying 'connect false' because the default behavior when you spin up a vcluster is it wants to try to connect to that and for demo purposes, I don't want to do that. So, we can go ahead and do this. We'll have some output here in a second. We can see vcluster's going to spin things up.

Now, while we're waiting, we can use this vcluster CLI again to list the vclusters that are running in this cluster that I'm connected to. For a bit, we're going to see, hey, pod initializing. Maybe you can't see all of this at the bottom here, so you just have to believe me. But yeah, so we could see it's initializing and probably on this one, we'll see that it's actually spun up and it's in a ready state. This was real-time, a live demo that's happening right now. The only kind of demo magic here is that I preloaded the images into the nodes so we're not waiting for downloads, but other than that, this was real-time. We created a vcluster, very cool. One of the things that I had on that values file was saying, hey, in addition to the vcluster, spin up an Ingress so I can connect to that vcluster. We're just going to go ahead and check the Ingress. Sure enough, it's up, it's got IPs. Now, we can use the vcluster connect command, tell it which vcluster we want to connect to "my vcluster", pretty straightforward again, 'update current false' because I don't want to update my kube context directly here, and I give it some URL.

This is just an Etsy host on my laptop, so I can connect to it nice and easy. Cool. So you can see we've popped out a kubeconfig here. We can just go ahead and cat that. If you've seen a kubeconfig, you've seen a kubeconfig. There's nothing really exciting here. The only really interesting stuff, and we kind of scrolled past it here, is that it went ahead and updated the server address to what I told it to, so that's useful. Now, using the cube config environment variable, we can set that, and we can go ahead and get nodes on this vcluster. We can also get pods, maybe, hopefully. There we go. And we can see, hey, sure enough, we're in this new thing. It's only got one node. We'll talk a little bit more about that in a little bit as well, and we've got core DNS. So for all intents and purposes, it's a brand new Kubernetes cluster spun up in like 30ish seconds live here. And we're connecting to it, and it looks and feels like a real Kubernetes cluster because it is.

Now, we can do real Kubernetes things. So we're just going to do some really basic stuff here. We'll create an NGINX namespace, just picking on NGINX because it's easy. And then we're going to go ahead and create a deployment, really again super basic stuff, just for purposes of demo. All of this is looking and feeling like regular Kubernetes things because it is. And sure enough, we can go ahead and see if we check the pods that we do have our NGINX pods running nice and easy.

Now, if we back up a layer and go up to the host cluster or the parent cluster and we look at the pods in this namespace, we can see the vcluster. So this is the vcluster thing that we deployed with vcluster CTL. It's simple, just a simple stateful set or deployment depending on values you pass. Two containers in the pod, so that's the API server, so in this case, the default K3s API server, and then the other one is the Sinker that Rich mentioned. That's kind of where the magic happens.

And then you can see some other stuff here. We can see NGINX, NGINX. Well, those look a lot like those NGINX pods that we saw inside of the virtual cluster because they are. The vcluster basically synced those resources into the host cluster and did a bit of namespace translation so that we can maintain uniqueness. Because we can run multiple vclusters in the same namespace. In all of those vclusters, we could have a namespace called NGINX, and in all of those namespaces called NGINX, we can have a pod called NGINX. So we need to be able to keep uniqueness even when something like this happens. So you can see we've got the name, the namespace in the virtual cluster that it's running in, NGINX, and then the name of the virtual cluster that it runs in. You can also see core DNS. So this is a super fast intro to vcluster, just the basics, and I'll turn it back to Rich.

Yeah, and we're going to get into more demo stuff as we go. Can we do questions at the end? We've got a lot of material I need to cover. I just wanted to mention that if there's something interesting you see in these demos and you want to look at it yourself later, we'll have a link to the GitHub repo that we'll show you all if you want to have a look at that.

So, let's get now into the tips and tricks. There are 10 of them. Number one is Distros. The Distros is the Kubernetes that's installed inside the virtual cluster, and as Carl mentioned, that's a K3s by default. That's what we started off with when we built the proof of concept for this. It was like, can we take K3s and make one of these virtual clusters.

But as time went on, we had people approach us wanting other things. Someone came to us and said, "Hey, can you all add k0s to the list?" We thought that was a really cool idea. We were on a stream called "Containers from the Couch" that the AWS folks do, which is really great. A couple of hours after, we had a PR from Justin Garrison that added EKS to the list of the distros that you can use. So, you can actually run the EKS Kubernetes inside your virtual cluster if you want. We also have what's called vanilla Kubernetes, which is just a standard Kubernetes distribution that has ETD in it.

Tip number two is about creating these virtual clusters. There are three ways to do that. You can use the vcluster create command, like you saw in the demo that Carl did. You can use the vcluster Helm chart. If you noticed when Carl ran vcluster create, it actually ran a Helm command to spin up the virtual cluster. You can do that with the CLI, but you can also use Helm directly and install that same Helm chart yourself. We'll talk about that more. Now, as of a couple of months ago, there's a cluster API provider, which is pretty cool.

The Helm is used by the vcluster create command. It's a great option if you're doing some sort of GitOps workflow or you want some other kind of automation, as opposed to interacting with the CLI. This is the repo for vcluster. There's a directory in here for the charts, and then there's a directory for each one of those Distros we discussed. This is the values file for the Helm chart for the k3s Distros. It's handy because you can look at this file and see what things you can override. The flags in here correspond to the command line flags that the CLI has, so you can do the same things you can with the CLI. We'll have links to all these repos at the end, so you don't need to write this one down.

The cluster API provider, if you're not familiar, is used to provision Kubernetes clusters and manage their lifecycle. To use the provider, you need the cluster CTL CLI that comes with the cluster API itself. It creates the virtual clusters in the local cluster, which means the cluster you are sending the cluster CTL commands to is where the virtual clusters will get created. This is the repo for the cluster API provider. Here's a brief example straight from the docs of how you'd spin one of these up with a shell script. It has some Helm values in there if you're using the values file, or you can comment that out, and then it runs the cluster CTL generate command to actually spin up the virtual cluster.

Tip number three is about exposing the vcluster. So, we have one of these virtual clusters and we want to connect to and interact with it. There are several ways to do that. We can use port forwarding, and as Carl mentioned, the vcluster connect command uses that by default for remote clusters. You can also use a node Port, which is now the default for local clusters. This was added just a few months ago, but if vcluster detects you're running MiniKube or Docker Desktop, it'll set it up as a node Port, so you don't have to port forward to access it. You can use a load balancer and there's a Double Dash Expose flag for that, or you can use an Ingress, like Carl did in his demo.

Demo tip number four: Inception. So, you can run a vcluster inside a vcluster, and you can go even farther than that. I did this talk originally at Cloud Native Rejects in Detroit, and there was another talk about vcluster where Mike Tuon from Adobe was nesting a whole bunch of these virtually. I think he went like 18 deep or something crazy like that. There's not necessarily a great use case for this; maybe you'll think of one, but it's fun to do even if you don't have a use case for it. And back to Carl now, who's going to show you about this stuff.

All right, cool. So once again, just to level set, this is our cluster, same as what we saw before. Looks normal. But what if we looked at the pods? Kind of scrolled a little wonky here, but hopefully, you can see this. You can see this is not quite the same pods that we saw before. So, what's going on here? Well, we created a vcluster called "My vcluster" in the previous demo, and sure enough, we can see "My vcluster", we can see our NGINX pods that we saw before. But what's going on here? Why, what are we looking at?

We saw in the previous one where we had the engine pods and NGINX, and some hash, the namespace that the Pod was running in the vcluster, and then the vcluster. When we looked at the host cluster, we lied to you. I lied to you. Rich didn't lie; I did all the lying. So, sorry about this. For the previous demo, we were operating inside of vcluster 1. So again, like Rich said, there's probably not super great reasons for this, but we can absolutely nest vclusters. And when you do, this is actually kind of a cool way to illustrate we can keep translating the names. I know you guys were thinking that's going to be a really long hostname, but obviously, we have to concatenate this, and it'll get hashed. So we don't see the rest of the "myvcluster", it just gets hashed and shortened, that way we don't run into Kubernetes name length limitations.

This is one of my favorite things too. This is also looking from the perspective that we saw in the previous demo from inside of vcluster one. In this case, we can see we're actually using the kubeconfig for vcluster one, which we hid from you in the last demo, sorry again. And we can go ahead and get the pods. This looks like what we saw before, right? All the pods are ultimately getting scheduled in the host cluster because that's where they all get scheduled. But we have these API servers that can continue to nest.

One of the really nice things, and actually my personal use for vcluster, is all the development that I do, I do in a vcluster. The really nice thing is I can go ahead and delete that vcluster, and everything that's in it, all the pods that got created are cleaned up. Not immediately; they have to terminate, but now I've got a clean slate. My development flow for building stuff for Loft is: spin up a vcluster, I run Loft at home, then in Loft, I create a vcluster. I connect to that vcluster, I do whatever it is that I do for the day, which often means breaking things. So then I can clean that up, and I'm left with no crust in my home Kubernetes cluster. Because we all want a crustless home Kubernetes cluster, I'm sure. Anyway, we can see the NGINX pods were already gone the first time that I looked at the pods here, and then we can see that the vcluster pod itself was terminating. And then we can go back to our host cluster, and we're just left with that vcluster one and the core DNS for that vcluster. So, the cleanup part is, I think, often overlooked and probably my favorite part. Back to Rich. Those kubeconfigs get generated automatically when you run the vcluster create command. It gives you a kubeconfig to connect to the virtual cluster.

Tip number five is isolated mode. This is another thing that got added in 2021, and that's a theme of this talk. A lot of these things we're talking about are features that got added last year. When we started off, we had use cases like Dev environments and CI/CD in our heads. We were thinking a lot about ephemeral environments, but as time has gone on, people have started using this thing for lots of different things. There's now even people starting to use vcluster in other tools that they're making. Codefresh not long ago, like six months ago, released a new feature where they give you a free Argo environment using vcluster under the hood. So, we made this isolated mode. You enable it with a flag, double dash isolate, and it creates some objects: a pod security standard, a resource quota, and limit range, and it creates a network policy if the CNI you're using supports that. A way of getting some extra isolation.

Tip number six: applying manifests and charts. This is maybe my favorite feature. When you think about a cluster, if you're using it for development, that bare cluster isn't really usable for you yet. This feature fills that gap. You can apply Kubernetes manifests or Helm charts automatically when the vcluster is created. You specify the manifests and the charts you want to use in that values.yaml file. And this is normal Kubernetes yaml that you can use.

Tip number seven is plugins. So, this is a way to extend vcluster. And the way that these work is these change the behavior of the Sinker. We talked about the Sinker that was a thing that tells the underlying cluster to schedule the pods. You can change the behavior of the syncher to do all kinds of things. They're written in Go, and the plugins are very flexible. This is the vcluster SDK. So this is a repo for the SDK that you can use if you want to develop some of these plugins. Some of the reasons that you might do this would be to share resources inside a virtual cluster. For instance, there's something installed in the host cluster and they want it to be in the virtual clusters too without creating it in both places or vice versa. And this is an example of a couple of example repos or examples that we have. If you want to see, we've got a CERT manager plugin and there's a contour plugin as well. If you want to write a plugin and you're looking for some examples, likewise if you write a plugin and would like to share it with the community, open up an issue and we would be happy to add it to this.

All right, back to Carl.

Demo three: So we're going to create yet another vcluster, obviously talking about vclusters. But this time, before we do so, we're going to look at the values file that we're going to pass. And we'll just use your IQ to just show the init section. In this example, we're going to deploy some manifests. This is just really vanilla Kubernetes stuff. There's nothing crazy going on here. Like Rich mentioned, you could deploy Helm charts and stuff when a vcluster turns on as well. But for this demo, we've just got a simple namespace, a secret, a silly deployment that's using this little image I made that just echoes an environment variable, and a service and an Ingress to expose it. Again, this is all just Etsy hosts, so we can connect to it for the demo. Nothing too exciting there. If you docker any stuff, this is all pretty tame things.

This time, however, when we create the vcluster, we'll go ahead and use Helm. Personal preference on how you deploy vclusters, but Helm is probably the most popular, I'd imagine. In this case, we'll call it 'my vcluster 2'. We're very imaginative with naming here, obviously. And then we've got some regular Helm flags you'd expect: create namespace, give it a namespace, values file that we just looked at. Nice and easy, right? And obviously, we can see, "Hey, Helm things are happening." Pretty normal stuff. We can go ahead and take a look at the pods that are coming up just to make sure things are working. No demo magic or anything. The only thing here is we've seeded the images, so this is all real-time live vcluster magic. And we're just going to wait and kind of keep checking until all the pods are up and ready. Cool, they're up and ready. One more time, go team!

Now, we can look at the pods inside of the vcluster or rather, we can look from the host cluster perspective to make sure that the deployment I mentioned in the init manifest is actually coming online. And sure enough, we've got our core DNS. That's fantastic! And then here's my little silly demo app; it's already running. Fantastic! We can also check the service. We've got a service, the demo service. The service names also get translated just like all the other names that you would see. So this should be familiar if you're used to vcluster things. We can also check the Ingress just to make sure that hopefully it gets an IP address soon. Sometimes it just takes a minute. I can interrupt my demo and make it look bad. That's cool.

But yeah, so we're just going to wait for that, and then, really quickly, if it wants to work—this is the risk with live demos—right, there we go. Cool. Then we can go ahead and test this app. As I mentioned before, this is just a super silly little, totally ripped off from HashiCorp's little Echo HTTP thing. The only difference is I made an echo an environment variable instead of just some hardcoded thing or whatever, or some argument you pass to it. So, 'race car'—very imaginative. Palindromes are fun, I guess. So that's cool. But where are we seeing this? Where is this environment variable coming from?

As I mentioned before, this image that I made just takes an environment variable and echoes it. If we look at this, we'll see that we're mounting a secret. That secret was in the init manifest that we saw, so it's called 'My Secret'—fantastic. And then here's the image; we already know. I didn't lie to you about this one; this is what it does. And again, we can go ahead and check that secret and see what's going on. One cool thing, too, and this is kind of secondary to everything, but 'vcl cluster connect' and then we could dash and pat commands straight into the vcluster, so we don't necessarily have to set up an Ingress or do anything crazy like that. But what's interesting is that the decoded contents of the secret that I mounted actually say 'taco cat'. So, that is not 'race car' for everybody playing along. I know they're very similar and easy to get confused. But the question then becomes, what's going on? Where is this coming from? What gives?

So, we mentioned plugins, right? Here is yet another silly plugin that I wrote. It's called 'prefer parent resources'. You can probably guess where this is going based on the name, but basically, this is a plugin that says, 'Hey, I would like, if these resources exist in the parent cluster, the host cluster, as opposed to the virtual cluster, I want to prefer those ones.' So, if they exist in both places, prefer the parent. And then, just to confirm, all we did was this values file, and we said, 'Hey, run this plugin.' Now, if we— you might have noticed that there were three containers in this pod before, when all the other ones had two. So you might have been wondering, 'Hey, what's going on?' We can look at the container names, and sure enough, vcluster pulled that image and put it as a plugin.

That plugin was written using the vcluster SDK. It's a Go plugin, not super complicated. And then it's packaged as a container to run by the vcluster chart. So we can then go ahead and look at the host cluster, and we can see there is a secret also called 'My Secret'. Sensing a trend here? And we can decode it and see that, 'Hey, race car!' So that's where it came from. If I deleted this secret and restarted that pod, the secret from the virtual cluster would get mounted. But this way, it preferred that parent resource. This is a trivial example, but it gives you some idea of the flexibility of plugins and how you can control sync and all that fun stuff.

Lastly, because I wrote this thing, I know that I could GP the log, and I can see, 'Hey, it's mutating a pod. I found this one in the parent cluster, the host cluster. I want to prefer it over the one in the virtual cluster and move on with my life.' Alright, back to Rich. Thank you.

Carl, alright. Tip number eight is 'pausing and resuming'. This is another new feature that we added. As we mentioned, it's swift to create these virtual clusters and to delete them. But there might be times when you don't want everything to go away. You want to keep the cluster but get rid of the workloads. That's what you do with 'pausing'. So there's a 'vcluster pause' command and a 'vcluster resume'. The pausing works by setting the number of replicas to zero. This works for stateful sets, and vcluster itself is a stateful set. It also works for the replica sets. So it makes an annotation of the number of replicas and then sets them to zero. When you resume, it sets them back to the previous value. This spins down all the pods, and then you can bring them back up, but all of your configuration is still there.

Tip number nine: 'High Availability'. If you had asked me when we started off with vcluster if we needed to think about high availability, I would have laughed. But as time has gone on, people have started using this for real tasks. So we added this HA feature. It creates multiple copies of the vcluster components. It works with both the K3s Distro, which is the default one, and the vanilla Kubernetes one. You specify the settings in the values YAML file. Here's an example command line where you run 'vcluster create' and point it at that values file, specifying the distro.

Number 10. This is actually brand new. This is from 0.14 of vCluster, which literally just shipped last week, so this is pretty exciting: generic CRD sync. We talked about that use case that a lot of people are writing plugins for, which is, "I've got a resource, whether it's an Ingress or a secret, that I want to share from the host cluster to the virtual cluster or vice versa." This has been a lot of the reason why people have been writing plugins. But, hey, what if you want to do this and you'd rather not write a big Go plugin?

Really big news here: we have this new feature. It provides a way to enable the use of any *controller operator in the host cluster inside a virtual cluster. The reason for the asterisk is just that this is very new. You define the mappings between the host and virtual clusters just in a YAML file, which is really cool. And, at this point, we can't promise it's going to work for everything; it's a brand-new feature. But if you try it and you run into stumbling blocks, we'd love to hear about it. There was previously a vCluster plugin that you could use to do this, so if you happened to hear about that, this would be the preferred way instead of using the plugin now. And this is just an alpha that came out last week. And back to Carl for our final demo.

You're almost done with me, I swear. Alright, cool. So once again, we'll look at YAML. We all love to put YAML in the bucket; it's what we do in Kubernetes. So, Rich mentioned HA. So here's a really simple HA values file, right? Some really obvious ones: "Enable HA." Yeah, that probably seems pretty reasonable. And then in this case, we also say, "Let's also scale up the replicas for all of the things that we care about." And like any sane person would probably want to do that if you're enabling HA. There's some other stuff here that might be relevant here in a moment, but we'll get to that.

So we can go ahead and look at some pods in this namespace demo. And obviously, there's some stuff that jumps out: ETD 0,1,2 - pretty good. There's API server, controller, the actual vCluster thing, and some CNS. But what you might be noticing is there's like some other stuff, like maybe that demo that we just saw. So this is the last lie, or the last lie is now over: we have been in a vCluster the whole time. In fact, we've been in an HA cluster the whole time. So the demo environment is actually k3s, spun up with k3d, five nodes, straightforward, vanilla. And then inside of that, we deployed an HA cluster. One of the things you saw up here, maybe if you had quick eyes, was that we synced all nodes. So by default, vCluster only syncs the nodes that have workloads running on them. If you recall back to the first demo and we said, "Hey, k get pods," we saw cord DNS. "k get nodes," and we saw one node. Well, that's because there was only one workload running on only one node, so that's what vCluster showed us. And then in demo two, we said, "Hey, k get nodes," we saw all five nodes because we were lying to you and syncing all of the nodes into vCluster to our environment to make it look like a real cluster. I mean, it is a real Kubernetes cluster, right? So it's just a white lie. It doesn't count.

Anyway, we can see we've got all of the stuff running: here's our "my vCluster 2" we just created in the previous one. Again, we saw the demo, and then "my vCluster 1." And from the actual, "no more lies" context, that is the k3d environment. If we do "vCluster list," we see only the one, the HA1, because that's the only one that's actually in the context. We would have to switch context to "vCluster 1" to see the other vClusters, you know, keep getting nested and so on and so forth. Yep. And then, sure enough, if we go out and we look, we can see all of the things. It's kind of hard to see, but one of the things that you might have been wondering: I've been saying, "Hey, we're going to connect to this vCluster with an Ingress," and you've probably seen "Nginx" in a couple of places, "Nginx Ingress." Well, where did that come from? We were looking at all the pods and all of the vClusters, and there was no Ingress controller. So, what gives? The Nginx Ingress is running in the actual k3d cluster, and we've just synced it all the way down into, you know, three levels deeper, whatever we did for that first demo. And we used that to expose our vCluster. So this is one of the core benefits: we can have these cluster-wide things that are probably common, Nginx being a really good example of that. And then we could create vClusters, and then the users or administrators of those vClusters, they don't have to install it because we can just say, "Hey, actually, you can use the one in our host cluster." And I think that is pretty cool. And I swear I'm done lying to you. So if there are questions, there will be no lied answers, I promise. But yeah, that's it for me.

Alright, thank you, Carl. You are the liar, but I guess I'm an accomplice. Something. Yeah. Alright, so just wrap up here. Some resources that we've mentioned already, but here's a bunch of links. The biggest one by far is if you want to remember one thing: it's vL.com. If you go to vL.com, that's where the docs are, that's where the link to the GitHub repo is, all of that kind of stuff. This is the GitHub repo for vCluster. We mentioned the SDK; that's the repo for it. And those example plugins that we showed you, the Cluster API provider, the docs for that new generic CRD sync feature that we mentioned, and this is Carl's demo repo. If you would like to lie to other people too, spread the lie.

And I want to mention we have a community Slack too, which is a really great place to ask questions and get help. The maintainers are in there; there's a bunch of users too. There's a vCluster channel. There is also now, as of a couple of months ago, I think, a vCluster channel in the main Kubernetes Slack too. So if you're in the Kubernetes Slack, the community Slack, there is a vCluster channel in there. It is a lot newer, and there aren't as many people in there. The maintainers are in there and a few other people, so you're welcome to ask questions in there, but there are probably more people in the official vCluster Slack. So, with that, I think we're wrapped up. And again, I would love to take questions. Carl and I will be available.

I think we might just have time for a quick question or two. We're here. Okay, let's do it.

Of course. So, when you, from a virtual cluster, do a `kubectl get nodes`, you will get a subset of the parent node. For example, I use a plugin for `kubectl shell node` and specify the node name. As a tenant, am I able to access the parent cluster node that has power over the other virtual clusters? That's a good question. A very good question that I don't know if I have the exact answer for. By default, vcluster syncs only the nodes that have workloads running on. We saw that you can also enable syncing all the nodes. There are some misrepresentations about the nodes, so we can look here, but I don't think that we show everything. Part of the idea is, there are virtual nodes. By default, you could disable that and show the real nodes. I suspect the answer is it should be safe; they can't access the nodes because the node that we represent is some virtual thing we've synced into the cluster. Right. So, you're saying?

Totally. I suspect we could confirm, but they shouldn't be able to do anything malicious. We'll just continue with the misrepresentations.

Firstly, congrats on that, it looks great. Thanks. Honestly, I use `get3d` for all the tests I'm doing locally. I'm not sure I would use vcluster because I don't need this layering, but I have a couple of questions. First, about efficiency. What are we compromising in terms of efficiency?

From an efficiency perspective, the nodes get scheduled into the real cluster. There's a virtual cluster in that there's an API server running in a container in your real cluster, but the nodes get scheduled against the real scheduler in the real cluster. So the overhead is minimal. However, there might be some resource limitations. Right, on the API. Yes. One cool thing, though, is that you sort of get API Federation by doing this. If you have a shared cluster with many users, they will only hit the API server of their own virtual cluster. So there's an argument that it's even more resource-efficient.

My second question is regarding permissions. How are permissions managed from the outside? Can the nested cluster access the external cluster resources, and how are those permissions managed?

Yeah, so from a resource perspective, if you sync resources like an Ingress, for example, you can create a resource in the virtual cluster that has the Nginx Ingress annotation. This allows you to use the resource from the host cluster, if that makes sense. Pod-to-pod communication, with a default CNI that's not implementing any complex security measures, still works. If you're aware of the DNS name of a pod in the host cluster and you're in the vCluster, you can communicate with it. By default, there's the isolated mode that Rich talked about. You can control things with your CNI and adjust as needed. But the default behavior isn't isolated mode. So, if you're aware of this, you can communicate. The unique aspect of vCluster is that it emphasizes API isolation more than data plane isolation. We can facilitate that with isolated mode, and of course, if your CNI supports it.

When it comes to DaemonSets, you might expect them to run locally, but they're not local anymore. There's some "host path mapper" magic that's been introduced recently. I'm not the foremost expert on this topic, but there's been discussion about DaemonSets and how to make them work with virtual clusters. To the best of my knowledge, and don't quote me on this, the host path mapper can map hosts from nodes into the virtual cluster. This means you can access a host resource. However, this isn't enabled by default and requires opting in. So, DaemonSets might be a challenge by default. But we can check, and we can consult with the primary vCluster expert to get definitive answers.

One other point to consider is that, besides the isolated mode, if you're running a shared cluster with virtual clusters and are concerned about isolation, you'll still want to implement other security measures to protect your Kubernetes cluster, especially using admission control on the underlying cluster.

And to answer the last question about the script for the demo: no, I didn't create that. I'm not that tech-savvy. There's a link in my repo. It's a script called "Demo Magic." It's a shell script available on GitHub. It offers various functionalities. Part of what we did involves sending hidden commands to mask the fact that we were in these clusters. Additionally, I used some rudimentary bash scripting to loop commands like "vCluster list" until they're ready, combined with some regex to make that work.

Thank you all for coming. This was truly an enjoyable experience. Please do explore vCluster if you believe it can benefit you. And remember, you can always join us on Slack if you have any questions. If there are more questions, we're here to help.