Introduction

Operating Kubernetes can be a challenge with the number of issues that can occur when deploying workloads. Whilst at Civo Kubernetes we aim to ease provisioning and maintaining your Kubernetes clusters, your workloads are something that need to be controlled and monitored yourself.

The Komodor platform makes it easy for you to troubleshoot your workloads running on a Kubernetes cluster, whether on Civo or elsewhere. It helps you monitor your entire Kubernetes stack and helps you uncover root causes of issues.

In this tutorial, I will show you how you can connect a Civo Kubernetes cluster to the Komodor platform, get real-time visibility, and troubleshoot your issues faster.

Prequisites

To follow along, make sure you have the Civo CLI installed and set up so you can create a cluster. You will also need kubectl for your operating system so you can manage your cluster. In order to connect your cluster to Komodor, you will also need to install helm.

For Komodor, You can go to the Komodor sign up page and get started for free without needing a credit card.

Komodor signup screenshot

Cluster creation

The following Civo CLI command will create a 5 node cluster named komodor-demo:

$ civo k3s create komodor-demo --nodes 5 --size g4s.kube.large --wait
The cluster komodor-demo (6d1c4808-aef9-4e4c-b736-d1bf7e4da4e3) has been created

We will need to get the Kubeconfig for the cluster and save it to our desired location. If you do not specify a path, it will save it to the default location of ~/.kube/config. In the example below I am specifying a custom path.

$ civo k3s config komodor-demo --save --local-path /Users/saiyam/civo/komodor.config

Access your cluster with:

KUBECONFIG=/Users/saiyampathak/civo/komodor.config kubectl get node

Let's make sure that kubectl knows to use our cluster's configuration file:

$ export KUBECONFIG=/Users/saiyam/civo/komodor.config
$ kubectl get nodes
NAME                                                STATUS   ROLES    AGE   VERSION
k3s-komodor-demo-f815-68eaae-node-pool-d109-f2nqx   Ready    <none>   17m   v1.22.2+k3s1
k3s-komodor-demo-f815-68eaae-node-pool-d109-slj91   Ready    <none>   16m   v1.22.2+k3s1
k3s-komodor-demo-f815-68eaae-node-pool-d109-7txk8   Ready    <none>   16m   v1.22.2+k3s1
k3s-komodor-demo-f815-68eaae-node-pool-d109-gcrhx   Ready    <none>   17m   v1.22.2+k3s1
k3s-komodor-demo-f815-68eaae-node-pool-d109-lsx5p   Ready    <none>   17m   v1.22.2+k3s1

Connecting Civo to Komodor

Now in order to use the power of Komodor you need to integrate Civo Kubernetes and Komodor by installing an agent onto the cluster through which Komodor will be able to capture all the metrics and present it.

Komodor "explore services" screen showing no services set up.

This is the initial screen after you log in.

Navigate to the Integrations tab and choose Kubernetes.

"add cluster" option on Komodor

Give a name for the cluster connection:

Naming a Kubernetes cluster on Komodor

You will be shown a command that you would need to run on the komodor-demo cluster. It will look a bit like the following:

A screenshot of the instructions for adding a cluster to Komodor using helm

Once you run the command, you will be able to see the agent installed onto the cluster, and the services(Kubernetes workloads) start appearing in the Komodor dashboard

$ kubectl get pods -n komodor
NAME                          READY   STATUS    RESTARTS   AGE
k8s-watcher-d6db8cd8c-4hz2b   1/1     Running   0          52s

Komodor Features

There are various Komodor features that you can use to troubleshoot your workloads and get to the root cause faster. Let's look at them one by one.

Services

Komodor services are the workloads running in your cluster and you can filter them by clusters/namespace etc.

Komodor services view displaying Civo cluster data

Let's create a deployment with the wrong image to see what happens. We are intentionally supplying an incorrect image name here:

$ kubectl create deploy nginx --image=abcd
deployment.apps/nginx created

On the dashboard we will be able to see a timeline that depicts the series of events regarding the deployment we just created. This can be really useful when defining your SLAs (service level agreements) and minimizing troubleshooting hours. It tells you the time when something started failing, due to which change and for how much time. You can also zoom in onto the timeline and see various events:

Komodor services insights view

Summary view

On the timeline view, you can click on the events and see a summary view, which can help you understand what the issue was, and even shows a record of what changed, or a code diff.

"Deploy failed" summary view on Komodor showing details of why a failure to deploy an image occurred

You can also add external links, and connect Slack for notifications with just a few clicks.

In this case, you should be able to see from the events that the error is ErrImagePull as we gave the wrong image. In comparison to more typical Kubernetes troubleshooting, you do not have to interact with kubectl at all in order to find out what the issue is.

Let's fix this image and see what happens to the timeline and summary. In your terminal:

kubectl set image deployment/nginx abcd=nginx 
deployment.apps/nginx image updated

Now, over on the Komodor side, when we zoom in on the timeline you can see it's green.

Deployment timeline on Komodor showing a red failure duration, followed by a green success duration

When we click on the green event to view the summary, it will show what changed in it.

Deployment summary on Komodor showing deployment completed and timestamps

This is the power of timeline and summary view.

Quick view

In addition to these, in the services tab itself, you can describe the deployment, see pod logs and events directly from Komodor.

Live Kubernetes data options on Komodor

Output of the various Kubernetes data options from Komodor laid out on a page

Monitors

Monitors in Komodor are fairly new at the time of writing this article. This is a series of playbooks that run against your clusters for various monitors and then can alert you based on different conditions. In order to test monitors, first let's create a naughty deployment that will force a node to go into a not-ready state.

Save the following YAML file on your machine.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    project: destroy
  name: destroy
  namespace: memory-pressure
spec:
  replicas: 55
  selector:
    matchLabels:
      app.kubernetes.io/name: destroy
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: destroy
    spec:
      containers:
        - name: node-destroyer
          image: polinux/stress
          command: ["stress"]
          args: ["--vm", "1", "--vm-bytes", "500M", "--vm-hang", "120000", "--verbose"]
      nodeSelector:
        kubernetes.io/hostname: k3s-komodor-demo-f815-68eaae-node-pool-d109-gcrhx

After applying this yaml file onto the cluster using kubectl apply -f filename, will keep on creating pressure and the node will go into a not ready state many times. Leave it for some time so that you get the real benefit from the timeline view.

After some time, this is what the timeline view should look like:

Timeline view showing warning events since the deployment of the "naughty deployment" that causes pressure on the node

When you click on some of the events you can also see "Open Monitor view". These are a set of automated playbooks that runs a series of checks against various resources like Deployments and Nodes and can give you more rich information about what happened. It also tells you why those checks were done in the first place to educate you.

Availability monitor (deployment availability):

Availability monitor describing checks done and the reasons for each

Node monitor:

Node monitor describing checks done and the reasons for each

You can see the number of checks for each monitor with the description of what was these monitors check and how that was achieved.

Events

You can also see a complete history of events as a timeline and the issues occurred. You can also filter them based on various parameters such as namespaces, clusters, etc.

Komodor event time graph

Other features

You can add integrations from the integrations tab and integrate commonly-used tools like PagerDuty and Slack to get notifications and the like.

Integration options on Komodor

You can get a view of your cluster nodes, workloads, services and all the Kubernetes objects to get a minimal Kubernetes dashboard-type view from within Komodor. This saves you the trouble of going to the command line or any other UI.

Komodor dashboard node view

Komodor dashboard pod view showing the kube-system namespace

You can set the timezone according to your location, so that you can see when the events occurred with regard to your timezone.

You can add team members and share links to Komodor events with them. These links will open to display the event the same way as it looks to the other team member, wherever they are in the world. This makes troubleshooting easier when you are working remotely and debugging together.

Team addition on Komodor

Wrapping up

Komodor gives you helpful debugging functionality as soon as you configure it to watch your cluster. You can view event timelines and diagnose configuration changes with the stored diff of the change, allowing you to arrive to a much more rapid conclusion of what the cause of the issue was. New features such as the monitor playbooks, and integration with alerting systems such as Slack or PagerDuty, mean your cluster management may well be made smoother by using it.

For more information on how you can use Komodor, check out my YouTube channel where I take you on a tour of Komodor and tell you how it can help with your Kubernetes troubleshooting.

If you give Komodor on Civo a try, let us know on Twitter!