This guide is for anyone looking to get more productive with kubectl, the Kubernetes command-line management tool. If you deal with Kubernetes clusters and their components, you will be doing it through kubectl. While this guide assumes you have kubectl already installed, I hope you find something useful here. I have tried to cover tricks for benefiting beginner kubectl usage, as well as more advanced tips.

This guide will cover tips for:

  • Setting up autocompletion for kubectl
  • Managing Kubernetes contexts and configuration
  • Namespaces
  • Troubleshooting pods
  • Using selectors
  • Sorting kubectl output results
  • Monitoring cluster resources
  • Output formats
  • Event logging
  • Executing commands in a pod
  • Copying data to and from containers
  • Useful shell plugins

How to set Kubectl Autocomplete

The first thing I will always do when using kubectl is set autocomplete, which allows you to use tab auto-completion when composing commands. This is extremely useful for example if you are unsure of the exact name of a resource in a namespace you can compose your command with -n namespace first and then tab will present the resources in that particular namespace, which saves running a kubectl get to retrieve all resources first.

Configure autocomplete

If your're using bash as your shell:

source <(kubectl completion bash) # setup autocomplete in bash into the current shell, bash-completion package should be installed first.
echo "source <(kubectl completion bash)" >> ~/.bashrc # add autocomplete permanently to your bash shell.

If you use zsh:

source <(kubectl completion zsh)  # setup autocomplete in zsh into the current shell
echo "[[ $commands[kubectl] ]] && source <(kubectl completion zsh)" >> ~/.zshrc # add autocomplete permanently to your zsh shell

kubectl configuration and contexts

If you manage multiple Kubernetes clusters it is useful to get to grips with using contexts. It is possible to switch between cluster configs using the KUBECONFIG environment variable.

KUBECONFIG=~/.kube/config

or you can use the --kubeconfig flag for kubectl to specify the path to a config file as part of the command, like kubectl get nodes --kubeconfig ~/custom_config.

With contexts you can have multiple Kubeconfigs merged into one file and just switch between contexts to administer different clusters.

A context is a combination of a cluster, authentication, and namespace. Therefore they can be used to switch between combinations of clusters, users with different role permissions (RBAC), and namespaces.

You can get currently-configured contexts as follows:

$ kubectl config get-contexts                  
CURRENT   NAME                CLUSTER     AUTHINFO         NAMESPACE
*         staging-context     staging     staging-user     default
          production-context  production  production-user  default

Current context is shown with *, but can also be retrieved with kubectl config current-context.

You can switch switch contexts using kubectl without having to specify a whole new kubeconfig file:

kubectl config use-context production-context

If you need to view the full configuration, you can use:

kubectl config view

The configuration file can be edited directly or with the use of kubectl config set-... or kubectl config delete-... followed by the appropriate property. More information can be found in the official Kubernetes documentation on defining clusters, users, and contexts.

Ways to work with Namespaces in Kubernetes

Kubernetes resources exist within namespaces, which provide segregation. Just like using contexts helps when working across multiple namespaces, there are a few useful ways to work with namespaces.

To run a command within a namespace use the -n or --namespace option.

kubectl get pods -n namespace-name

To run a command across all namespaces use -A or --all-namespaces

kubectl get pods -A

If you want to run a number of commands in a namespace without having to specify the namespace each time you can set your current namespace within the context. This saves you a lot of keystrokes and prevents you from accidentally changing something outside the namespace you're working in.

kubectl config set-context --current --namespace=namespace-name

This actually updates your current context and sets the defaut namespace for kubectl commands to namespace-name. If you then change your active context, this will not persist as it is configured per context.

How to troubleshoot Kubernetes pods

Pods can fail for a number of reasons, either related to the Kubernetes cluster being unable to create the pod, or the container(s) inside the pod failing after launch.

Pending pods

If a pod is stuck in a pending state, this means the Kubernetes scheduler is unable to schedule the pod to a node in the cluster. To investigate, you can check the events for the pod by running a describe command.

$ kubectl describe pod pod-name

The events should provide some indication of why the pod cannot be scheduled, for example if nodes have taints the pod does not tolerate.

Failed pods

If a pod is successfully scheduled, but enters an Error or CrashLoopBackoff state this is because the container has encountered an error. In this situation examining pod and container logs can be useful:

Examining container logs

To fetch the container logs of the default container in the pod (defined in the kubectl.kubernetes.io/default-container annotation).

kubectl logs pod-name -n namespace

If there are multiple containers in the pod you can use the -c option to specify which container to fetch logs from.

kubectl logs pod-name -n namespace -c container1

(The above will fetch logs from a container named container1 inside the pod named pod-name)

To view logs from all containers use the --all-containers option.

If you wish stream the live container logs to your terminal you can append the -f (follow) option, which is useful when watching for a log related to an action you are performing.

kubectl logs pod-name -n namespace -c container1 -f

Another useful option is --since which will print logs newer than a duration of time, e.g. previous 5 minutes of logs:

kubectl logs pod-name -n namespace -c container1 --since=5m

The --previous option for kubectl logs is useful if a container has restarted (for example in a CrashLoopBackoff) as it will load the logs from the previous container. You would use this as kubectl logs pod-name -n namespace --previous for the default container, and so on.

Watching for changes

As mentioned above, you can follow the streaming of logs of active containers. You can also watch for changes. Commonly you might want to watch for changes in a particular resource, for example when waiting for a deployment with a number of pods to become ready.

One method for this is the -w (--watch=true) option for kubectl get. This will output the results of the get command and then watch for changes in the resources and output them.

kubectl get pods -w

With the above you might be watching for pods in the default namespace to become ready, for example. Another option for this is kubectl wait. This is used to wait for a resource to match a specific state, such as Ready:

kubectl wait --for=condition=Ready pod/civo-pod --timeout=30s

This will wait for the pod civo-pod to be in a Ready state, and timeout after 30 seconds if not met. If you wish to wait for a condition to not be met add =false eg --for=condition=Ready=false.

Wait can also be useful if waiting for resources to delete:

kubectl wait --for=delete pod/civo-pod

Using labels and selectors to target Kubernetes resources

Labels are a very useful tool in Kubernetes. They can be used for various purposes, mainly related to identifying and targeting resources based on them having a label(s) assigned. This is useful as names of resources can be, and usually are dynamic (e.g. replica pods as part of a Deployment have random characters appended to the end).

For example, you can use selectors to find all pods with the label app: civo-app:

kubectl get pods --selector=app=civo-app

You can also use boolean logic in selectors. This will find all nodes which do not have the label node-role.kubernetes.io/master, ie all worker nodes

kubectl get node --selector='!node-role.kubernetes.io/master'

Note the ! before the selector negates the search.

You can also select resources by values of certain fields. Unfortunately not all fields are supported, and vary depending on the resource type. More information on this can be found in the official Kubernetes documentation on fields and selectors.

Some examples of selecting by field value:

Find all pods with the status.phase field matching Running

kubectl get pods --field-selector status.phase=Running

You can also chain selectors as a comma-separated list:

kubectl get pods --field-selector=status.phase!=Running,spec.restartPolicy=Always

This will find all pods which are not Running and have a restartPolicy set to Always.

Sorting Kubernetes query results

Kubernetes queries can return a lot of information, and it can be useful to sort it into an order.

Get pods and sort by name:

kubectl get services --sort-by=.metadata.name

Get persistent volumes and sort by capacity

kubectl get pv --sort-by=.spec.capacity.storage

Monitoring resource usage

Any sysadmin will know how useful top is when troubleshooting resource usage on a system. Kubectl has a useful equivalent command to output resource usage of pods and nodes. Metrics API for the metrics and presents them to you. The kubectl top command does not actually calculate the resource usage, but queries the Metrics API for the metrics and presents them. Therefore, you will first need to ensure the metrics server is installed, as it is possible to build a cluster without it. On Civo, metrics-server is selected by default for all new clusters.

Get CPU(cores) and memory(bytes) usage of pods:

kubectl top pods

Get resource usage of nodes:

kubectl top nodes

You can also dig down and get the resource usage of individual containers in a pod using --containers.

kubectl top pod civo-pod --containers

If your commands return a lot of results, it can be useful to combine with the --sort-by option mentioned in the previous section, to sort by memory or cpu usage.

kubectl top nodes --sort-by memory

Working with kubectl output formats

When querying Kubernetes resources you can specify the output format of the results, which is useful for inspecting the full configuration and also manipulating it to form some extremely powerful commands.

To specify the output format of a kubectl get command use the -o, --output= option.

YAML

The -o yaml option will output the full YAML spec for the matching resource(s). This is useful if you need to inspect the values of the fields and also if you wish to save the specification to a file to be used elsewhere. To do so just redirect > the output to a file in your terminal, like this:

kubectl get pod civo-pod -o yaml > civo-pod.yaml

JSON

One of the most useful options when building powerful commands with kubectl is outputting the results in JSON format. The extent of the potential uses are beyond the scope of this guide, but a couple of examples are shown. The results can be output in a JSON format (-o json) and also to print fields specified in a jsonpath expression (-o jsonpath=<template>).

We would advise reviewing the official Kubernetes documentation on output formats, and also JSON and jsonpath manipulation.

To manipulate the json output we will use the command line JSON processor jq, which is available for your operating system.

The command below prints a list off all pods which have a container restart count greater than 10. It consists of the following parts:

  • kubectl get pods -A - gets pods in all namespaces
  • jq -r '.items[] - outputs all matching items (pods) in raw json with the jq tool
  • select(.status.containerStatuses[].restartCount >= 10) - selects all items with a value for .status.containerStatuses[].restartCount (restartCount field in the containerStatuses list) greater than or equal to 10
  • .metadata.namespace + "/" + .metadata.name + " = " + (.status.containerStatuses[].restartCount | tostring) - print the results in the form namespace/pod name = restart count
kubectl get pods -A -o json | jq -r '.items[] | select(.status.containerStatuses[].restartCount >= 10) |.metadata.namespace + "/" + .metadata.name + " = 
" + (.status.containerStatuses[].restartCount | tostring)'

You could also output the node name to identify if there was any correlation to restarts and a particular node.

Another useful example is using the output of one command to build other commands using jq. The following example will search for virtualmachineinstances.kubevirt.io resources in all namespaces which have a status of Paused and output commands to restart them using the virtctl command. (If you are interested in what these resources and commands are for, they are to do with KubeVirt).

virtualmachineinstances are virtual machine custom resources which relate to instances of a kubevirt virtual machine running on a Kubernetes node and virtctl is a command line tool for managing those virtual machines.

kubectl get virtualmachineinstances.kubevirt.io -A -o json | jq '.items[] | select(.status.conditions[] | select(.type=="Paused" and .status =="True")) 
| {"namespace": .metadata.namespace, "name": .metadata.name}' -rc | jq '"virtctl restart " + .name + " -n " + .namespace' -r

This would result in an output like

virtctl restart virtual-machine-instance -n machine-namespace

where the cirtualmachineinstance is called virtual-machine-instance which resides in the namespace machine-namespace. You can then copy and paste this command into a terminal to run it. You could also add | bash to the end of the command to automatically pipe the results into bash and execute them, but this should be used with caution.

Custom columns

If you want to customise the output of a kubectl get command to present information which is not usually presented you can use the -o=custom-columns option. This allows you to create custom column names with values taken from the contents of the returned results.

One example of this is to output all images running in the current namespace grouped by pod.

kubectl get pods --output=custom-columns="NAME:.metadata.name,IMAGE:.spec.containers[*].image"

An example output might look like

NAME                                              IMAGE
kube-apiserver-01                                 k8s.gcr.io/kube-apiserver:v1.21.9

Inspecting Kubernetes Events for Troubleshooting

When trying to troubleshoot what has happened on a Kubernetes cluster it can be useful to inspect the events. Events are occurrences of a specific action carried out by the kubernetes cluster and can be useful for debugging. Events fall into categories including failed, evicted, storage-specific, scheduling, and node-specific events.

You can view events of a specific resource using the kubectl describe command, events are shown at the bottom of the output.

kubectl describe pod civo-pod

You can also get all events in a namespace and use the sort option to sort by when they occurred.

kubectl get events --sort-by=.metadata.creationTimestamp

The --field-selector option can also be useful to filter events matching certain criteria, or even use -o json with jq to inspect the full JSON output of events. As you can see, chaining the filters and output manipulation can result in powerfully specific output when needed.

Executing Shells on Kubernetes Pods

The kubectl exec command is used to execute one-time commands inside a container or enter an interactive shell in a container.

For example, this command will execute ls / inside the pod civo-pod:

kubectl exec civo-pod -- ls /

If you use the -i (stdin) and -t (tty) options it will switch to terminal mode. This will send stdin (your keyboard) to the specified command and stdout/stderr from the command to the client. This is usually used with bash to enter an interactive bash terminal on a container. For example:

kubectl exec civo-pod -c bash-container -i -t -- bash

The -c option specifies the container named bash-container inside pod civo-pod

Copying data to/from containers

The kubectl cp command can be used to copy directories and files to and from containers.

Copy /tmp/file.txt to /root/ in a pod called civo-pod in namespace civo-namespace:

kubectl cp /tmp/file.txt civo-namespace/civo-pod:/root/

Copy /root/file.txt from a pod called civo-pod in namespace civo-namespace to the local /tmp directory:

kubectl cp civo-namespace/civo-pod:/root/file.txt /tmp/

ZSH kubectl plugin

There are some extremely powerful kubectl commands, but some can become quite lengthy and time consuming to type. The autocomplete feature can help speed up typing commands, but once you have an understanding of kubectl commands it can be useful to use aliases for commonly-used commands. If you use ZSH there is a handy Oh My ZSH plugin for Kubectl, which included aliases for common kubectl commands. The list of available aliases can be seen at the official repository.

Wrapping up and further reading

If you use Kubernetes at all, familiarity with the variety of things you can achieve with Kubectl is a must. Aside from the commands themselves, and the plugins like the ZSH one mentioned above, a set of very useful command-line tools exist to further enhance your cluster management and debugging experience. You can read about some of them in this post on cluster administration from the command line.