Introduction

The Civo cluster autoscaler is an implementation of the Kubernetes autoscaler that adjusts the number of nodes in a cluster down or up based on specific conditions, namely:

  • If pods fail to initialize or run because nodes lack the resources to run them
  • If nodes are not fully utilized, the pods performing the workload could reasonably be scheduled on other nodes.

You can set up the Civo cluster autoscaler on any running cluster by adding it as an application from the Marketplace. By default, the autoscaler is configured to scale between 1 and 10 nodes, but this tutorial will cover editing the configuration to fit your use case.

Setting up the cluster autoscaler

As a marketplace application, you can add the cluster autoscaler to any running Civo Kubernetes cluster. You will find the autoscaler in the Management group of the Marketplace when first setting up a cluster:

Civo Marketplace showing the autoscaler

You can also add the autoscaler to an already-running cluster by navigating to the Marketplace tab on the cluster's management page in the dashboard:

Autoscaler installation onto a running cluster

Adding the autoscaler requires only selecting it as an application and installing it. It will be configured to run in the kube-system namespace as a pod:

# The KUBECONFIG for a cluster with an installed autoscaler has been set as the current context
$ kubectl get pods -A
NAMESPACE     NAME                                  READY   STATUS      RESTARTS   AGE
kube-system   civo-ccm-7cb9c4b58f-7j6qr             1/1     Running     0          3m42s
kube-system   coredns-7796b77cd4-6rk9k              1/1     Running     0          3m42s
kube-system   civo-csi-node-b98bd                   2/2     Running     0          3m28s
kube-system   civo-csi-controller-0                 4/4     Running     0          3m42s
kube-system   helm-install-traefik-crd-dm927        0/1     Completed   0          3m37s
kube-system   helm-install-traefik-2286p            0/1     Completed   1          3m37s
kube-system   metrics-server-ff9dbcb6c-9d6dg        1/1     Running     0          3m42s
kube-system   traefik-sc2zv                         1/1     Running     0          3m3s
kube-system   cluster-autoscaler-855989d6d4-p7ns8   1/1     Running     0          106s

You can also install the autoscaler using the Civo command-line client's civo kubernetes applications command.

Behind the scenes

When you install the autoscaler onto a cluster, the system will inject a number of Kubernetes secrets into the deployment which allows the Cloud Controller Manager to scale the cluster up or down depending on the need.

You can view the full pod configuration of the autoscaler as it is set up by running kubectl describe pod cluster-autoscaler -n kube-system. Some of the notable elements, which will be discussed further below, are:

Command:
      ./cluster-autoscaler
      --v=4
      --stderrthreshold=info
      --cloud-provider=civo
      --nodes=1:10:workers
      --skip-nodes-with-local-storage=false
      --skip-nodes-with-system-pods=false

Limits:
      cpu:     100m
      memory:  300Mi
Requests:
      cpu:     100m
      memory:  300Mi

These lines define the running configuration of the autoscaler in its default configuration, including the minimum and maximun number of nodes to scale to. The autoscaler watches for events in the cluster for CPU and memory requests from pods. If no nodes have the requested capacity available for scheduling, this triggers a scaling event in the cluster and a node is added to the pool.

Environment:
      CIVO_API_URL:     <set to the key 'api-url' in secret 'civo-api-access'>     Optional: false
      CIVO_API_KEY:     <set to the key 'api-key' in secret 'civo-api-access'>     Optional: false
      CIVO_CLUSTER_ID:  <set to the key 'cluster-id' in secret 'civo-api-access'>  Optional: false
      CIVO_REGION:      <set to the key 'region' in secret 'civo-api-access'>      Optional: false

These secrets are automatically injected to the cluster by the application when you install it from the Civo marketplace. These allow the Cloud Controller Manager to call upon the Civo API to scale the cluster up or down. The Optional: false lines are significant because each of these secrets must match what your cluster and account details require in order for the scaling to work successfully.

Cluster autoscaler in practice

To demonstrate the autoscaler in practice, start by creating a 1-node Civo Kubernetes cluster and choose a Small node size to begin with.

Note: Most applications benefit from having Medium or larger nodes. This demonstration intentionally uses smaller nodes to show autoscaling in practice.

Make sure you select the Civo cluster autoscaler marketplace application in the process of starting the cluster. Additionally, ensure you do not remove the default metrics-server application. Cluster autoscaling relies on metrics being reported by the cluster.

Once the cluster is running, download the KUBECONFIG file from the cluster's information page and set it as your current context to be able to apply a deployment onto it.

The following snippet, deployment.yaml defines a ngnix installation with larger resource requirements than the cluster you have created will be able to comfortably provide. Once you apply the deployment to your cluster, this should trigger a scale-up of your cluster's nodes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"

Add this deployment to your cluster by running kubectl apply -f deployment.yaml once you have saved the snippet to a file.

Within a few moments you should be able to see an additional node be created to your cluster, raising the node count to 2:

Node added by autoscaler

The autoscaler will follow this basic principle for any cluster you add it to: if the resource constraints provided by the current state of the cluster mean the desired state of deployments cannot be achieved, another node will be added to the pool. Conversely, if resources in the cluster are not being used effectively, and the desired number of pods could be run on fewer nodes, the autoscaler can remove a node from the pool. The autoscaler will continually monitor the metrics provided by the cluster's metrics-server to make the decision to scale up or down based on the configuration set.

Editing the cluster autoscaler configuration

The cluster autoscaler default configuration can be edited to your use case. Editing the autoscaler configuration involves making alterations to its deployment definition. This can be done through the kubectl edit command:

kubectl edit deployment cluster-autoscaler -n kube-system

This will open up the default editor to allow you to make changes to the definition. When you are finished, saving the file will commit the changes back to the cluster, which will attempt to reconcile its state to match what you have written.

As an example, you can edit the minimum number of nodes from 1 to 2, which would keep the minimum worker node count at 2 even if the configuration of the nginx deployment from the section above would be scaled down.

If you edit the autoscaler configuration with kubectl edit deployment cluster-autoscaler -n kube-system and change the line --nodes=1:10:workers to read --nodes=2:10:workers instead, the cluster will treat that value as the new floor for the node count. As such, if you scale the cluster to, for example, 6 nodes in the pool, the autoscaler will begin monitoring resource usage on these new nodes to potentially bring them back to line with your specification.

To trigger a scale down to the new minimum node count, you can edit the nginx-deployment resource to edit the replicas to 2. This will trigger a reconciliation of the nodes by the autoscaler within a few seconds.

You can also control the maximum number of nodes that the pool can scale to. This is set by the second number on the --nodes=1:10:workers line. If you wanted a cluster to be able to scale beyond 10 nodes, you could edit that number accordingly. Note that your account quota will take precedence over any node scaling, and you would begin seeing "Quota Limit Reached" errors in the Cloud Controller Manager pod if the autoscaler would try to scale the node count beyond what your quota allows for.

Wrapping up

Civo's cluster autoscaler uses the same API and implementation as the upstream "official" Kubernetes autoscaler. By monitoring a cluster's resource statistics, the autoscaler can make sure the services you want to provide remain accessible during peaks in demand, but also scale down when demand drops. You can configure a floor and a maximum count of nodes for the autoscaler to make sure that your spending does not exceed what you expect it to, for example.

Give the autoscaler a try and see how smart your cluster can be!