Automating infrastructure as code: Deploying Kubernetes with Digger and GitHub actions

Terraform is a great tool for automating infrastructure configuration. As your configuration and environment grow larger, and additional team members come in to help manage infrastructure, a better model needs to be in place to help run your Terraform code in a collaborative manner.

Many solutions exist in both the open source (Atlantis, tf-controller) and commercial (Terraform Cloud, Env0, Spacelift) space. A solution that came to my attention recently is Digger.

Throughout this tutorial, we'll explore Digger, an open-source tool designed to run Terraform processes within your CI pipeline, leveraging your existing CI infrastructure.

What is Digger?

Digger is an open-source tool that allows you to run all Terraform processes using the same CI infrastructure you already use. It reuses your CI infrastructure with jobs, logs, compute, orchestration, etc., so you can benefit from your existing software deployment methodology.

Prerequisites

Before getting started with this tutorial, you will need to have a Civo account. Then, to launch a Civo Kubernetes cluster using Digger.dev, you can follow these steps:

• In your Civo account, obtain an API token by going to Settings -> Profile and clicking the Security tab.
• Create a new GitHub Repository.
• In the new GitHub repository, go to Settings -> Secrets and Variables -> Actions. Create a CIVO_TOKEN secret with the API key above.
• In the New Repository, ensure you enable GitHub Actions, as well as grant the pipeline read and write permissions.

Configuring the Civo Provider in Terraform

In the repository, create the following files (links provided with example code):

Step 1: Specify required provider as maintained by Civo

1terraform {
2  required_providers {
3    civo = {
4      source = "civo/civo"
5    }
6  }
7}

Step 2: Configure the Civo Provider

1provider "civo" {
2  region = "LON1"
3}

Step 3: Query xsmall instance size

1data "civo_size" "xsmall" {
2  filter {
3    key    = "type"
4    values = ["kubernetes"]
5  }
6
7  sort {
8    key       = "ram"
9    direction = "asc"
10  }
11}
12
13data "civo_size" "small" {
14  filter {
15    key    = "type"
16    values = ["kubernetes"]
17  }
18
19    filter {
20        key = "name"
21        values = ["g4s.kube.small"]
22        match_by = "re"
23    }
24}

Step 4: Create a firewall

1resource "civo_firewall" "core-firewall" {
2  name                 = "core"
3  create_default_rules = false
4  ingress_rule {
5    label      = "k8s"
6    protocol   = "tcp"
7    port_range = "6443"
8    cidr       = ["0.0.0.0/0"]
9    action     = "allow"
10  }
11  ingress_rule {
12    label      = "k8s"
13    protocol   = "tcp"
14    port_range = "80"
15    cidr       = ["0.0.0.0/0"]
16    action     = "allow"
17  }
18  ingress_rule {
19    label      = "k8s"
20    protocol   = "tcp"
21    port_range = "443"
22    cidr       = ["0.0.0.0/0"]
23    action     = "allow"
24  }
25}

Step 5: Create a cluster without specific cluster type by default is K3s

1resource "civo_kubernetes_cluster" "core-cluster" {
2  name         = "core"
3  firewall_id  = civo_firewall.core-firewall.id
4  applications = "cert-manager,traefik2-nodeport"
5  pools {
6    size       = element(data.civo_size.small.sizes, 0).name
7    node_count = 3
8  }
9}

Setting up the GitHub Actions pipeline

The next step is to sketch out the GitHub Actions pipeline and core Digger configurations. The main branch configuration needed for this is:

1name: CI
2
3on:
4  pull_request:
5    branches: [ "main" ]
6    types: [ closed, opened, synchronize, reopened ]
7  issue_comment:
8    types: [created]
9    if: contains(github.event.comment.body, 'digger')
10  workflow_dispatch:
11
12jobs:
13  build:
14    runs-on: ubuntu-latest
15    # Based on https://github.com/diggerhq/digger/issues/395#issuecomment-1601257947
16    permissions: write-all
17    steps:
18      - name: digger run
19        uses: diggerhq/digger@v0.2.0
20        with:
21          # All current locking mechanisms require large-cloud resources
22          disable-locking: true
23          # AWS Setup Below
24          # setup-aws: true
25          # aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
26          # aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
27          # aws-region: us-east-1
28        env:
29          GITHUB_CONTEXT: ${{ toJson(github) }}
30          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
31          CIVO_TOKEN: ${{ secrets.CIVO_TOKEN }}

How to configure Digger

1auto_merge: false
2projects:
3    - name: "core-cluster"
4      dir: core-cluster

Testing the pipeline

Once everything has been committed to the main branch, you'll need to make a PR to actually test the pipeline. Create a new branch, initial-commit. Make a minor change to any of the Terraform in the core-cluster directory. Push up that branch, then create a new PR.

In that PR, write a new comment to trigger a terraform plan to see the actions the Terraform code will perform.

Review the output of the terraform plan output. This will let you and any peer reviewers see what infrastructure changes will occur.

Ensure all resources being created or modified are expected, then write a comment of digger apply.

On the PR, you will see the results of the Terraform apply.

Once the apply is complete, merge your PR into the main branch.

Once the pipeline has been completed successfully, you can log in to your Civo account and verify that the Kubernetes cluster has been created.

Now we have completed our first cluster, provisioned completely from Digger!

While this configuration will get you far, in Part 2, we will explore hardening our pipeline with better state and lock management, all within the Civo architecture!

Summary

Now that we have finished this tutorial, you should be able to use Digger to automate Terraform processes within a Civo Kubernetes cluster using GitHub Actions. You can read part 2, “Hardening our Infrastructure as Code Pipeline with Digger,” here.

If you are interested in learning more about the topics discussed in this tutorial, check out some of these resources:

• More tutorials on Infrastructure as Code can be found here.
• How to choose an Infrastructure as Code Solution talk from Lee Briggs.