Deploy Prometheus, Grafana, and Alertmanager on Civo Kubernetes with Helmfile

This tutorial shows how to deploy Prometheus, Grafana, and Alertmanager on a Civo Kubernetes cluster using Helmfile and the kube-prometheus-stack chart.

Civo manages the Kubernetes control plane separately from your worker nodes and uses K3s as its Kubernetes distribution. Because of that, some default scrape targets and alert rules in kube-prometheus-stack need adjustment for a clean installation. This guide also configures persistent storage for Prometheus and Grafana using Civo's civo-volume storage class, including retention limits that fit offline volume expansion.

By the end of this tutorial, you will have:

• Prometheus collecting metrics from worker nodes and Kubernetes workloads
• Grafana with persistent storage and preconfigured Kubernetes dashboards
• Alertmanager ready to route notifications
• A version-controlled Helmfile that can reproduce the deployment
• Civo-specific configuration that avoids unavailable control-plane targets and storage rollout failures

Architecture

kube-prometheus-stack is a single Helm chart. One Helmfile release installs Prometheus, Grafana, Alertmanager, the Prometheus Operator, node-exporter, and kube-state-metrics together — including the Operator CRDs (such as ServiceMonitor and PrometheusRule) that the Operator uses to reconcile scrape targets and alert rules. You do not install those components one by one.

Helmfile keeps the chart version, repository, and values in Git. Instead of remembering a long helm install command, you apply a declarative file and the cluster matches that file.

Prerequisites

To follow this tutorial, you need:

• A Civo Kubernetes cluster
• kubectl installed locally
• Helm 3+ installed locally (this tutorial was verified with Helm 4.1.4; Helm 3 remains fully supported)
• Helmfile installed locally
• The helm-diff plugin for helmfile diff (helm plugin install https://github.com/databus23/helm-diff)

If you do not already have a cluster, create one by following the Civo documentation for Creating a Kubernetes cluster. You can use the Dashboard or the Civo CLI. For a monitoring demonstration, start with at least two or three worker nodes.

Step 1: Confirm your cluster context

Confirm you are pointed at the intended Civo cluster:

1kubectl config current-context
2kubectl get nodes

You should see worker nodes similar to (versions will vary by cluster):

1NAME                                       STATUS   ROLES    AGE   VERSION
2k3s-monitoring-demo-...-node-pool-a1b2     Ready    <none>   2m    v1.34.3+k3s1
3k3s-monitoring-demo-...-node-pool-c3d4     Ready    <none>   2m    v1.34.3+k3s1
4k3s-monitoring-demo-...-node-pool-e5f6     Ready    <none>   2m    v1.34.3+k3s1

Notice that ROLES shows <none>. On Civo, the control plane is managed and does not appear as a node in your cluster. That fact drives several values changes later in this guide.

Also confirm the default storage class:

1kubectl get storageclass

Expected output:

1NAME                    PROVISIONER    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION
2civo-volume (default)   csi.civo.com   Delete          WaitForFirstConsumer   true

civo-volume is Civo's native storage class and supports volume expansion. Per Civo's documentation at the time of writing, expansion is an offline operation (the volume must be unmounted first). Future CSI driver releases may change this behavior, so verify against the current Civo documentation when planning storage growth.

Step 2: Create the Helmfile project

By the end of this step you will have:

1monitoring/
2|-- helmfile.yaml
3`-- values/
4    `-- prometheus-stack.yaml

Create a working directory:

1mkdir -p monitoring/values
2cd monitoring

Create helmfile.yaml:

1# helmfile.yaml
2repositories:
3  - name: prometheus-community
4    url: https://prometheus-community.github.io/helm-charts
5
6helmDefaults:
7  kubeContext: YOUR_CLUSTER_CONTEXT   # replace with your kubectl context name
8  createNamespace: true
9  wait: true
10  timeout: 600
11
12releases:
13  - name: prometheus-stack
14    namespace: monitoring
15    chart: prometheus-community/kube-prometheus-stack
16    version: 86.1.0
17    values:
18      - values/prometheus-stack.yaml

Two settings are especially important:

• kubeContext — Specifies the Kubernetes context Helm uses for this deployment. Replace YOUR_CLUSTER_CONTEXT with the output of kubectl config current-context so applies always hit the intended Civo cluster.
• version — Pinning the chart version keeps staging and production aligned. This tutorial was tested with 86.1.0; leave the version unset and you get whatever is newest on the day you run the command. Before pinning a newer version, check helm search repo ... --versions and the chart CHANGELOG — minor bumps in this chart occasionally change default value paths.

Confirm available chart versions if you need to choose a different pin:

1helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
2helm repo update
3helm search repo prometheus-community/kube-prometheus-stack --versions | head

On Windows, use an equivalent command to display only the first few results.

Install the helm-diff plugin. It is required for helmfile diff. Some Helmfile versions also use it during apply to preview changes. Prefer this over helmfile init, which can prompt for confirmation:

1helm plugin install https://github.com/databus23/helm-diff

If the plugin is already installed, helm plugin list will show diff.

Step 3: Configure the stack for Civo

Create values/prometheus-stack.yaml:

1# values/prometheus-stack.yaml
2
3# Managed control plane scrapes are not useful on Civo; kube-proxy is
4# typically disabled here too (no useful separate scrape target on K3s).
5kubeScheduler:
6  enabled: false
7kubeControllerManager:
8  enabled: false
9kubeEtcd:
10  enabled: false
11kubeProxy:
12  enabled: false
13
14defaultRules:
15  disabled:
16    KubeMemoryOvercommit: true
17
18prometheus:
19  prometheusSpec:
20    retention: 15d
21    retentionSize: 18GB
22    resources:
23      requests:
24        cpu: 250m
25        memory: 1300Mi
26    storageSpec:
27      volumeClaimTemplate:
28        spec:
29          storageClassName: civo-volume
30          accessModes: ["ReadWriteOnce"]
31          resources:
32            requests:
33              storage: 20Gi
34
35grafana:
36  deploymentStrategy:
37    type: Recreate
38  persistence:
39    enabled: true
40    type: pvc
41    storageClassName: civo-volume
42    size: 5Gi
43  resources:
44    requests:
45      cpu: 100m
46      memory: 256Mi

Why these control-plane targets are disabled

On Civo, the control plane is managed outside your worker nodes, so scheduler, controller-manager, and etcd are not useful scrape targets from the customer cluster. K3s typically still runs kube-proxy, but it usually does not expose a separate, useful scrape endpoint the way a classic kubeadm install would.

Without disabling these components, several default alerts fire when those metrics are missing — often through expressions such as absent(...) — which can leave critical alerts such as KubeSchedulerDown and KubeControllerManagerDown firing permanently. Setting enabled: false prevents those ServiceMonitors and associated default monitoring resources from being installed — don't just silence the alerts, or you'll inherit the same noise on every new cluster.

KubeMemoryOvercommit is disabled for a similar chart-default reason: on small multi-node Civo clusters that also host the monitoring stack, total memory requests often trip the alert immediately even when the cluster is healthy. Behavior can vary by node size and workload mix, so revisit this override if your cluster is larger or you rely on overcommit warnings.

Why storage class and retention are set explicitly

civo-volume is already the default StorageClass, but pinning it prevents an unexpected change if the cluster default is modified later.

The chart's default retention is time-based and may not include a size limit appropriate for your PVC. Per Civo docs at the time of writing, volume expansion is offline: the volume must be unmounted before it can grow, which usually means stopping Prometheus. Set both retention and retentionSize, and keep retentionSize slightly under the PVC size so Prometheus can compact before the volume fills.

Why Grafana uses Recreate

civo-volume is ReadWriteOnce. A rolling update tries to start the new Grafana pod before the old pod releases the volume. That commonly results in a multi-attach error and a stuck rollout.

Recreate stops the old pod first. Grafana is briefly unavailable during upgrades, which is acceptable for this setup. Apply the same pattern to any Deployment that mounts a single RWO PVC on Civo.

Step 4: Preview and deploy

Always preview changes first:

1helmfile diff

Then apply:

1helmfile apply

If helmfile apply fails partway, run helmfile diff to see what still differs from the files, fix the values, and apply again. If the diff looks correct but pods or volumes are still stuck, check kubectl -n monitoring get events --sort-by=.lastTimestamp for PVC attach failures, admission webhook errors, or other cluster-side issues. To undo a bad values change, revert the Git commit (or edit the values file back) and run helmfile apply so the cluster matches the known-good files again.

The first installation can take several minutes while CRDs are installed and persistent volumes attach.
Verify the release:

1helm list -n monitoring
2kubectl -n monitoring get pods
3kubectl -n monitoring get pvc
4kubectl -n monitoring get svc

Expected pods include Prometheus, Alertmanager, Grafana, the operator, kube-state-metrics, and one node-exporter pod per node. PersistentVolumeClaims for Prometheus and Grafana should reach Bound.

Alertmanager runs without a PVC in this guide. Add persistent storage if you need silences and notification state to survive restarts.

Example pod output after a successful apply:

1NAME                                                     READY   STATUS    RESTARTS   AGE
2alertmanager-prometheus-stack-kube-prom-alertmanager-0   2/2     Running   0          2m
3prometheus-prometheus-stack-kube-prom-prometheus-0       2/2     Running   0          2m
4prometheus-stack-grafana-...                             3/3     Running   0          2m
5prometheus-stack-kube-prom-operator-...                  1/1     Running   0          2m
6prometheus-stack-kube-state-metrics-...                  1/1     Running   0          2m
7prometheus-stack-prometheus-node-exporter-...            1/1     Running   0          2m

Pod names and Grafana READY counts can vary slightly by chart version (sidecars). What matters is that the core components are Running and the PVCs are Bound.

If pods stay Pending, check the PVCs — civo-volume uses WaitForFirstConsumer, so a claim can remain Pending until a Pod is scheduled.

Pods in the monitoring namespace

PersistentVolumeClaims bound on civo-volume

Step 5: Verify Prometheus

Port-forward the Prometheus service:

1kubectl -n monitoring port-forward svc/prometheus-stack-kube-prom-prometheus 9090:9090

Open http://localhost:9090/targets.

Confirm that:

• Listed targets for your stack are UP (Grafana, Alertmanager, Prometheus, operator, kube-state-metrics, node-exporter, kubelet, CoreDNS, apiserver where available)
• There are no scheduler, controller-manager, etcd, or kube-proxy target groups

On a fresh Civo cluster, the targets page should look clean.

Prometheus targets for the monitoring stack:

Then open http://localhost:9090/alerts.

The Watchdog alert should be firing. That is expected. It is an always-on heartbeat used to prove Alertmanager delivery still works. You may also see InfoInhibitor firing; that is a companion alert used to reduce noisy info-level notifications.

If KubeSchedulerDown or KubeControllerManagerDown is firing, the values file was not applied. Run helmfile diff and confirm the kubeScheduler and kubeControllerManager blocks are present.

Prometheus alerts showing Watchdog:

Run a basic PromQL check in Prometheus → Graph:

1up

You should see a vector of healthy scrape targets.

Step 6: Access Grafana

Port-forward Grafana:

1kubectl -n monitoring port-forward svc/prometheus-stack-grafana 3000:80

Grafana uses username admin. The password is stored in a Secret as base64. Decode it before logging in — do not paste the encoded string into the login form.

1kubectl -n monitoring get secret prometheus-stack-grafana \
2  -o jsonpath='{.data.admin-password}' | base64 -d
3echo

Open http://localhost:3000 and sign in as admin with the decoded password.

The Prometheus datasource and standard Kubernetes dashboards are provisioned by the chart. Open Kubernetes / Compute Resources / Cluster and confirm node and pod metrics are populated.

Grafana Kubernetes Compute Resources Cluster dashboard:

Port-forward is appropriate while you validate the stack. Exposing Grafana publicly requires an Ingress, TLS, and authentication policy. Leave that for a follow-up change rather than enabling an open LoadBalancer during setup.

Step 7 (optional): Route alerts to Slack

Create the Slack webhook as a Kubernetes Secret. Do not commit the webhook URL into the values file.

1kubectl -n monitoring create secret generic alertmanager-slack \
2  --from-literal=webhook-url='https://hooks.slack.com/services/XXX/YYY/ZZZ'

Append this configuration to values/prometheus-stack.yaml:

1alertmanager:
2  alertmanagerSpec:
3    secrets:
4      - alertmanager-slack
5  config:
6    route:
7      group_by: ['alertname', 'namespace']
8      group_wait: 30s
9      group_interval: 5m
10      repeat_interval: 4h
11      receiver: slack
12      routes:
13        - matchers: ['alertname =~ "Watchdog|InfoInhibitor"']
14          receiver: 'null'
15    receivers:
16      - name: 'null'
17      - name: slack
18        slack_configs:
19          - api_url_file: /etc/alertmanager/secrets/alertmanager-slack/webhook-url
20            channel: '#alerts'
21            send_resolved: true
22            title: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'

api_url_file mounts the webhook from the Secret instead of embedding it in Git.

Apply the change:

1helmfile apply

Two details to keep in mind:

1. Alertmanager matchers are fully anchored. severity =~ "warn" does not match warning.
2. Routing Watchdog to null means your deadman switch is not monitored externally. For production, send Watchdog to an outside heartbeat service such as Dead Man's Snitch or Healthchecks.io so you are notified if Alertmanager itself stops delivering.

Your repository should now match the project layout from Step 2. Changing retention, resources, or alert routing is an edit plus helmfile apply. Recreating the stack on a new cluster is a context switch plus helmfile apply.

Cleanup

To remove the stack:

1helmfile destroy

PersistentVolumeClaims may remain after the release is deleted, depending on reclaim behavior and chart settings. Deleting the Helm release also does not necessarily delete the underlying Civo volume, depending on the StorageClass reclaim policy. Delete unused PVCs deliberately, and remember that detached Civo volumes continue to incur charges until removed.

Key takeaways

• Deploy the complete kube-prometheus-stack with Helmfile.
• Configure Prometheus and Grafana with persistent storage on Civo.
• Disable unsupported managed control plane scrape targets.
• Use Recreate for Grafana to avoid ReadWriteOnce volume conflicts.
• Manage monitoring as code with a version-controlled Helmfile.

Frequently Asked Questions

Summary

This Helmfile setup makes kube-prometheus-stack work cleanly on Civo with three Civo specific fixes.
Disable managed control plane scrape targets (scheduler, controller-manager, etcd, kube-proxy) so Prometheus does not alert on metrics you cannot scrape from the worker side.

Set Prometheus retention and retentionSize so the TSDB disk has a predictable ceiling and growth stays aligned with civo-volume behavior.

Switch Grafana to deploymentStrategy.type: Recreate so upgrades work with civo-volume (ReadWriteOnce) without multi attach issues.

With those tweaks, deploying and updating the stack is a repeatable apply from Git instead of a one off Helm command.