Optimize GPU costs for machine learning with just-in-time Kubernetes scaling

Machine learning projects face an escalating challenge due to the increasing costs of GPU infrastructure. Training modern deep learning models requires substantial computational resources, yet many teams find expensive GPU nodes sitting idle between training runs. This inefficiency turns model iteration into an ongoing budget problem rather than a pure research or product problem.

Just-in-time GPU provisioning solves this by creating capacity only when workloads demand it. Instead of maintaining always-on GPU clusters, teams dynamically scale GPU resources, ensuring accelerators are available for training and inference while avoiding idle-hour charges.

This tutorial walks through building a lightweight, on-demand GPU scheduler that also posts notifications about cost and training events. Specifically, the system will:

• Monitor Kubernetes pods that request GPUs.
• Scale GPU node pools using Civo CLI (or REST API in production).
• Enforce simple utilization and budget guardrails.
• Post Slack/Telegram alerts for scaling and budget events (simulated in this tutorial).

Architecture overview

The solution consists of several components working together:

• Civo Kubernetes cluster: Base CPU-only cluster with dynamic GPU node pools
• Watchdog controller: Monitors workload patterns and triggers scaling decisions
• Cost monitor: Tracks spending against budgets using Civo's billing API
• Alert System: Provides real-time notifications about scaling events and cost thresholds
• Policy engine: Enforces business rules around utilization and spending limits

The resulting architecture combines node labels/taints, a small watchdog script, and lightweight alerts to make GPU spending observable and predictable.

Prerequisites

Keep credentials out of source code by using Kubernetes Secrets. Assign minimal RBAC permissions to any in-cluster automation to follow least-privilege principles.

Step 1: Create a GPU cluster on Civo

Instead of spinning up a separate GPU-only cluster, add a GPU node pool to the existing CPU cluster. This approach keeps workloads organized, CPU nodes handle system tasks and light jobs, while GPU nodes are reserved only for ML training and inference. It also simplifies networking, namespaces, and monitoring since everything stays under one Kubernetes control plane.

There are two approaches to doing this:

• Civo Dashboard:
• • Go to Civo Dashboard → Kubernetes → Create Cluster
  • Choose a standard CPU size (e.g., g4s.kube.small).
  • Set initial node count = 1 to avoid accidental charges during testing.
  • Go to cluster → Create New Pool.
  • Select a GPU-optimized size (e.g., g4g.kube.small).
  • Set initial node count = 1.
• Civo CLI:
• • Create or scale CPU and GPU pools programmatically.
  • Example CLI commands:

1# Step 1: Create a CPU-only cluster
2civo kubernetes create CLUSTER_NAME --size=g4s.kube.small --nodes=1
3
4# Step 2: Add a GPU pool (start with 0 nodes to avoid idle costs)
5civo kubernetes node-pool create CLUSTER_NAME --size=g4g.kube.small --count=1
6
7# Step 3: Scale GPU pool when training starts
8civo kubernetes node-pool scale CLUSTER_NAME NODEPOOL_ID -n NUMBER

The advantage of Civo’s GPU

Unlike other providers, Civo pre-installs NVIDIA drivers, CUDA toolkit, and container runtime on GPU nodes. This eliminates the common "bootstrap time" problem where teams wait 10-15 minutes for driver installation after node creation.

After creating the pool, confirm that nodes join the cluster and that the NVIDIA device plugin or vendor operator is running. GPU nodes require drivers and device plugins to expose nvidia.com/gpu resources in Kubernetes. Here’s how to do it:

1kubectl describe node <gpu-node-name> | grep nvidia.com/gpu

Step 2: Tag GPU workloads

Tagging and taints prevent the accidental scheduling of CPU workloads on GPU nodes, allowing the scheduler to detect GPU work. Inspect Node Labels and Taints:

1kubectl describe node <gpu-node-name>

• Look for labels like node.kubernetes.io/instance-type, nodepool=gpu, or custom tags.
• Note taints like gpu=true: NoSchedule which prevent generic pods from landing there.

Apply Labels, Taints, and Affinity:

If missing, add labels via:

1kubectl label nodes <gpu-node-name> gpu=on-demand
2kubectl taint nodes <gpu-node-name> gpu=true:NoSchedule

If nodes cannot be labelled, skip this step and use nodeAffinity in the job spec (below is the working example).

Example Job Spec (gpu-test.yaml):

1# gpu-test.yaml
2apiVersion: v1
3kind: Pod
4metadata:
5  name: gpu-test
6  Labels:
7    gpu: on-demand
8spec:
9  restartPolicy: OnFailure
10  containers:
11  - name: fake-gpu
12    image: busybox
13    command: ["sh", "-c", "echo 'Simulating GPU work'; sleep 60"]
14    resources:
15      limits:
16        nvidia.com/gpu: 1
17  tolerations:
18  - key: "gpu"
19    operator: "Equal"
20    value: "true"
21    effect: "NoSchedule"
22  affinity:
23    nodeAffinity:
24      requiredDuringSchedulingIgnoredDuringExecution:
25        nodeSelectorTerms:
26        - matchExpressions:
27          - key: gpu
28            operator: In
29            values:
30            - on-demand

Step 3: Write watchdog script

Watchdog is a lightweight Python service that continuously monitors GPU job activity in your cluster, triggers node-pool scaling when needed, and posts notifications for team visibility. Think of it as the “eyes and hands” of the cost-aware GPU scheduler.

Design Principles:

• Monitor pods, not obs: Pod-level inspection (status: Pending/Running) is more reliable.
• Secure authentication: Read the Civo API key from a Kubernetes Secret.
• Least-privilege RBAC: ServiceAccount with read-only access to pods/jobs.
• Backoff & cooldown: Avoid thrashing with exponential backoff or cooldown between scale actions.

Folder structure:

1civo-gpu-scheduler/
2├── gpu-test.yaml        # GPU test pod manifest
3├── watchdog_local.py    # Watchdog script for local simulation
4├── watchdog.py          # Production-ready script (with CLI/API calls)
5└── README.md

Example watchdog (Python, CLI-based for simplicity):

Import dependencies and configure logging:

1import time, logging
2logging.basicConfig(level=logging.INFO)

Define state management variables:

1scaled_jobs = set()
2COOLDOWN = 10
3LAST_ACTION = 0

Function to detect pending GPU jobs:

1def pending_gpu_jobs():
2    return ["gpu-test"]  # Replace with real pod query

Continuous Monitoring with Scale-Up and Scale-Down: Loops to check GPU job status, trigger scale-up when pending jobs exist, and cooldown has passed, and trigger scale-down when no jobs exist, and cooldown has passed.

1while True:
2    pending = pending_gpu_jobs()
3    now = time.time()
4
5    # -----------------------------------------------
6    # Scale-Up Logic
7    # -----------------------------------------------
8    # If GPU jobs are waiting and cooldown passed,
9    # simulate a scale-up and send alerts.
10    if pending and (now - LAST_ACTION) > COOLDOWN:
11        for job in pending:
12            logging.info(f"Pending GPU pod detected: {job}, simulating scale-up")
13            print("Scale-up simulated (Civo API/CLI call would go here)")
14            print(f"Slack alert: GPU scale-up triggered for job {job}")
15            print(f"Telegram alert: GPU scale-up triggered for job {job}")
16            scaled_jobs.add(job)
17        LAST_ACTION = now
18
19    # -----------------------------------------------
20    # Scale-Down Logic
21    # -----------------------------------------------
22    # If no jobs remain and cooldown passed,
23    # simulate a scale-down and send alerts.
24    if not pending and scaled_jobs and (now - LAST_ACTION) > COOLDOWN:
25        logging.info("No pending GPU pods: simulating scale-down")
26        print("Scale-down simulated (Civo API/CLI call would go here)")
27        print("Slack alert: GPU scale-down triggered")
28        print("Telegram alert: GPU scale-down triggered")
29        scaled_jobs.clear()
30        LAST_ACTION = now
31
32    # -----------------------------------------------
33    # Sleep Interval
34    # -----------------------------------------------
35    # Avoids busy looping. Runs check every 5 seconds.
36    time.sleep(5)

Alerts are simulated. In production, replace print() with real Slack/Telegram webhooks.

RBAC example

The watchdog requires specific Kubernetes permissions to monitor pods and jobs. This RBAC configuration follows least-privilege principles:

1apiVersion: v1
2kind: ServiceAccount
3metadata:
4  name: gpu-watchdog
5  namespace: kube-system
6---
7apiVersion: rbac.authorization.k8s.io/v1
8kind: Role
9metadata:
10  name: gpu-watchdog-role
11  namespace: kube-system
12rules:
13- apiGroups: [""]
14  resources: ["pods"]
15  verbs: ["get", "list", "watch"]
16- apiGroups: ["batch"]
17  resources: ["jobs"]
18  verbs: ["get", "list", "watch"]
19---
20apiVersion: rbac.authorization.k8s.io/v1
21kind: RoleBinding
22metadata:
23  name: gpu-watchdog-binding
24  namespace: kube-system
25subjects:
26- kind: ServiceAccount
27  name: gpu-watchdog
28  namespace: kube-system
29roleRef:
30  kind: Role
31  name: gpu-watchdog-role
32  apiGroup: rbac.authorization.k8s.io

Step 4: Set cost guardrails

Effective cost management requires proactive guardrails that prevent budget overruns while maintaining development velocity. Civo's transparent billing model makes implementing these controls straightforward.

Suggested guardrails

Metrics and signals

Prefer telemetry over heuristics. Deploy NVIDIA DCGM exporter and Prometheus to compute nvidiagpuutilization. Use PromQL rolling averages to implement thresholds. If metrics are not available, fall back to pod-count heuristics.

Budget checking via Civo API

Civo exposes billing endpoints to list account charges. Use the charges API to compute cumulative usage and enforce budget caps. Example API resource: GET https://api.civo.com/v2/charges. Parse responses safely and handle missing fields.

Example budget check (safe parsing and error handling recommended):

1import requests, os
2CIVO_API_KEY = os.getenv("CIVO_API_KEY")
3HEADERS = {"Authorization": f"bearer {CIVO_API_KEY}"}
4
5
6def get_charges():
7    resp = requests.get("https://api.civo.com/v2/charges", headers=HEADERS, timeout=10)
8    resp.raise_for_status()
9    return resp.json()

Step 5: Test the workflow

Comprehensive testing ensures the system behaves correctly under various scenarios, from normal operations to edge cases and failure conditions.

Testing steps

1. Deploy the watchdog as a Deployment in the cluster.
2. Submit a short GPU job labeled gpu=on-demand.
3. Verify the job becomes Pending → watchdog detects it → simulated scale-up occurs.
4. Confirm simulated Slack/Telegram alerts appear with relevant context.
5. Observe job runs on GPU node → completion triggers scale-down and alert.
6. Test error conditions: API failures, network issues, budget limits.

Here’s an example of the output you should see after running the watchdog script.

Monitor the dashboard and billing for expected activity. Pre-pull images or maintain a warm node for faster startup if needed.

Troubleshooting

Common issues and their solutions when deploying the GPU scheduler:

• Pod pending: Check that tolerations/affinity labels match GPU node configuration exactly. Mismatched labels are the most common cause of scheduling failures.
• Cold start issues: Civo's fast provisioning helps, but image pulls can still add 2–5 minutes. Pre-pull common images or maintain standby nodes for latency-sensitive workloads.
• Webhook failures: Check logs and network egress rules. Ensure the cluster can reach external notification services.
• Quota/API errors: Inspect CLI/API output carefully. GPU quotas may limit scaling, especially for new Civo accounts.

Summary

By leveraging Civo's flexible node-pool management, strategic workload tagging, and telemetry-driven scaling, you can optimize your GPU usage and costs. Implementing best practices such as using Secrets for API keys, minimal RBAC, and cooldown windows can further enhance your deployment's efficiency and reliability. With these strategies, you can ensure predictable and policy-driven GPU spending. By following these guidelines, you'll be well on your way to maximizing the potential of your GPU-accelerated workloads on Civo.

If you are looking to learn more about some of the topics discussed in this tutorial, check out these resources:

• Managing Node Pools and GPU Configuration
• Civo Scaling Node Pools with CLI and API
• Civo Cloud API: Charges Endpoint