Run three small LLMs on one Civo GPU with NVIDIA time-slicing

Small language models rarely need an entire GPU. A model with a few billion parameters uses only a few gigabytes of memory and a fraction of the compute available on a card like the L40S, leaving much of the GPU idle when a single model is running on it. The problem gets worse when you need to serve several models at the same time, as the usual approach is to give each model its own GPU, meaning you pay for multiple cards that are only partially utilized. This happens because Kubernetes treats a GPU as a single resource, giving one pod access to the GPU without a built-in way to divide it between workloads, so adding another model typically means adding another GPU. 

Time-slicing provides a way around this by using a small ConfigMap to tell NVIDIA's device plugin to expose one physical GPU as multiple schedulable slots, allowing Kubernetes to place a separate pod on each slot. 

In this tutorial, we'll run three small models on a single L40S, each with its own endpoint, while addressing the part that often causes problems: Time-slicing shares GPU compute, with GPU memory still shared by all workloads.

What you'll build

You'll set up one Civo Kubernetes cluster with an L40S GPU node and configure the GPU to handle three workloads at the same time.

The setup has four parts:

• Time-slicing ConfigMap: You'll configure the NVIDIA device plugin to expose the L40S as three GPU slots.
• Three vLLM servers: You'll run three different models (Qwen3-1.7B, SmolLM3-3B, and Qwen3-4B), each in its own Kubernetes Deployment.
• Three Services: You'll give each model its own Service and endpoint.
• Memory limits: You'll cap the GPU memory available to each server so the models don't compete for the same memory.

Once everything is up and running, you can send requests to all three models through their own endpoints while they share the same L40S. You'll also test the setup without memory limits so you can see firsthand why time-slicing alone isn't enough.

Prerequisites 

Before you begin, make sure your Civo account is ready and the required tools are installed.

Account

• A Civo account with access to Kubernetes and GPU nodes. New accounts get $250 in free credit for the first month. 

Local tools

• Civo CLI for provisioning the cluster and GPU node pool
• kubectl for communicating with the cluster
• Helm for installing the NVIDIA GPU Operator

Tested versions

Project structure

1llm-timeslicing/
2└── kubernetes/
3    ├── namespace.yaml               # creates the llms namespace
4    ├── timeslicing-configmap.yaml   # tells the device plugin to expose 3 GPU slots
5    ├── deployment-qwen-1_7b.yaml     # Qwen3-1.7B server + Service
6    ├── deployment-smollm3-3b.yaml    # SmolLM3-3B server + Service
7    └── deployment-qwen-4b.yaml       # Qwen3-4B server + Service

With the required tools installed and your Civo account set up, you're ready to work through the tutorial from start to finish.

How it fits together

Before deploying anything, it helps to see what each part does. Each piece has one job.

Image by Author

The diagram shows how the setup fits together, from bottom to top: one physical L40S GPU is exposed as three GPU slots by the NVIDIA device plugin, with a model server and its own endpoint running on each slot. All three workloads run inside a single Civo Kubernetes cluster.

At the bottom is the L40S GPU node. Without time-slicing, that GPU would typically be assigned to a single model server. Here, you'll configure it to handle three.

The NVIDIA GPU Operator provides the drivers and device plugin needed to make the GPU available to Kubernetes as the nvidia.com/gpu resource. By default, the device plugin reports a single GPU.

The time-slicing ConfigMap changes this so Kubernetes sees three GPU slots instead. The physical GPU switches between workloads running on those slots. These slots share the GPU's compute by taking turns, but they do not divide its memory: every server sees the whole card and can try to claim all of it.

At the top, three vLLM servers run the models: Qwen3-1.7B, SmolLM3-3B, and Qwen3-4B. Each server has its own endpoint and a memory cap (vLLM's gpu-memory-utilization) so the three models can run on the same GPU without evicting each other.

The workflow works as follows:

• The GPU Operator and ConfigMap expose the L40S as three schedulable GPU slots.
• Each Deployment runs on one slot and loads its assigned model.
• A memory limit keeps the three models within the GPU's 48 GB of memory.
• All three endpoints can handle requests concurrently, sharing the same physical GPU.

Each component has a specific job: the GPU Operator makes the GPU available to Kubernetes, the ConfigMap enables time-slicing, vLLM serves the models, and the memory limits prevent one model from consuming memory needed by the others.

Creating the cluster

With the tools installed, create a Kubernetes cluster with two node pools: a small CPU node for system workloads and an L40S GPU node for the models.

This setup keeps system components, GPU Operator controllers, DNS, and metrics on the cheaper CPU node, leaving the GPU node dedicated to model serving and avoiding resource contention.

Authenticate the Civo CLI and set your region:

1civo apikey save my-key YOUR_CIVO_API_KEY
2civo apikey use my-key
3civo region use NYC1

Create the cluster with a small CPU node:

1civo kubernetes create llm-timeslicing \
2  --size g4s.kube.small \
3  --nodes 1 \
4  --region NYC1 \
5  --wait

The --wait flag keeps the terminal open until the cluster is ready, which takes about 90 seconds. Then add the L40S GPU node pool:

1civo kubernetes node-pool create llm-timeslicing \
2  --size an.g1.l40s.kube.x1 \
3  --nodes 1 \
4  --region NYC1

The GPU node takes a few minutes to join. Save the kubeconfig so kubectl can connect:

1civo kubernetes config llm-timeslicing --region NYC1 --save

Verify both nodes are ready:

1kubectl get nodes

You'll see two nodes: the small CPU node and the L40S GPU node. Both must show Ready before moving on.

Two nodes in Ready state

At this point the cluster is running, but the GPU is not yet visible to Kubernetes. Making it visible is what you'll set up in the next section.

Enabling time-slicing

Making one GPU appear as three takes two steps: install the NVIDIA GPU Operator so Kubernetes can detect the GPU, then apply a ConfigMap that tells the device plugin to split it into slots.

Install the NVIDIA GPU operator

The GPU Operator installs the NVIDIA driver and device plugin, which makes the GPU available to Kubernetes. Since Civo's GPU image already includes the NVIDIA Container Toolkit, the Operator is installed with it disabled.

Add the NVIDIA Helm repository:

1helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
2helm repo update

Install the Operator:

1helm upgrade --install gpu-operator \
2  -n gpu-operator --create-namespace \
3  nvidia/gpu-operator \
4  --version 25.10.1 \
5  --set driver.enabled=true \
6  --set toolkit.enabled=false \
7  --set devicePlugin.enabled=true \
8  --set gfd.enabled=true \
9  --set operator.defaultRuntime=containerd \
10  --set validator.cuda.runtimeClassName=nvidia

A STATUS: deployed message means the install succeeded. The Operator then starts several pods on the GPU node. The driver loads first, and the other components restart around it, so give it a few minutes to settle:

1kubectl get pods -n gpu-operator -w

Wait until the driver, device plugin, and validators are all Running or Completed, then press Ctrl+C.

gpu-operator pods running

With the Operator running, the GPU is now visible to Kubernetes. Check how many the node reports:

1kubectl get nodes -o custom-columns="NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"

The node shows GPU = 1. One physical card, one schedulable unit.

node showing GPU = 1

Apply the Time-Slicing ConfigMap

The ConfigMap tells the device plugin to advertise the single L40S as three slots. Create kubernetes/timeslicing-configmap.yaml:

1apiVersion: v1
2kind: ConfigMap
3metadata:
4  name: time-slicing-config
5  namespace: gpu-operator
6data:
7  three-slots: |-
8    version: v1
9    flags:
10      migStrategy: none
11    sharing:
12      timeSlicing:
13        resources:
14          - name: nvidia.com/gpu
15            replicas: 3

The replicas: 3 line is what turns one GPU into three slots. Apply it:

1kubectl apply -f kubernetes/timeslicing-configmap.yaml

Then point the device plugin at the ConfigMap:

1kubectl patch clusterpolicies.nvidia.com/cluster-policy \
2  -n gpu-operator --type merge \
3  -p '{"spec": {"devicePlugin": {"config": {"name": "time-slicing-config", "default": "three-slots"}}}}'

The device plugin restarts to pick up the new configuration. After 30 to 60 seconds, check the node again with the same command as before:

1kubectl get nodes -o custom-columns="NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"

The node now reports GPU = 3. The physical card hasn't changed, but Kubernetes now sees three schedulable slots where there was one.

node showing GPU = 3

With three slots available, the cluster is ready to run three model servers on the single L40S, which is what you'll deploy next.

Choosing the three models 

The three models are small, capable, and easy to deploy: Qwen3-1.7B for fast general responses, SmolLM3-3B from Hugging Face, and Qwen3-4B for tasks needing more reasoning. This gives you a realistic mix of model sizes and capabilities.

All three are Apache 2.0 licensed and ungated, so there are no access requests or tokens to manage. vLLM also downloads the weights automatically from the specified Hugging Face repository when each pod starts.

Fitting all three on one L40S comes down to memory. With time-slicing, each server sees the full 48GB, so vLLM may try to use most of it by default. Without a limit, the servers would compete for the same memory.

The --gpu-memory-utilization flag solves this by capping each server. Setting it to 0.28 gives each roughly 13GB, keeping total usage around 40GB. A shorter context window and disabled graph capture reduce the footprint further.

Time-slicing divides GPU compute, not memory, so each server needs its own memory cap. With that in place, you're ready to create the three Deployments.

Deploying the three model servers

Each model runs in its own Deployment with a matching Service. The three files are identical except for the model name and labels, so we'll show one in full and only the lines that change for the others.

First, create a namespace to keep the model resources together:

1apiVersion: v1
2kind: Namespace
3metadata:
4  name: llms

Apply it:

1kubectl apply -f kubernetes/namespace.yaml

Here is the Deployment and Service for the first model, kubernetes/deployment-qwen-1_7b.yaml. For now it runs vLLM with just the model and port, the plainest possible configuration:

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: qwen-1-7b
5  namespace: llms
6  labels:
7    app: qwen-1-7b
8spec:
9  replicas: 1
10  selector:
11    matchLabels:
12      app: qwen-1-7b
13  template:
14    metadata:
15      labels:
16        app: qwen-1-7b
17    spec:
18      containers:
19        - name: vllm
20          image: vllm/vllm-openai:latest
21          args:
22            - "--model"
23            - "Qwen/Qwen3-1.7B"
24            - "--port"
25            - "8000"
26          ports:
27            - containerPort: 8000
28          resources:
29            limits:
30              nvidia.com/gpu: 1
31          volumeMounts:
32            - name: cache
33              mountPath: /root/.cache/huggingface
34            - name: shm
35              mountPath: /dev/shm
36          readinessProbe:
37            httpGet:
38              path: /health
39              port: 8000
40            initialDelaySeconds: 120
41            periodSeconds: 10
42          livenessProbe:
43            httpGet:
44              path: /health
45              port: 8000
46            initialDelaySeconds: 600
47            periodSeconds: 15
48      volumes:
49        - name: cache
50          emptyDir: {}
51        - name: shm
52          emptyDir:
53            medium: Memory
54            sizeLimit: 2Gi
55---
56apiVersion: v1
57kind: Service
58metadata:
59  name: qwen-1-7b
60  namespace: llms
61spec:
62  selector:
63    app: qwen-1-7b
64  ports:
65    - port: 8000
66      targetPort: 8000

A few lines do the important work here:

• image: vllm/vllm-openai:latest uses the official vLLM image, so there’s no image to build. vLLM downloads the model from Hugging Face when the pod starts.
• nvidia.com/gpu: 1 gives the pod one of the three GPU slots created by time-slicing.
• The livenessProbe waits before checking the pod, giving the model enough time to download and load.

For the second model, copy the file to deployment-smollm3-3b.yaml. Replace the model name and app label with smollm3-3b everywhere they appear, then update the model in args to:

1           - "--model"
2            - "HuggingFaceTB/SmolLM3-3B"

For the third model, copy it to deployment-qwen-4b.yaml, change the name and app label to qwen-4b in the same places, and set the model to:

1           - "--model"
2            - "Qwen/Qwen3-4B"

Apply all three:

1kubectl apply -f kubernetes/deployment-qwen-1_7b.yaml
2kubectl apply -f kubernetes/deployment-smollm3-3b.yaml
3kubectl apply -f kubernetes/deployment-qwen-4b.yaml

The three servers are now scheduled, one per GPU slot, and will start downloading their models from Hugging Face. But when all three try to load at the same time, this basic setup runs into a problem. We’ll fix that in the next section.

Watching the pods crash, and fixing it

Watch the pods as they start:

1kubectl get pods -n llms -w

The pods pull their weights over a few minutes, but they don't settle into Running. Instead, they restart and fall into CrashLoopBackOff:

pods in CrashLoopBackOff

This is expected, and it's the most important lesson in the tutorial. Check the logs of one of the failing pods to see why:

1kubectl logs -n llms -l app=qwen-4b --previous | grep -m1 "Free memory on device" 

The log ends with a ValueError from vLLM: there wasn't enough free memory on the card for the server to start.

The card shows roughly 44 GiB instead of the advertised 48 GB. That's normal: manufacturers use decimal GB, while nvidia-smi and vLLM use binary GiB. 48 GB is about 44.7 GiB, with a small amount also reserved for ECC. 

Here's the real problem. With time-slicing, all three servers see the full GPU. By default, vLLM tries to reserve most of that memory for itself. The first server takes the bulk of it, leaving too little for the others, which then fail and restart. 

The key point is that time-slicing shares GPU compute, not memory. Each server therefore needs its own memory limit.

Three flags keep memory usage under control. Add them to the args section of all three files, so the full list reads:

1          - "--gpu-memory-utilization"
2            - "0.28"
3            - "--max-model-len"
4            - "2048"
5            - "--enforce-eager"

It's worth keeping all three flags because each solves a different memory problem:

• --gpu-memory-utilization 0.28 limits each server to about 28% of the GPU, leaving room for all three.
• --max-model-len 2048 reduces the context window and, in turn, the KV cache size.
• --enforce-eager disables CUDA graph capture, which can otherwise use several extra GB per server.

The memory cap alone may not be enough. A large context window and CUDA graphs can still push a server over its limit. Together, these settings keep each server small enough to run alongside the others.

Since the failed pods may still hold their GPU slots, the cleanest fix is to delete the Deployments and recreate them:

1kubectl delete deployment -n llms --all
2kubectl apply -f kubernetes/deployment-qwen-1_7b.yaml
3kubectl apply -f kubernetes/deployment-smollm3-3b.yaml
4kubectl apply -f kubernetes/deployment-qwen-4b.yaml

Watch them again:

1kubectl get pods -n llms -w

This time each server keeps to about 28% of the card, the three fit alongside each other, and all three reach Running. Each pod may restart once as it loads, then settle.

All three pods Running

With the flags in place, the collision is gone and the three models run side by side on the single L40S. The next step is to send each one a request and confirm the endpoints work.

Testing the three endpoints 

With all three pods running, each model now exposes an OpenAI-compatible API inside the cluster. To access them from your machine, forward each Service to a local port. Run the port-forwards in the background so all three stay available at once:

1kubectl port-forward -n llms svc/qwen-1-7b 8001:8000 &
2kubectl port-forward -n llms svc/smollm3-3b 8002:8000 &
3kubectl port-forward -n llms svc/qwen-4b 8003:8000 &

Each model is now on its own local port: Qwen3-1.7B on 8001, SmolLM3-3B on 8002, and Qwen3-4B on 8003.

Three port-forwards running in the background

Confirm all three are up by listing the model each endpoint serves:

1for p in 8001 8002 8003; do curl -s http://localhost:$p/v1/models | jq -r '.data[0].id'; done

Each endpoint returns its model name, confirming all three are live at the same time:

All three model names returned

Now send an actual request. This command asks each model the same question and prints its answer: 

1declare -A models=( [8001]="Qwen/Qwen3-1.7B" [8002]="HuggingFaceTB/SmolLM3-3B" [8003]="Qwen/Qwen3-4B" )
2for p in 8001 8002 8003; do
3  echo "=== ${models[$p]} ==="
4  curl -s http://localhost:$p/v1/chat/completions \
5    -H "Content-Type: application/json" \
6    -d "{\"model\": \"${models[$p]}\", \"messages\": [{\"role\": \"user\", \"content\": \"In one sentence, what is Kubernetes?\"}]}" \
7    | jq -r '.choices[0].message.content'
8  echo ""
9done

Each model answers the same question in its own words, three separate models responding from one GPU: 

A completion from each of the three endpoints

The wording differs slightly between them, which confirms they're genuinely three different models rather than one answering three times. With the endpoints working, the final check is to confirm they really are sharing one card. 

Verifying the sharing is real

The three endpoints work, but from the outside, there’s no way to tell whether they’re sharing one GPU or using three. To verify, check the card directly with nvidia-smi. First, inspect the GPU from inside one pod, then check what each model is using:

1echo "=== GPU ==="
2kubectl exec -n llms deploy/qwen-1-7b -- nvidia-smi \
3  --query-gpu=name,memory.used,memory.total,memory.free --format=csv,noheader
4
5echo ""
6echo "=== Per-model processes ==="
7for d in qwen-1-7b smollm3-3b qwen-4b; do
8  line=$(kubectl exec -n llms deploy/$d -- nvidia-smi \
9    --query-compute-apps=pid,used_gpu_memory --format=csv,noheader)
10  echo "$d  ->  $line"
11done

The output shows a single L40S with all three models on it:

GPU and per-model memory output

This confirms it's working. One NVIDIA L40S has about 31GB of its memory in use, with all three model servers sharing the same card. If they were running on separate GPUs, you'd see only one server on this card.

That's the setup in action: one L40S split into three schedulable slots, running three models at once, with memory limits keeping them from interfering with each other. A single GPU that could previously serve only one workload is now serving three.

What's next

This setup gives you a foundation you can extend without changing the core approach:

• Run more models per GPU by increasing replicas and lowering each server’s memory cap.
• Mix model sizes by giving larger models more memory and smaller ones less.
• Add a gateway to route traffic through a single endpoint instead of three.
• Scale to multiple GPUs by adding GPU nodes. Kubernetes handles scheduling across their slots.
• Use stronger isolation with NVIDIA MIG on supported GPUs.

The core stays the same: the ConfigMap, Deployments, and memory caps remain in place. You only need to adjust the scale and surrounding infrastructure.

Key takeaways

You served three separate models from one L40S by splitting it into three schedulable slots, bringing the GPU cost to roughly a third of running three cards. The key lesson is that time-slicing shares compute, not memory. The slots take turns using the GPU, but each server still needs a memory cap.

A shorter context window and disabled graph capture help keep each server small enough to coexist. Since the cluster, GPU node, and networking all run under one Civo account and bill, the cost per endpoint is roughly a third of the three-GPU approach.