Tabby code assistant on Civo L40S GPUs: Private corporate Copilot

Every time a developer pauses mid-function, their editor sends the surrounding code to an external server. For teams working on proprietary algorithms, unreleased features, or internal APIs, this creates a real data governance question — and for regulated industries like finance or healthcare it can be a policy violation.

Commercial code completion tools also price per seat, which means costs scale directly with headcount regardless of how heavily each developer actually uses the tool. For large or fast-growing teams, that model adds up quickly.

Tabby is a self-hosted, open-source inline code completion server that runs entirely inside your own infrastructure with no external database and no cloud dependency. Your VS Code extension points to your server, completions stream back from your own GPU, and your code never leaves the cluster.

This tutorial covers inline code completion specifically, the ghost-text experience that appears as you type and is accepted with a single Tab keypress. Not chat, not agentic workflows, just fast, private, accurate completions in your editor.

The model doing the work is Qwen2.5-Coder 7B, and it fits this stack for three concrete reasons:

• Trained on 5.5 trillion tokens of code across dozens of programming languages
• Achieves state-of-the-art completion accuracy at its size class, consistently outperforming larger models on standard benchmarks
• At 7 billion parameters, it sits comfortably within the 48 GB VRAM of a Civo L40S GPU node, leaving enough headroom for the KV cache that keeps completions fast under team load

By the end of this tutorial, you will have a Tabby inference server running on a Civo Kubernetes cluster, a VS Code extension pointed at it, and inline ghost-text completions streaming back from a GPU that you control.

What you’ll build

A private AI code completion server on Civo Kubernetes that accepts requests from your IDE and streams inline suggestions back from a dedicated GPU node. Your code stays inside the cluster at every step.

The flow works like this:

1. You type a function signature in VS Code and pause
2. The Tabby VS Code extension captures the surrounding context and sends it to your cluster over a kubectl port-forward connection
3. The Tabby pod running on the L40S GPU node passes the context to Qwen2.5-Coder 7B and streams tokens back
4. Ghost-text completions appear inline in your editor, accepted with a single Tab keypress

The core components:

• A Civo Kubernetes cluster with a dedicated L40S GPU node pool running Tabby inference
• A Tabby server pod configured with Qwen2.5-Coder 7B, handling all completion requests from the team
• A Kubernetes persistent volume caching the model weights, so the pod restarts instantly without re-downloading 15 GB
• A Kubernetes Deployment manifest managing the Tabby pod, pinned to the L40S node via a node selector

Typing a function signature is the only manual step in the entire flow.

Prerequisites

Before you run the first command, make sure the following are in place. The tutorial assumes all four tools are installed, and your Civo account has GPU access approved.

• A Civo account with L40S GPU quota approved. GPU quota is not enabled by default, so contact Civo support to request it before you begin.

• Civo CLI manages your cluster from the terminal
• kubectl communicates with your cluster after provisioning
• Helm installs the NVIDIA GPU Operator
• VS Code where completions will appear

Tested versions

This tutorial uses the NYC1 region throughout. Substitute your preferred region in all CLI commands if different.

Project structure

1tabby-civo/
2  kubernetes/
3    tabby-namespace.yaml      # namespace for all Tabby resources
4    tabby-pvc.yaml            # persistent volume for model cache
5    tabby-deployment.yaml     # Tabby pod and ClusterIP service

Once all four tools are installed and your GPU quota is confirmed, you have everything you need to follow this tutorial from start to finish.

How it fits together

Before touching the terminal, it helps to see how the pieces connect. There are four layers involved, and each one has a single responsibility. 

The L40S GPU node is where all the computing happens. Kubernetes schedules the Tabby pod exclusively on this node using a node selector, so the GPU is never shared with system workloads. The 48 GB of VRAM holds the full Qwen2.5-Coder 7B model in FP16, leaving enough headroom for the KV cache that buffers concurrent completion requests.

The Tabby server pod sits on top of the GPU node and does three things: loads the model weights on startup, exposes an HTTP API that the IDE extension talks to, and manages user authentication tokens. It is configured entirely through a Kubernetes Deployment manifest that pins the pod to the L40S node via a node selector.

The persistent volume attaches to the Tabby pod and caches the model weights to disk. Without it, every pod restart triggers a fresh download of 15 GB of weights. With it, the pod restarts in seconds.

The port-forward connection bridges your local VS Code to the cluster. You run kubectl port-forward once, and the Tabby VS Code extension treats localhost:8080 as its endpoint. No ingress, no TLS configuration, no DNS record needed for this deployment.

The request flow:

1. You pause mid-function in VS Code
2. The extension captures the surrounding context and sends it to localhost:8080
3. The port-forward tunnels the request to the Tabby pod on the cluster
4. Tabby passes the context to Qwen2.5-Coder 7B and streams tokens back
5. Ghost text appears in your editor

Every piece runs inside your Civo cluster. Nothing in that flow touches an external server.

Cluster sizing and cost 

This deployment uses two nodes. The CPU node runs system pods and the port-forward process. The L40S GPU node runs the Tabby inference pod exclusively, with Qwen2.5-Coder 7B sitting at roughly 14 GB of the available 48 GB VRAM.

For a 5-hour test, the total comes to roughly $6.45. You pay for the hours the node runs and nothing else.

Creating the cluster

Before deploying anything, you need a Kubernetes cluster with two node pools: one CPU node for system workloads and one L40S GPU node for Tabby inference.

Authenticate the Civo CLI and create the cluster:

1civo apikey save my-key YOUR_API_KEY_HERE
2civo apikey use my-key
3
4civo kubernetes create tabby-cluster \
5  --size=g4s.kube.small \
6  --nodes=1 \
7  --region=NYC1 \
8  --wait

If the CLI times out, run civo kubernetes ls --region=NYC1 and confirm the cluster shows ACTIVE with All Workers Up: True before moving on.

Save the kubeconfig so kubectl can connect to the cluster:

1civo kubernetes config tabby-cluster --region NYC1 --save --switch

Now add the L40S GPU node pool:

1civo kubernetes node-pool create tabby-cluster \
2  --size=an.g1.l40s.kube.x1 \
3  --nodes=1 \
4  --region=NYC1

Wait 3 to 5 minutes, then verify both nodes are Ready:

1kubectl get nodes

Both nodes must show Ready before proceeding. If the L40S node stays NotReady after 5 minutes, run kubectl describe node NODE_NAME and check the Events section at the bottom for the cause.

Installing the NVIDIA GPU operator

Without the GPU Operator, nvidia.com/gpu: 1 in the deployment manifest is not recognized, and the Tabby pod will stay in Pending indefinitely.

1helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
2helm repo update

Install the GPU operator:

1helm upgrade --install gpu-operator \
2  -n gpu-operator --create-namespace \
3  nvidia/gpu-operator \
4  --set driver.enabled=true \
5  --set toolkit.enabled=false \
6  --set devicePlugin.enabled=true \
7  --set gfd.enabled=true \
8  --set operator.defaultRuntime=containerd \
9  --set validator.cuda.runtimeClassName=nvidia

toolkit.enabled=false is intentional. Civo nodes come with the container toolkit pre-configured, and enabling it again causes conflicts.

Wait 5 minutes, then verify all pods are Running or Completed:

1kubectl get pods -n gpu-operator

Confirm the GPU is allocatable:

1kubectl get nodes \
2  -o custom-columns="NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"

The L40S node must show 1 before moving on.

Configuring the Kubernetes manifests 

Three manifest files define the full Tabby deployment. Create them inside the kubernetes/ folder before applying anything. 

Namespace

A dedicated namespace keeps all Tabby resources isolated from system workloads and makes cleanup a single command. 

Create kubernetes/tabby-namespace.yaml:

1apiVersion: v1
2kind: Namespace
3metadata:
4  name: tabby

Every resource in this tutorial lives inside the tabby namespace. Deleting the namespace at the end removes all of them in one shot. 

PersistentVolumeClaim

The PVC caches the Qwen2.5-Coder 7B model weights to disk. Without it, every pod restart re-downloads approximately 15 GB of weights. With it, restarts take seconds. Civo provisions the volume automatically via its CSI driver when the pod first schedules. 

Create kubernetes/tabby-pvc.yaml:

1apiVersion: v1
2kind: PersistentVolumeClaim
3metadata:
4  name: tabby-model-cache
5  namespace: tabby
6spec:
7  accessModes:
8    - ReadWriteOnce
9  resources:
10    requests:
11      storage: 50Gi

50 GB gives the model weights room with space to spare for the SQLite database Tabby uses for user accounts and auth tokens. 

Deployment and service

The nodeSelector pins the pod to the L40S node. The nvidia.com/gpu: 1 resource limit tells Kubernetes to schedule it only on a node with an available GPU. The securityContext ensures the pod can write the SQLite database in the mounted volume. The readinessProbe on /v1/health prevents the pod from showing Ready until the model is fully loaded. 

Create kubernetes/tabby-deployment.yaml:

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: tabby
5  namespace: tabby
6spec:
7  replicas: 1
8  selector:
9    matchLabels:
10      app: tabby
11  template:
12    metadata:
13      labels:
14        app: tabby
15    spec:
16      securityContext:
17        fsGroup: 0
18        runAsUser: 0
19      nodeSelector:
20        nvidia.com/gpu.present: "true"
21      containers:
22        - name: tabby
23          image: registry.tabbyml.com/tabbyml/tabby:latest
24          command:
25            - /opt/tabby/bin/tabby
26          args:
27            - serve
28            - --model
29            - Qwen2.5-Coder-7B
30            - --device
31            - cuda
32          ports:
33            - containerPort: 8080
34          resources:
35            limits:
36              nvidia.com/gpu: 1
37              memory: 32Gi
38            requests:
39              memory: 16Gi
40          volumeMounts:
41            - name: model-cache
42              mountPath: /data
43          readinessProbe:
44            httpGet:
45              path: /v1/health
46              port: 8080
47            initialDelaySeconds: 60
48            periodSeconds: 10
49      volumes:
50        - name: model-cache
51          persistentVolumeClaim:
52            claimName: tabby-model-cache
53---
54apiVersion: v1
55kind: Service
56metadata:
57  name: tabby
58  namespace: tabby
59spec:
60  selector:
61    app: tabby
62  ports:
63    - protocol: TCP
64      port: 8080
65      targetPort: 8080
66  type: ClusterIP

The ClusterIP service gives the Tabby pod a stable internal address at port 8080. It has no external IP and is not reachable from outside the cluster. The port-forward in the next step is what bridges your local machine to it.

Deployment and Secure Exposure

With the three manifest files in place, you have everything Kubernetes needs to schedule the Tabby pod, provision the volume, and expose the service internally. Apply them in order; the namespace must exist before the PVC, and the PVC must exist before the deployment tries to mount it. 

Apply all three manifests:

1kubectl apply -f kubernetes/tabby-namespace.yaml
2kubectl apply -f kubernetes/tabby-pvc.yaml
3kubectl apply -f kubernetes/tabby-deployment.yaml

Watch the pod start:

1kubectl get pods -n tabby -w

The first startup is slow. Tabby pulls the container image (~1.7 GB) and then downloads the Qwen2.5-Coder 7B weights (~15 GB). Expect 15 to 20 minutes on a fresh PVC. Tail the logs to watch the download progress:

1kubectl logs -f deployment/tabby -n tabby

Wait for this line before proceeding: Listening at http://0.0.0.0:8080

On subsequent restarts, the model loads from the PVC in under 30 seconds.

Administrative setup

Tabby requires an admin account before it accepts IDE connections. Without one, every request, including the health check, returns a 401. The web UI handles the initial setup and token generation, and the only way to reach it at this stage is through a port-forward since there is no external endpoint yet. 

Run a port-forward to access the web UI: 

1kubectl port-forward svc/tabby 8080:8080 -n tabby

Open http://localhost:8080 in your browser. You will be prompted to create the first admin account. Set a username and password, then log in.

To generate an authentication token for your IDE:

1. Click your profile icon in the top right
2. Go to Account Settings
3. Copy the token shown there

With the admin account created and the token in hand, the server is ready to accept IDE connections. 

Connecting the IDE and testing the flow

The port-forward must stay running in a separate terminal for the extension to reach the cluster. Open a new terminal and start it before continuing. 

Install the Tabby extension from the VS Code Marketplace. 

Open the Command Palette (Ctrl+Shift+P) and run:

1Tabby: Connect to Server

Enter the following when prompted:

• Endpoint: http://localhost:8080
• Token: the token you copied from Account Settings

After connecting, the Tabby status bar item in the bottom-right corner of VS Code shows a connected indicator.

Testing the flow

Open a new Python file in VS Code and type the following function signature:

1def parse_user_records(records: list[dict], date_format: str) -> list[dict]:

Pause typing after the colon. Within 1 to 3 seconds, ghost text appears inline, suggesting the full function body. Press Tab to accept.

On subsequent requests, latency drops to under 300 ms. If no ghost text appears:

1. Check that the port-forward is still running in the other terminal
2. Run kubectl get pods -n tabby and confirm the pod is 1/1 Running
3. Check that the Tabby status bar item in VS Code is showing connected

Verify the GPU is serving completions:

1# Confirm the server is responding
2curl http://localhost:8080/v1/health
3
4# Confirm authentication is working
5curl -H "Authorization: Bearer YOUR_TOKEN" http://localhost:8080/v1/health

The response should include "device":"cuda" and "cuda_devices":["NVIDIA L40S"].

What's next

The deployment you built here is intentional in its simplicity: one node, one pod, port-forward access. A few natural extensions from here:

• Add more team members by going to the Tabby admin panel and generating a token per developer. Each connects their VS Code extension to the same endpoint.
• Expose over HTTPS by replacing the port-forward with a Civo LoadBalancer and an Nginx ingress with TLS. The cluster endpoint becomes a stable URL your whole team can hit without running a local tunnel.
• Enable chat alongside completions by adding --chat-model Qwen2.5-Coder-7B-Instruct to the deployment args. Both models fit within the L40S VRAM budget, and Tabby's web UI gains a chat panel for code explanation and review.
• Index your repositories by connecting Tabby to your private Git repositories for RAG-based context awareness. Completions become aware of your internal APIs, naming conventions, and existing code patterns.
• Autoscale during office hours by adding a second L40S node pool and configuring a Kubernetes HorizontalPodAutoscaler. The second node spins up at the start of the workday and scales back down at night.

Summary

Your code never left your infrastructure. Every completion request went from your editor to your cluster and back, with no third-party API, no data retention policy to read, and no audit trail on someone else's server.

The economics are straightforward. A single L40S node at $1.29 per hour is a flat infrastructure cost. It does not increase as more developers connect, which is structurally different from per-seat SaaS tools that charge per user per month.

The operational surface stayed small. One cluster, one GPU node, three manifest files, and a port-forward. Nothing required a web console, a managed service account, or a support ticket to provision.