Deploy Llama 3 as a serverless CPU inference endpoint on Kubeflow

Serving a large language model usually means renting a GPU, and GPUs are expensive and often out of stock. For a lot of workloads, though, you do not need one. A small, quantized Llama 3 model runs on ordinary CPU cores, and if you already have a Kubernetes cluster you can serve it there as an autoscaling endpoint that scales to zero when nobody is using it.

This guide takes a quantized Llama 3.2 model, packages it into a small serving container, and deploys it on Kubeflow with KServe. Every file is shown in full, and the endpoint is tested with a real request at the end.

This is a follow-on to Get up and running with Kubeflow on Civo Kubernetes. It assumes you already have the Kubeflow install from that guide, and it reuses the same cluster, dashboard, and namespace.

What is Llama 3

Llama 3 is Meta's family of open-weight language models. The family covers a range of sizes, from very large models aimed at data-center GPUs down to small models meant to run on modest hardware. This guide uses Llama 3.2 1B Instruct, a one-billion-parameter instruction-tuned model. It is small enough to serve comfortably on CPU cores, and quantized it is around 800 MB, which also keeps the serving image small enough to build and push quickly.

If you want more capable answers and can accept slower generation, the same steps work with Llama 3.2 3B Instruct. The one change you make is the model URL, called out where it appears.

The weights are open, so you can download them, quantize them, and serve them yourself without calling a hosted API.

Why Kubeflow and KServe

Kubeflow is a platform for running machine learning workloads on Kubernetes. KServe is its model-serving component. It takes a container that serves predictions and wraps it in an autoscaling, network-addressable endpoint, with the routing, revisions, and scaling handled for you.

A few reasons this is a good fit for CPU inference:

• Scale to zero: KServe runs on top of Knative Serving. When the endpoint is idle it can scale down to zero pods, so an occasionally-used model costs you nothing to keep deployed. The first request after idle pays a cold start while a pod comes back.
• A standard request format: KServe defines a predict protocol, so every model you deploy is called the same way, at /v1/models/<name>:predict.
• It is already installed: If you followed the Kubeflow install in the prerequisite guide, KServe and Knative are already running on your cluster. You do not set anything else up.
• It is just Kubernetes: The endpoint is a custom resource. You deploy it with kubectl, watch it with kubectl get, and delete it with kubectl delete, like anything else on the cluster.

Why CPU, and why quantization

A language model in its original 16-bit precision leans on GPU math to run at a usable speed, and even a small model takes several gigabytes just to hold the weights. Quantization reduces the precision of the weights, for example from 16 bits down to about 4 bits each. That shrinks the file by roughly three quarters and makes it small and cache-friendly enough to run on CPU at a reasonable speed. The 1B model used here drops to around 800 MB once quantized.

The format used here is GGUF, the file format from the llama.cpp project. The specific quantization is Q4_K_M, a 4-bit mix that is the common default for CPU serving because it keeps quality close to the original while cutting the size by roughly three quarters.

The serving library is llama-cpp-python, which wraps llama.cpp so you can load a GGUF file and generate from it in a few lines of Python.

Prerequisites

This guide builds directly on Get up and running with Kubeflow on Civo Kubernetes. Work through that first. It leaves you with:

• A Civo Kubernetes cluster named kubeflow-demo running the full Kubeflow platform.
• The Central Dashboard reachable over HTTPS and a working login.
• The default user profile and its namespace, kubeflow-user-example-com, which is where you will deploy the model.

Because Kubeflow installs KServe and Knative Serving as part of the platform, you do not install anything cluster-side here. You only add the model.

You will also need, on your own machine:

• kubectl, pointed at the kubeflow-demo cluster.
• Docker with Buildx, to build and push the serving image.

Confirm KServe and Knative are healthy before you start. The KServe controller runs in the kubeflow namespace, and Knative Serving runs in knative-serving:

1kubectl get pods -n kubeflow | grep kserve
2kubectl get pods -n knative-serving

The KServe controller in the kubeflow namespace and the Knative Serving control plane, all Running.

A note on cluster size

Kubeflow itself already asks a lot of a cluster, and the model server wants a couple of dedicated CPU cores on top of that. The single node from the install guide does not have that much room to spare.

The clean way to give the model its own space is to add a small node pool for it. On the kubeflow-demo cluster, one extra g4s.kube.large node adds 4 vCPU and 8 GB of RAM:

1civo kubernetes node-pool create kubeflow-demo \
2  --nodes 1 \
3  --size g4s.kube.large

Give it a minute, then confirm the new node is Ready:

1kubectl get nodes

The model pod's CPU request, set later in the InferenceService, is what causes the scheduler to place it on this node.

Step 1: Write the serving container

KServe can run any container that speaks its predict protocol. You are going to write a small Flask app that loads the GGUF file and answers requests at /v1/models/llama3:predict, and a Dockerfile that packages it.

Create a working directory:

1mkdir llama3-serving && cd llama3-serving

app.py

This is the whole server. On startup it downloads the model if it is not already present, loads it, then exposes a readiness path that KServe polls and a predict path that runs generation. The predict path follows the KServe v1 protocol: the request body has an instances list, and the response has a predictions list.

1cat > app.py <<'EOF'
2import os
3import urllib.request
4
5
6from flask import Flask, request, jsonify
7from llama_cpp import Llama
8
9
10MODEL_PATH = os.environ.get("MODEL_PATH", "/models/model.gguf")
11MODEL_URL = os.environ.get(
12    "MODEL_URL",
13    "https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf",
14)
15MODEL_NAME = os.environ.get("MODEL_NAME", "llama3")
16N_THREADS = int(os.environ.get("N_THREADS", os.cpu_count() or 4))
17N_CTX = int(os.environ.get("N_CTX", "2048"))
18
19
20
21
22def ensure_model():
23    if os.path.exists(MODEL_PATH):
24        return
25    os.makedirs(os.path.dirname(MODEL_PATH), exist_ok=True)
26    print("Downloading model from", MODEL_URL, flush=True)
27    urllib.request.urlretrieve(MODEL_URL, MODEL_PATH)
28    print("Model downloaded to", MODEL_PATH, flush=True)
29
30
31
32
33ensure_model()
34
35
36app = Flask(__name__)
37
38
39llm = Llama(
40    model_path=MODEL_PATH,
41    n_ctx=N_CTX,
42    n_threads=N_THREADS,
43    verbose=False,
44)
45
46
47
48
49@app.get("/v1/models/" + MODEL_NAME)
50def ready():
51    # KServe polls this path to decide the predictor is ready.
52    return jsonify({"name": MODEL_NAME, "ready": True})
53
54
55
56
57@app.post("/v1/models/" + MODEL_NAME + ":predict")
58def predict():
59    body = request.get_json(force=True)
60    instances = body.get("instances", [])
61
62
63    predictions = []
64    for instance in instances:
65        prompt = instance.get("prompt", "")
66        max_tokens = int(instance.get("max_tokens", 256))
67        temperature = float(instance.get("temperature", 0.7))
68
69
70        result = llm.create_chat_completion(
71            messages=[{"role": "user", "content": prompt}],
72            max_tokens=max_tokens,
73            temperature=temperature,
74        )
75        predictions.append(result["choices"][0]["message"]["content"])
76
77
78    return jsonify({"predictions": predictions})
79
80
81
82
83if __name__ == "__main__":
84    app.run(host="0.0.0.0", port=8080)
85EOF

A few things worth pointing out:

ensure_model() runs before the server starts. On the first boot it downloads the GGUF to /models; on later boots, if the file is already there, it returns immediately. This is what keeps the model out of the image.

The model name is llama3, set once and used to build both routes. KServe expects the predict path to match the InferenceService name, so keep these aligned.

The server listens on port 8080, which is the port KServe routes to by default.

Each instance in the request can carry its own prompt, max_tokens, and temperature, so a single call can batch several prompts.

Dockerfile

The Dockerfile installs the serving library and copies in the app. It does not include the model. Keeping the model out of the image makes the image small, around 250 MB, which matters in the next step because you push it over your own connection. The model is downloaded once when the pod starts, straight from Hugging Face over the cluster's network, which is fast.

1cat > Dockerfile <<'EOF'
2FROM python:3.11-slim
3
4
5# Install llama-cpp-python from the project's prebuilt CPU wheel index, so no
6# native compile happens in the image build.
7RUN pip install --no-cache-dir \
8      --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu \
9      llama-cpp-python==0.3.34 \
10      flask==3.0.3
11
12
13ENV MODEL_PATH=/models/model.gguf
14ENV MODEL_NAME=llama3
15
16
17COPY app.py /app/app.py
18WORKDIR /app
19
20
21EXPOSE 8080
22CMD ["python", "app.py"]
23EOF

One detail in that Dockerfile matters. The llama-cpp-python install pulls from the project's prebuilt CPU wheel index. That is deliberate. The package's default install on PyPI is a source distribution that compiles llama.cpp from scratch, which is slow and, if you build for a different architecture than your machine, tends to fail. The prebuilt wheel is compiled for linux/amd64 upstream, so there is no compile step in your build at all.

The model URL lives in (and can be overridden with the MODEL_URL environment variable). It points at the bartowski/Llama-3.2-1B-Instruct-GGUF repository on Hugging Face, a public, un-gated re-quantization of Meta's Llama 3.2, so the download needs no Hugging Face token. The Q4_K_M file is about 800 MB.

To use the larger 3B model instead, set MODEL_URL (in app.py or as an environment variable on the container) to the Q4_K_M file in the bartowski/Llama-3.2-3B-Instruct-GGUF repository. It is about 1.9 GB, so the pod takes longer to start on the first pull.

Step 2: Build the image and push it to ttl.sh

The cluster needs to pull this image from a registry it can reach. Rather than set up a registry account, you can use ttl.sh, an anonymous, ephemeral registry. It needs no login. You push to a name you choose, and the image is deleted automatically after a time-to-live that you set in the tag. That is exactly what you want for a throwaway tutorial image.

The convention is to name the image with a unique identifier so it does not collide with anyone else's. Generate one:

1IMG_UUID=$(uuidgen | tr '[:upper:]' '[:lower:]')
2echo "$IMG_UUID"

That gives you an image path of the form ttl.sh/<your-uuid>/llama3-cpu:24h, where the 24h tag tells ttl.sh to keep the image for 24 hours.

One thing to get right: Civo's nodes are amd64, so the image has to be built for linux/amd64. If you are on an Apple Silicon Mac (which is arm64), a plain docker build produces an arm64 image and the pod fails to start with an exec-format error. Build for the target platform explicitly with Buildx, and push in the same step:

1docker buildx build \
2  --platform linux/amd64 \
3  --push \
4  -t "ttl.sh/${IMG_UUID}/llama3-cpu:24h" \
5  .

Most of the time in this step goes to two things: downloading the ~800 MB model to bake into the image, and pushing the finished image up to ttl.sh. The push is bounded by your upload bandwidth, so on a home connection it can take several minutes. Later builds reuse the cached layers.

The cross-architecture build for linux/amd64, pushed straight to ttl.sh.

Step 3: Write the InferenceService and deploy it

The InferenceService is the KServe custom resource that turns your container into an endpoint. This one uses a custom-container predictor pointing at the image you just pushed.

Substitute your own image path (the ttl.sh/<your-uuid>/... line) before applying:

1cat > inferenceservice.yaml <<EOF
2apiVersion: serving.kserve.io/v1beta1
3kind: InferenceService
4metadata:
5  name: llama3
6  namespace: kubeflow-user-example-com
7spec:
8  predictor:
9    annotations:
10      sidecar.istio.io/inject: "false"
11    minReplicas: 1
12    containerConcurrency: 1
13    timeout: 600
14    volumes:
15      - name: model
16        emptyDir: {}
17    containers:
18      - name: kserve-container
19        image: ttl.sh/${IMG_UUID}/llama3-cpu:24h
20        ports:
21          - containerPort: 8080
22            protocol: TCP
23        env:
24          - name: N_THREADS
25            value: "3"
26        volumeMounts:
27          - name: model
28            mountPath: /models
29        resources:
30          requests:
31            cpu: "2"
32            memory: 4Gi
33          limits:
34            cpu: "3"
35            memory: 5Gi
36        readinessProbe:
37          httpGet:
38            path: /v1/models/llama3
39            port: 8080
40          initialDelaySeconds: 20
41          periodSeconds: 10
42          failureThreshold: 30
43EOF

What the important fields do:

• name: llama3 has to match the model name in app.py, because that is what makes the predict path /v1/models/llama3:predict.
• annotations: sidecar.istio.io/inject: "false" turns off the Istio sidecar on the model pod. Kubeflow puts every pod in the kubeflow-user-example-com namespace behind an Istio authorization policy, and with the sidecar in place, calls from a notebook to the endpoint are refused with a 403. Turning the sidecar off on this pod lets in-cluster clients reach it. The endpoint is still only reachable inside the cluster.
• minReplicas: 1 keeps one pod running at all times, which avoids cold starts while you are testing. To get scale-to-zero, set this to 0. The endpoint will then scale down when idle and pay a cold start (the fresh pod downloading and loading the model) on the next request.
• containerConcurrency: 1 tells Knative each pod handles one request at a time, which matches how the model generates.
• volumes and volumeMounts give the container a writable /models directory, which is where app.py downloads the GGUF on startup.
• resources reserve 2 CPU cores for generation. That request is also what steers the pod onto the g4s.kube.large node you added, since the original node does not have that much free. A request much closer to the node's core count leaves no room for the pod to reschedule cleanly, so keep it a little below.
• readinessProbe points at the model's readiness path. The generous failureThreshold gives the container time to download and load the model into memory before KServe marks it ready.

Apply it into the kubeflow-user-example-com namespace:

1kubectl apply -f inferenceservice.yaml

Watch it come up. The InferenceService reports READY: True once the underlying Knative revision is serving and the pod passes its readiness probe:

1kubectl get inferenceservice llama3 -n kubeflow-user-example-com -w

The InferenceService becomes Ready and is assigned an in-cluster URL.

You can watch the pod itself in parallel. It pulls the small image quickly, then spends its startup time downloading the model and loading it into memory:

1kubectl get pods -n kubeflow-user-example-com \
2  -l serving.kserve.io/inferenceservice=llama3

The predictor pod Running. The two containers are the model server and the Knative queue-proxy.

The endpoint also shows up in the Central Dashboard. Open KServe Endpoints in the sidebar, with the namespace set to kubeflow-user-example-com, and you will see llama3 listed with a green Ready status:

The served model in the Central Dashboard, with a custom predictor and the v1 protocol.

Step 4: Run an inference request

With the endpoint Ready, send it a prompt. The predict protocol takes a JSON body with an instances list, and each instance carries the prompt and generation settings.

The simplest way to reach the endpoint from your own machine, without wiring up external ingress, is a short-lived kubectl port-forward to the model's pod. In one terminal, forward the predictor's port:

1kubectl port-forward -n kubeflow-user-example-com \
2  "$(kubectl get pod -n kubeflow-user-example-com \
3    -l serving.kserve.io/inferenceservice=llama3 \
4    -o jsonpath='{.items[0].metadata.name}')" \
5  8080:8080

In a second terminal, send the request:

1curl -s http://localhost:8080/v1/models/llama3:predict \
2  -H "Content-Type: application/json" \
3  -d '{
4    "instances": [
5      {
6        "prompt": "In two sentences, explain what Kubernetes is to a new developer.",
7        "max_tokens": 200,
8        "temperature": 0.7
9      }
10    ]
11  }' | python3 -m json.tool

A real request to the endpoint and the generated response.

The predictions list in the response holds the generated text, one entry per instance you sent.

Reaching it from inside the cluster

Anything running in the cluster, for example a Kubeflow notebook, can call the endpoint by its in-cluster address. KServe assigns the InferenceService a URL, which you can read back:

1kubectl get inferenceservice llama3 \
2  -n kubeflow-user-example-com \
3  -o jsonpath='{.status.url}'

That prints http://llama3.kubeflow-user-example-com.svc.cluster.local. From a notebook in the same namespace, a request looks like this:

1import requests
2
3
4url = "http://llama3.kubeflow-user-example-com.svc.cluster.local/v1/models/llama3:predict"
5payload = {
6    "instances": [
7        {"prompt": "Give me three tips for writing clear commit messages.",
8         "max_tokens": 180}
9    ]
10}
11print(requests.post(url, json=payload).json()["predictions"][0])

This works because you turned the Istio sidecar off on the model pod in Step 3. With the sidecar in place, this same call is refused with a 403.

Calling the same endpoint from a JupyterLab notebook running inside the cluster.

Step 5: Clean up

You can remove the model without touching the rest of the cluster. Deleting the InferenceService removes the endpoint, its Knative revision, and the pod:

1kubectl delete inferenceservice llama3 -n kubeflow-user-example-com

If you added the node pool just for this, and you no longer need it, remove it too. List the pools to get its ID, then delete it:

1civo kubernetes node-pool list kubeflow-demo
2civo kubernetes node-pool delete kubeflow-demo <pool-id>

The image on ttl.sh needs no cleanup. It expires on its own at the time-to-live you set in the tag.

To tear down the whole cluster, including Kubeflow, delete it by name:

1civo kubernetes delete kubeflow-demo

Summary

In this tutorial, we explored the deployment of a production-ready Large Language Model (LLM), specifically Llama 3.1, using Civo’s Kubeflow as a Service on a CPU. This service harnesses the power of Kubernetes and KServe to offer a serverless framework that simplifies the management and scaling of machine learning models.