Get up and running with Kubeflow on Kubernetes

One of the most important projects when it comes to MLOps on Kubernetes is Kubeflow. In 2026, AI-powered experiences are fast becoming the default for many teams; as such, Kubeflow is a natural choice for teams either already on Kubernetes or looking to train or serve models using the planet's most popular container orchestrator.

This guide takes you from an empty Civo account to a working Kubeflow install with the Central Dashboard reachable over real HTTPS, signed in, and running a Jupyter notebook. It is written to be a clean, self-contained starting point, so a later tutorial can pick up this same cluster and serve a model on it.

What is Kubeflow?

If you're unfamiliar, Kubeflow is an open-source platform for machine learning and MLOps on Kubernetes introduced by Google. The different stages in a typical machine learning lifecycle are represented with different software components in Kubeflow, including model development, model training, model serving, and automated machine learning.

Those components are separate projects that install together as a platform. You will use each of them from the same dashboard:

• Kubeflow Notebooks give you managed JupyterLab, VS Code, or RStudio servers running inside the cluster for model development. Each notebook gets its own CPU, memory, and storage.
• The Training Operator runs distributed training jobs for PyTorch, TensorFlow, and others as Kubernetes-native custom resources, so a training run is just another object the cluster schedules.
• KServe handles model serving. It turns a trained model into an autoscaling inference endpoint, including scale-to-zero when the endpoint is idle.
• Katib is the AutoML component. It runs hyperparameter tuning and neural architecture search by launching many trials and comparing their results.
• Kubeflow Pipelines orchestrates the whole thing. You describe an ML workflow as a graph of steps, and Pipelines runs, tracks, and versions each execution.

Underneath, Kubeflow uses Istio for in-cluster routing and Dex for authentication. You do not manage those directly, but it helps to know they are there, because the dashboard sits behind the Istio ingress gateway and every request passes through the Dex login.

Prerequisites

You will need:

• A Civo account.
• The Civo CLI installed and authenticated.
• kubectl installed on your machine.
• kustomize version 5.4.3 or newer, installed as a standalone binary. Kubeflow's manifests use features that the kubectl kustomize built-in does not always handle the same way, so install the real thing:

1brew install kustomize
2kustomize version

Kubeflow is not a light install. The full platform includes Istio, Dex, Notebooks, Pipelines, Katib, KServe, the Training Operator, the Central Dashboard, and Profiles. Upstream recommends at least 8 CPU cores and 16 GB of RAM. The cluster below gives you that with headroom to spare for a notebook and, later, a served model.

Step 1: Create a Civo Kubernetes cluster

First, check which Kubernetes versions Civo currently offers:

1civo kubernetes versions

Kubeflow 1.10 targets Kubernetes 1.32. Civo retires older patch versions quickly, so at the time of writing the closest available stable release is 1.35.0-k3s1, which this guide uses. Kubeflow installs cleanly on it with the two small manifest fixes described in Step 3.

Create a single-node cluster sized for the full platform. A single g4p.kube.medium node gives you 8 vCPU and 32 GB of RAM, which is enough for Kubeflow plus a notebook. Name everything kubeflow-demo so it is easy to find and clean up later:

1civo kubernetes create kubeflow-demo \
2  --region nyc1 \
3  --size g4p.kube.medium \
4  --nodes 1 \
5  --version 1.35.0-k3s1 \
6  --cni-plugin flannel \
7  --wait \
8  --save

The --wait flag holds the terminal until the cluster is ACTIVE, and --save merges the cluster's credentials into your kubeconfig. When it returns, confirm the node is Ready:

1kubectl get nodes -o wide

You can see the cluster's details, including its public IP, with:

1civo kubernetes show kubeflow-demo

Note the External IP in that output. You will need it in Step 4 for the HTTPS hostname. In this guide it is 212.2.245.220.

Civo's k3s ships with Traefik as its ingress controller, listed under Installed Applications as traefik2-nodeport. Leave it in place. You will use it to terminate TLS in Step 4.

Step 2: Fetch the Kubeflow manifests

Clone the manifests repository at the 1.10.0 tag so your install is reproducible:

1git clone --depth 1 --branch v1.10.0 https://github.com/kubeflow/manifests.git
2cd manifests

The full platform is assembled by a single kustomization at example/kustomization.yaml. It pulls in every component: cert-manager, Istio, Dex, oauth2-proxy, Pipelines, Katib, the Central Dashboard, Notebooks, Profiles, the Training Operator, and KServe.

Step 3: Apply two small fixes, then install

Kubeflow 1.10.0 was released against an older kustomize and against image locations that have since changed. Two edits keep the install reproducible on a current cluster. Make them before you install.

Fix 1: the user Profile name

The manifests use kustomize's older variable syntax to fill in a default user Profile. kustomize 5.x removed that feature, so the Profile ships with a literal, invalid name and the install never finishes. Replace the template with the values it was meant to produce:

1cat > common/user-namespace/base/profile-instance.yaml <<'EOF'
2apiVersion: kubeflow.org/v1beta1
3kind: Profile
4metadata:
5  name: kubeflow-user-example-com
6spec:
7  owner:
8    kind: User
9    name: user@example.com
10EOF

Fix 2: the MinIO image

Kubeflow Pipelines ships an object store using an old MinIO image hosted on Google Container Registry. That image path was removed and now returns a 404, which leaves the pipeline API server crash-looping. Point MinIO at the official image, which still accepts the same configuration:

1sed -i.bak \
2  's#gcr.io/ml-pipeline/minio:RELEASE.2019-08-14T20-37-41Z-license-compliance#minio/minio:RELEASE.2022-11-17T23-20-09Z#' \
3  apps/pipeline/upstream/third-party/minio/base/minio-deployment.yaml

Raise the node's inotify limits

Several Kubeflow controllers watch a large number of files. On a fresh node the default inotify limits are too low, and at least one controller (Katib) crash-loops with a "too many open files" error. Raise the limits on the node with a one-off privileged pod:

1kubectl apply -f - <<'EOF'
2apiVersion: v1
3kind: Pod
4metadata:
5  name: node-sysctl-fix
6  namespace: kube-system
7spec:
8  restartPolicy: Never
9  hostPID: true
10  containers:
11  - name: sysctl
12    image: busybox:1.36
13    securityContext:
14      privileged: true
15    command:
16    - /bin/sh
17    - -c
18    - |
19      sysctl -w fs.inotify.max_user_instances=1280
20      sysctl -w fs.inotify.max_user_watches=655360
21EOF

Install

Now apply the platform. Kubeflow's own installer runs the apply in a loop, because custom resource definitions and the resources that depend on them are applied together, and the dependent resources fail until their CRDs are registered. Re-running resolves the ordering:

1while ! kustomize build example | kubectl apply --server-side --force-conflicts -f -; do
2  echo "Retrying to apply resources"
3  sleep 20
4done

The first pass applies the CRDs and namespaces, and later passes fill in the rest. Give it a few minutes.

When the loop settles, watch the pods come up. The install is spread across several namespaces, so watch them all:

1kubectl get pods -A

It is normal for pods to sit in Init or ContainerCreating for a while, and for a few to restart once or twice as their dependencies start. Wait until everything in the kubeflow namespace is Running and Ready:

1kubectl get pods -n kubeflow

Applying the fixes above also creates the default user Profile and its namespace, kubeflow-user-example-com. That is the namespace you will work in from the dashboard.

Step 4: Put the dashboard on HTTPS

The dashboard has to be served over HTTPS. Kubeflow's web apps set secure cookies for the login session, and browsers will not send those cookies over plain HTTP, so an HTTP-only dashboard cannot keep you logged in.

The dashboard itself sits behind Istio's ingress gateway, which inside the cluster only listens on plain HTTP. Rather than hand-editing Istio to terminate TLS, using Traefik is an easy way out.

If you don’t own a domain yet. sslip.io is a free wildcard DNS service that resolves any <ip>.sslip.io hostname to the IP embedded in it. Using your cluster's public IP, 212.2.245.220, the hostname becomes 212-2-245-220.sslip.io.

 Substitute your own node's IP throughout this step.

Create a Let's Encrypt issuer

cert-manager needs an issuer that knows how to talk to Let's Encrypt. This one solves the HTTP-01 challenge through Traefik:

1kubectl apply -f - <<'EOF'
2apiVersion: cert-manager.io/v1
3kind: ClusterIssuer
4metadata:
5  name: letsencrypt-prod
6spec:
7  acme:
8    server: https://acme-v02.api.letsencrypt.org/directory
9    email: you@example.com
10    privateKeySecretRef:
11      name: letsencrypt-prod-account-key
12    solvers:
13    - http01:
14        ingress:
15          class: traefik
16EOF

Change the email to your own. It is only used by Let's Encrypt for expiry notices.

Request the certificate

Ask cert-manager for a certificate for your hostname. It will run the challenge and write the signed certificate into a secret named kubeflow-dashboard-tls:

1kubectl apply -f - <<'EOF'
2apiVersion: cert-manager.io/v1
3kind: Certificate
4metadata:
5  name: kubeflow-dashboard
6  namespace: istio-system
7spec:
8  secretName: kubeflow-dashboard-tls
9  issuerRef:
10    name: letsencrypt-prod
11    kind: ClusterIssuer
12  commonName: 212-2-245-220.sslip.io
13  dnsNames:
14    - 212-2-245-220.sslip.io
15EOF

Watch it until it reports Ready:

1kubectl -n istio-system get certificate kubeflow-dashboard -w

Tell Traefik to serve the certificate and route to Istio

Civo runs Traefik v2.9, which needs two Traefik resources here. The version that ships with Civo does not have these custom resource definitions installed, so add them first:

1kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v2.9/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml

The first resource is a TLSStore named default. Without it, Traefik keeps presenting its own self-signed certificate instead of the Let's Encrypt one. Pointing the default store at your certificate secret fixes that:d

1kubectl apply -f - <<'EOF'
2apiVersion: traefik.containo.us/v1alpha1
3kind: TLSStore
4metadata:
5  name: default
6  namespace: istio-system
7spec:
8  defaultCertificate:
9    secretName: kubeflow-dashboard-tls
10EOF

The second resource is an IngressRoute that matches your hostname and forwards to the Istio ingress gateway on port 80. Add one route for HTTPS and one for plain HTTP:

1kubectl apply -f - <<'EOF'
2apiVersion: traefik.containo.us/v1alpha1
3kind: IngressRoute
4metadata:
5  name: kubeflow-dashboard
6  namespace: istio-system
7spec:
8  entryPoints:
9    - websecure
10  routes:
11    - match: Host(`212-2-245-220.sslip.io`)
12      kind: Rule
13      services:
14        - name: istio-ingressgateway
15          port: 80
16  tls:
17    secretName: kubeflow-dashboard-tls
18---
19apiVersion: traefik.containo.us/v1alpha1
20kind: IngressRoute
21metadata:
22  name: kubeflow-dashboard-http
23  namespace: istio-system
24spec:
25  entryPoints:
26    - web
27  routes:
28    - match: Host(`212-2-245-220.sslip.io`)
29      kind: Rule
30      services:
31        - name: istio-ingressgateway
32          port: 80
33EOF

You can confirm the routing objects are in place:

1kubectl get ingressroute,tlsstore -n istio-system

Confirm HTTPS works

Check the served certificate and the endpoint. The certificate should be issued by Let's Encrypt, and a plain curl should verify it without the -k flag:

1echo | openssl s_client -connect 212-2-245-220.sslip.io:443 \
2  -servername 212-2-245-220.sslip.io 2>/dev/null \
3  | openssl x509 -noout -issuer -subject -dates
4
5
6curl -sSI https://212-2-245-220.sslip.io/

The 403 from istio-envoy is expected and is a good sign. It means the request reached Kubeflow's authentication layer, which refuses anything that is not signed in. A browser will be redirected to the login page instead.

Step 5: Log in and create a notebook

Open https://<yourip>.sslip.io/ in your browser, using your own hostname. You will land on the sign-in page.

Continue through to the Dex login form. The default credentials that ship with Kubeflow are:

• Username: user@example.com
• Password: 12341234

These defaults are public knowledge, so change them before you put anything real on this cluster. The password lives as a bcrypt hash in the Dex config map (kubectl -n auth edit configmap dex), and the user list is documented in the Kubeflow manifests README. For a throwaway cluster the defaults are fine to explore with.

After logging in, the Central Dashboard loads, scoped to the kubeflow-user-example-com namespace.

To prove the workflow end to end, create a notebook. In the sidebar, choose Notebooks, then New Notebook. Give it a name, leave JupyterLab selected as the image, and set a small CPU and memory request. The defaults of 0.5 CPU and 1 GB are enough to start.

Scroll down and click Launch. The notebook takes a minute or two to pull its image and start. When it is ready, it shows a green status in the list with a Connect button.

Where to go next

You now have a working Kubeflow install with the Central Dashboard on HTTPS, a signed-in session, and a running notebook. The cluster is a clean base for the rest of an ML workflow: training jobs through the Training Operator, hyperparameter sweeps with Katib, pipelines to tie steps together, and model serving with KServe.

When you are done, tear the whole thing down in one command, since every resource carries the kubeflow-demo name:

1civo kubernetes delete kubeflow-demo

That removes the cluster and the volumes Kubeflow created along with it.