How to deploy ClearML on Kubernetes for ML experiment tracking

A machine learning model is only useful when it's in production, serving actual users. Getting it to that level is a pipeline, from development to deployment. Each step includes checks, which is where Machine Learning Operations (MLOps) come in.

MLOps is a culture, and experiment tracking is one of its most foundational stages. In regular software development, behaviors are deterministic. If you deploy the same code twice, you get the same behavior. Machine Learning doesn't work that way. Results depend on data, library versions, hyperparameters, processing power, and a dozen other things. Without proper tracking, it's difficult to figure out why a model's outcome changes. As well as why it fails or breaks.

This tutorial focuses on experiment tracking with ClearML, an open-source MLOps platform designed for managing the lifecycle of machine learning applications. By the end, you’ll know what ClearML is and how you can deploy and use it for experiment tracking in Kubernetes.

Understanding ClearML’s architecture

ClearML, by design, includes modules for different machine learning tasks. The experiment tracking manager module is what facilitates the end-to-end control and visibility of machine learning experiments. This includes automated capturing of experimentation logs, metrics, and outputs. Under the experiment tracking manager module, the following runtime components work together:

• ClearML server: This component controls the storage and management of experiments and the tracking workflow in total. From the ClearML server, you can access the ClearML dashboard. It supports databases like MongoDB to store experiment metadata, Elastic Search for searching and indexing, and Redis for caching. It comes in two variants: one that is fully managed by ClearML and the open-source variant, which you can deploy with Docker Compose or Helm.
• ClearML SDK: This is what connects your code to your ClearML server. It’s a Python package, and is used to capture your machine learning code, outputs, hyperparameters, metrics, or anything else for experiment tracking.
• ClearML agent: This is the component that acts as a job scheduler, executing experiment tasks from queues. It orchestrates the experiment tracking workflow and can run on cloud instances and GPUs.

Prerequisites

To follow along, you should have the following:

• A Civo Kubernetes cluster provisioned with at least two nodes and g4s.kube.medium processing/storage power.
• Helm, Pip, and Kubectl installed on your machine. This article uses Kubectl v1.34.
• Python 3.8 or higher required.

Deploy ClearML server

The ClearML server includes three components that work together and communicate internally within Kubernetes. These components are required to be exposed as separate subdomains over HTTP or HTTPS for external access via an ingress controller. They are the API, web, and file servers, and they serve the following purposes:

• API server: To expose endpoints for writing and querying experiment data.
• Web server: To provide a single-page UI to manage experiments, consuming the API server’s endpoint to render experiment data visually.
• File server: To handle the storage of experiment data. From model files, images, to every asset your model uses and produces.

Install a Traefik ingress controller

Before you deploy the ClearML server, you must install an ingress controller in your Kubernetes cluster and configure it to expose your ClearML server externally. You’ll use the Traefik ingress controller in this tutorial and install it using Helm.

Add the Traefik Helm repository:

1$ helm repo add traefik https://traefik.github.io/charts
2"traefik" has been added to your repositories

Update the repository:

1$ helm repo update
2Hang tight while we grab the latest from your chart repositories...
3...Successfully got an update from the "traefik" chart repository
4Update Complete. ⎈Happy Helming!⎈

Install Traefik: 

1$ helm install traefik traefik/traefik
2NAME: traefik
3LAST DEPLOYED: Sat May 30 17:53:27 2026
4NAMESPACE: default
5STATUS: deployed
6REVISION: 1
7DESCRIPTION: Install complete
8TEST SUITE: None
9NOTES:
10traefik with docker.io/traefik:v3.7.1 has been deployed successfully on default namespace!

Confirm that your Traefik pod is running:

1$ kubectl get pods
2
3NAME                                 READY   STATUS      RESTARTS   AGE
4install-traefik2-nodeport-hi-pxkkg   0/1     Completed   0          33m
5traefik-5f694f77f5-sq2j5             1/1     Running     0          2m28s

Check that it has an External IP:

1$ kubectl get svc 
2
3NAME         TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)                      AGE
4traefik      LoadBalancer   10.43.165.247   74.220.23.118   80:31702/TCP,443:32464/TCP   5m22s

ClearML has a demo server you can use temporarily. Using the demo server is not suitable for production workloads, and you will deploy your own ClearML server instead.

Add the Helm chart repository for ClearML in your Kubernetes cluster:

1$ helm repo add clearml https://clearml.github.io/clearml-helm-charts
2
3"clearml" has been added to your repositories

Update the repository:

1$ helm repo update
2
3Hang tight while we grab the latest from your chart repositories...
4...Successfully got an update from the "clearml" chart repository
5Update Complete. ⎈Happhelm install clearml clearml/clearml -f values.yamly Helming!⎈

Create a values.yaml file via Nano and paste in the following configuration settings, replacing <external IP> with the external IP of your ingress controller:

1$ nano values.yaml
2
3apiserver:
4  ingress:
5    enabled: true
6    hostName: "api.<external IP>.nip.io"
7
8  additionalConfigs:
9    apiserver.conf: |
10      auth {
11        fixed_users {
12          enabled: true
13          pass_hashed: false
14          users: [
15            {
16              username: "john"
17              password: "iAmm&^TheOnlyJohnDoe(^n&3$*(++("
18              name: "John Doe"
19            }
20          ]
21        }
22      }
23
24fileserver:
25  ingress:
26    enabled: true
27    hostName: "files.<external IP>.nip.io"
28  persistence:
29    size: 5Gi
30
31webserver:
32  ingress:
33    enabled: true
34    hostName: "app.<external IP>.nip.io"
35
36elasticsearch:
37  enabled: true
38  image: "docker.elastic.co/elasticsearch/elasticsearch"
39  imageTag: "7.17.20"
40  esJavaOpts: "-Xmx1g -Xms1g"
41  resources:
42    requests:
43      cpu: 100m
44      memory: 1Gi
45    limits:
46      cpu: 1000m
47      memory: 2Gi
48  volumeClaimTemplate:
49    resources:
50      requests:
51        storage: 10Gi
52
53mongodb:
54  persistence:
55    size: 10Gi

Here’s an explanation of what’s contained in this file:

• An ingress configuration with a unique nip.io hostname for all three components: the API server, file server, and the web server.
• Set user credentials for the ClearML server. If you don’t set user credentials, you’ll be required to do so via the ClearML server login page.
• An Elastic Search version set to 7.17.20 to resolve an incompatibility with the Cgroup v2 interface used by K3s version 1.34.

• Allocates 10GB for MongoDB and the file server.
• Set 1Gi of Java heap memory for Elasticsearch’s internal process, with overall resource limits set to 1 CPU core and 2Gi RAM.

Save and close the file.

Install ClearML with the following command:

1$ helm install clearml clearml/clearml -f values.yaml
2
3NAME: clearml
4LAST DEPLOYED: Sat May 30 17:53:27 2026
5NAMESPACE: default
6STATUS: deployed
7REVISION: 1
8DESCRIPTION: Install complete
9NOTES:
101. Get the application URL:
11  <http://app.74.220.23.118.nip.io>

Before you access the web server, check that all pods related to ClearML are running:

1$ kubectl get pods -A | grep clearml
2
3default       clearml-apiserver-99455c685-dpggk                1/1     Running     0          4m34s
4default       clearml-apiserver-asyncdelete-565c875667-kfjmw   1/1     Running     0          4m34s
5default       clearml-elastic-master-0                         1/1     Running     0          4m34s
6default       clearml-fileserver-85cd59dbdb-psxcx              1/1     Running     0          4m34s
7default       clearml-mongodb-57f56d8c8b-qnnkk                 1/1     Running     0          4m34s
8default       clearml-redis-master-0                           1/1     Running     0          4m34s
9default       clearml-webserver-cf8c779bd-qxtpf                1/1     Running     0          4m34s
10default       install-helm-clearml-7531-17-vw99k               0/1     Completed   0          21m

Visit the web server http://app.<external IP> .nip.io over your preferred web browser. You’ll be prompted to input a username and password once it loads. Input the username and password you set in your values.yaml file:

Viewing the ClearML server login page

Upon successful login, you’ll be redirected to your dashboard, where you can manage your projects, experiments, pipelines, and so on.

Run the ClearML SDK locally

The ClearML server is just a management console; the component of the experiment tracking management module that actually lives in your code is the ClearML SDK. With the ClearML SDK, you can create experimentation workflows locally, connect them to your ClearML server, and view or manage your workflows from there.

You’ll develop a machine learning project locally (on your computer), install the ClearML SDK, and connect to your ClearML server running on your Civo kubernetes cluster.

One requirement needed to fully set up your project with the ClearML SDK is your workspace credentials. Once you install the SDK, you’ll be required to initialize it with these credentials; therefore, you must obtain them first.

Visit the following address http://app.<external_ip>.nip.io/settings/workspace-configuration. Click on Create Credentials to access and display your credentials. You can either leave it open or copy it to a text editor:

Obtaining workplace credentials on clearML server dashboard

Create a working directory called experiment and open it up with your default code editor:

1$ mkdir experiment && cd experiment && code

From your working directory, create a virtual environment:

1python3 -m venv venv
2source venv/bin/activate

Create a requirements.txt file and paste in the following code:

1$ nano requirements.txt
2
3clearml
4scikit-learn

Install the ClearML SDK as well as Scikit-learn, which you will use to build and train a model:

1$ pip install -r requirements.txt

Initialize the SDK and paste in your credentials when prompted to. Also, allow it some time to verify your credentials:

1$ clearml-init
2
3# Once verified
4
5Verifying credentials ...
6Credentials verified!
7
8New configuration stored in /home/mercy/clearml.conf
9ClearML setup completed successfully.

Next, create three scripts ex1.py , ex2.py , ex3.py and paste in the following code:

1# Experiment 1
2from clearml import Task
3from sklearn.datasets import load_iris 
4from sklearn.ensemble import RandomForestClassifier
5from sklearn.model_selection import train_test_split
6from sklearn.metrics import accuracy_score
7
8# Initialize ClearML Task
9task = Task.init(project_name="Iris Classification", task_name="10 estimators")
10
11# Define hyperparameters
12params = {
13    "n_estimators": 10,
14    "max_depth": 3,
15    "random_state": 42
16}
17task.connect(params)
18
19# Load and split the Iris dataset
20X, y = load_iris(return_X_y=True)
21X_train, X_test, y_train, y_test = train_test_split(
22    X, y, test_size=0.2, random_state=params["random_state"]
23)
24
25# Train the model
26clf = RandomForestClassifier(
27    n_estimators=params["n_estimators"],
28    max_depth=params["max_depth"],
29    random_state=params["random_state"]
30)
31clf.fit(X_train, y_train)
32
33# Log metrics
34logger = task.get_logger()
35train_accuracy = accuracy_score(y_train, clf.predict(X_train))
36test_accuracy = accuracy_score(y_test, clf.predict(X_test))
37
38logger.report_scalar("accuracy", "train", value=train_accuracy, iteration=1)
39logger.report_scalar("accuracy", "test", value=test_accuracy, iteration=1)
40
41print(f"Train accuracy: {train_accuracy:.4f}")
42print(f"Test accuracy: {test_accuracy:.4f}")
43
44task.close()

Use a similar code snippet for ex2.py and ex3.py. But use the following hyperparameters for them, respectively:

1# experiment 2
2task = Task.init(project_name="Iris Classification", task_name="50 estimators")
3params = {
4    "n_estimators": 50,
5    "max_depth": 3,
6    "random_state": 42
7}
8
9# experiment 3
10task = Task.init(project_name="Iris Classification", task_name="100 estimators")
11params = {
12    "n_estimators": 100,
13    "max_depth": 3,
14    "random_state": 42
15}

These will do the following:

• Use 10 n_estimators to train the Random Forest Classifier model, output the hyperparameters, and accuracy metrics, and save the experiment as 10 estimators for experiment one - ex1.py.
• Use 50 n_estimators, output the hyperparameters, and accuracy metrics, and save the experiment as 50 estimators for experiment two - ex2.py.
• Use 100 n_estimators, output the hyperparameters, and accuracy metrics, and save the experiment as 100 estimators for experiment three - ex3.py.

Run them one after the other:

1python ex1.py
2python ex2.py
3python ex3.py

From here, the ClearML server will:

• Create a project on your dashboard called Iris Classification.
• Register three separate experiments on the project.
• Capture the hyperparameters, accuracy metrics, execution environment, and the Python package versions for each run.
• Log the train and test accuracy for each experiment.

Note that the ClearML SDK will attempt to monitor your hardware resources, for example, your GPU statistics. Since you are running on your local machine, which uses a CPU, the SDK will turn off GPU monitoring.

For the most part, GPU usage in experimentation in machine learning is relevant data. Its utilization percentage, as well as memory used, are factors that affect performance in machine learning, which the ClearML SDK tries to capture.

Therefore, if you see the following - 2026-05-30 12:46:56,775 - clearml.Task - INFO - Finished repository detection and package analysis ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring at each output after running your scripts, do not fret. GPU monitoring is optional.

View experiments on ClearML server

Since you are connected to your ClearML server, your project and experiments will be visible on your dashboard.

Head to your server and click on “View All” to view your Iris Classification project:

Viewing projects on the ClearML server projects dashboard

Select your project:

Viewing the project - Iris Classification

You should see all experiments listed out for you:

Viewing experiments under the Iris Classification project

You can monitor trends for your train and test performance by clicking on the graph icon:

Viewing performance metrics for test and training data

Additionally, it’s possible to compare further across different values like hyperparameters, scalars, plots, and debug samples across experiments:

Clicking on the compare option to compare across different values for experiments.

Comparing across different values for experiments

From here, you can view artifacts, executions, and other relevant information for your experiments, which helps to find inconsistencies that affect experiment output and performance, if any.

Now you can make the following conclusions:

• All three experiments achieved the same test accuracy of 1.0.
• 10 estimators slightly outperformed 50 and 100, scoring 0.9666 against 0.9583.

This is the visibility the ClearML server gives you. All in one place. Without having you to write down or manually record these values and outputs for each run. Comparison is one click away.

Troubleshooting

Below are some common issues along with their potential solutions to help you navigate and resolve any hurdles that you may face in this tutorial:

1. Web server showing 404 in the browser: After you install ClearML server via Helm, check that all its pods are in a (1/1) Running state before you attempt to visit the web server from a browser. The following command will output the states of the pods - kubectl get pods -n default | grep clearml.
2. ClearML pods in Error, OOMKilled, Pending or CrashLoopBackOff state: View the logs of each pod and describe each pod using kubectl logs <pod name> and kubectl describe pod <podname> respectively. These commands will help you identify the root cause of the failures. Also, ensure that your kubernetes cluster has sufficient CPU, memory, and storage resources allocated for your ClearML server components. For example, the file server, MongoDB, Elasticsearch, and so on.
3. Test the API server connectivity: You can test whether the API server is reachable with the following command: curl -u "ACCESS_KEY:SECRET_ACCESS_KEY" -X GET "http://api.<external IP>/auth.login". A successful request will output a 200 response code with a JSON payload containing a bearer token. 
4. Cannot locate your access key and secret access key: You can obtain your ACCESS_KEY and SECRET_ACCESS_KEY from http://app.<external_ip>.nip.io/settings/workspace-configuration.

Next steps

How about you make your experiments reproducible with the ClearML agent? The ClearML agent lays emphasis on “reproducibility”. This means, while you can capture everything about an experiment when it runs (which the SDK solves), you should be able to rerun it with the same environment anytime to see if you will get the same output or result.

You can deploy it on Kubernetes as a Helm Chart or via the clearml-k8s-glue script, or even locally on your machine from your project via pip.

Summary

ClearML offers many features. In this tutorial, you have learned how to use it for experiment tracking, from deploying the ClearML server component on a Civo Kubernetes cluster via Helm, to initializing the ClearML SDK in your project, and connecting to your ClearML server. You have created a project and run three experiments under it to see how ClearML captures and organizes experiment data, and then compared their outputs.

This article uses two nodes to demonstrate how ClearML can be used for experiment tracking and adjusts the storage requirements for Elasticsearch, MongoDB, and the file server. If you’ll be working in a production environment, make sure to adjust the storage requirements to suit your use case and use at least three nodes when provisioning your Kubernetes cluster.