Kubernetes meets Llama 3.2: How to deploy AI models on GPU clusters

Large Language Models (LLMs) are revolutionizing the way we interact with artificial intelligence, enabling applications such as language translation, text summarization, and chatbots that are more natural and human-like. However, deploying LLMs requires significant computational resources and specialized infrastructure, which can be a barrier to adoption.

Simplifying LLM deployment with Civo’s LLM Boilerplate

Setting up a GPU-enabled Kubernetes cluster to run LLMs can be complex and time-consuming, especially for those who require seamless integration, data security, and regulatory compliance. To address this challenge, we've created a step-by-step guide to deploying a Kubernetes GPU cluster on Civo using the Civo LLM Boilerplate.

What you'll learn

In this tutorial, you'll learn how to automate the setup of a Kubernetes GPU cluster on Civo Cloud using Terraform and GithubActions, and deploy essential tools such as:

• Ollama LLM Inference Server
• Ollama Web UI
• NVIDIA Device Plugin
• An example Python LLM application

Project goal

The goal of this project is to enable customers to easily use Open Source LLMs, providing 1:1 compatibility with OpenAI's ChatGPT. Plus:

• Access to the latest Open Source LLMs made available from Ollama.
• Provide a user interface to allow non-technical users access to models.
• Provide a path to produce insights with LLMs while maintaining sovereignty over the data.
• Enable LLMs in regulatory use cases where ChatGPT can't be used.

Prerequisites

Before beginning, ensure you have the following:

• A Civo Cloud account
• A Civo Cloud API Key
• Terraform installed on your local machine

Deploying Llama 3.2 on Civo using Terraform

Project setup

1. Obtain your Civo API key from the Civo Dashboard.
2. Create a file named terraform.tfvars in the project's root directory.
3. Insert your Civo API key into this file as follows:

civo_token = "YOUR_API_KEY"

Project configuration

Project configurations are managed within the tf/variables.tf file. This file contains definitions and default values for the Terraform variables used in the project.

Deployment configuration

Deployment of components is controlled through boolean variables within the tf/variables.tf file. Set these variables to true to enable the deployment of the corresponding component.

Deploy LLM Boilerplate

To deploy, simply run the following commands:

Step 1: Initialize Terraform:

1terraform init

This command initializes Terraform, installs the required providers, and prepares the environment for deployment.

Step 2: Plan deployment:

1terraform plan

This command displays the deployment plan, showing what resources will be created or modified.

Step 3: Apply deployment:

1terraform apply

This command applies the deployment plan. Terraform will prompt for confirmation before proceeding with the creation of resources.

Building and deploying the example application

Step 1: Build the custom application container:

Enter the application folder:

1cd app

Build the Docker image:

1docker build -t {repo}/{image} .

Push the Docker image to a registry:

1docker push -t {repo}/{image}

Navigate to the Helm chart:

1cd ../infra/helm/app

Modify the Helm Values to point to your Docker registry, e.g.

1replicaCount: 1
2image:
3    repository: {repo}/{image}
4    pullPolicy: Always
5    tag: "latest"
6
7service:
8    type: ClusterIP
9    port: 80

Step 2: Initialize Terraform:

Navigate to the Terraform directory:

1cd ../tf

1terraform init

This command initializes Terraform, installs the required providers, and prepares the environment for deployment.

Step 3: Plan deployment:

1terraform plan

This command displays the deployment plan, showing what resources will be created or modified.

Step 4: Apply deployment:

1terraform apply

This command applies the deployment plan. Terraform will prompt for confirmation before proceeding with the creation of resources.

Deployment takes around 10 minutes to stand up the Civo Kubernetes Cluster, assign a GPU node, deploy the Helm charts and GPU configuration before downloading the models and running them on your Nvidia GPU.

Troubleshooting

If you experience any issues during the deployment (for example, if you experience a timeout), you can reattempt the deployment by rerunning:

1terraform apply

Deploy Llama 3.2 through GitHub actions

For those who prefer a fully automated cloud-based approach, GitHub Actions offers a powerful solution. As a part of GitHub's CI/CD platform, Actions allows you to automate your software workflows, including deployments. This method simplifies the deployment process, ensuring that it is repeatable and error-free, which is particularly beneficial for managing and updating large-scale machine learning models like Llama 3.2 without manual intervention.

First, navigate to the repository: https://github.com/civo-learn/civo-llm-boilerplate, and then use the template to create a new repository.

After doing so, go to the settings of your newly created repository and make sure GitHub Actions are allowed to run.

Make a new secret through the settings for the repository called CIVO_TOKEN and set it to your Civo account token.

Now, you can head to the actions tab and run the deployment.

Accessing and managing your deployment

Once you have successfully deployed Llama 3.2 using either Terraform or GitHub Actions, the next step is to verify and utilize the deployment:

Checking the Load Balancers

After deployment, you can check the load balancers attached to your Kubernetes cluster to locate the Open Web UI endpoint. Navigate to the load balancer section in your Civo Dashboard and find the DNS name labeled “ollama-ui-open-webui.”

Completing the initial open-web-ui setup, which involves registering an initial administrator account and configuring the deployment options, will grant you access to a “ChatGPT-like” interface, where you can interact with the deployed LLM directly.

From this window, you can further configure your environment, such as setting your security and access preferences and what access newly registered users can access. In addition, you can make additional users administrators in addition to the first registered account.

Deploying additional models

If you wish to expand your LLM capabilities, simply navigate to the settings menu found in the top right-hand corner of the Open Web UI screen. Select “models” from the left-hand menu to add or manage additional models. This feature allows for versatile deployment configurations and model management, ensuring that your setup can adapt to various requirements and tasks.

If you would like to change the default models deployed or disable GPU support, simply modify the ollama-values.yaml file in the infra/tf/values folder.

1ollama:
2 gpu:
3   # -- Enable GPU integration
4   enabled: true
5   # -- Specify the number of GPU to 1
6   number: 1
7 # -- List of models to pull at container startup
8 models:
9   - llama3.2-vision
10   - gemma
11   # - llava
12   # - mixtral
13   # Get more models from: https://ollama.com/library
14persistentVolume:
15 enabled: true
16 size: 250Gi # file size of model repository

Note: To access Llama 3.2 for non-vision features update the above code llama3.2-vision to code>llama3.2

Summary

Congratulations! You have successfully deployed a Kubernetes GPU cluster on Civo Cloud using Terraform and set up various components for running LLMs, including the Ollama inference server and web interface.

With this boilerplate, you now have a scalable and flexible infrastructure for leveraging Open Source LLMs, allowing you to customize deployments, integrate additional tools, or expand your cluster as needed.

If you want to learn more about LLMs, check out some of these resources:

• Deploy your own Private ChatGPT with Llama 3.1 and Civo GPU Clusters
• Deploying Llama 3.1 with Kubeflow on Kubernetes
• Building a RAG system with Gemini for financial forecasting on Civo Kubernetes
• Scaling AI workloads with Kubernetes