How to size GPU memory for Large Language Model workloads: A VRAM planning guide

VRAM is the single most consequential constraint in running large language models on cloud GPUs. Get the math right, and the workload fits, runs efficiently, and produces predictable cost. Get it wrong and the workload either fails to start, runs at a fraction of expected throughput, or forces an expensive late-stage migration to larger hardware.

The math itself isn't complicated. The reason teams get it wrong is that the components of VRAM consumption aren't always obvious until the workload actually runs. Model weights are the visible part. Optimizer state, activations, KV cache, and framework overhead are the parts that surprise people.

This is a working VRAM planning guide for LLM workloads. It covers what consumes GPU memory, how to estimate it accurately before provisioning, and how to match the workload to the right GPU configuration without over-spending on hardware the workload doesn't actually need.

What lives in VRAM during an LLM workload

Before sizing, the team needs a clear picture of what consumes GPU memory at each stage of an LLM workload.

For inference

The components that live in VRAM during inference:

• Model weights: The parameters themselves, at whatever precision the model is being served at
• KV cache: Cached key and value tensors for each token in the active conversation, growing with sequence length and concurrency
• Activations: Intermediate computations during the forward pass, typically small compared to weights and KV cache
• Framework overhead: PyTorch, vLLM, TensorRT, or whatever serving framework is in use takes its own memory for kernels, buffers, and runtime state

KV cache is the component most teams underestimate. For a model serving long-context conversations to many concurrent users, the KV cache can rival or exceed the model weights in memory consumption.

For training

Training is dramatically more memory-intensive than inference. The additional components:

• Optimizer state: Adam-style optimizers maintain first and second moments per parameter, doubling or tripling the memory footprint of the weights themselves
• Gradients: One gradient per parameter, at training precision
• Activations for backpropagation: Stored during the forward pass and reused during the backward pass; with long sequences and large batch sizes, this can dominate memory consumption
• Communication buffers: For distributed training, gradient synchronization requires staging buffers

A 7-billion-parameter model that fits comfortably in 24GB for inference might require well over 100GB for full training, depending on precision, batch size, and optimizer.

For fine-tuning

Fine-tuning sits between the two. Full fine-tuning has the same memory profile as training from scratch. Parameter-efficient methods like LoRA dramatically reduce the optimizer state and gradient components, often making the workload fit in inference-class memory budgets.

The precision dimension

Memory consumption is proportional to precision. The same model uses very different amounts of VRAM at different precisions:

• FP32 (32-bit floating point): Historical default, 4 bytes per parameter
• FP16/BF16 (16-bit floating point): Modern default for most ML workloads, 2 bytes per parameter
• FP8 (8-bit floating point): Supported on H100 and B200, 1 byte per parameter
• INT8 (8-bit integer): Common for inference quantization, 1 byte per parameter
• INT4 (4-bit integer): Aggressive inference quantization, 0.5 bytes per parameter

A 7-billion-parameter model takes roughly 28GB at FP32, 14GB at FP16/BF16, 7GB at FP8 or INT8, and 3.5GB at INT4 - just for the weights. The precision choice determines what GPU can run the workload at all, and what it costs.

For most production inference, FP16 or BF16 is the precision floor below which accuracy degradation starts to matter. INT8 quantization works for many models with minimal accuracy impact and is increasingly the default for inference at scale. INT4 quantization is more aggressive and requires per-model validation that the accuracy hit is acceptable.

For training, BF16 is the modern default, with FP8 emerging as a viable option on H100 and B200 for compatible workloads.

Estimating VRAM for inference

The practical formula for inference VRAM:

VRAM = Model weights + KV cache + activations + framework overhead

The components break down as:

• Model weights = parameters × bytes-per-parameter (precision dependent)
• KV cache = 2 × num_layers × num_attention_heads × head_dim × sequence_length × batch_size × bytes-per-parameter

A worked example for a 13-billion-parameter model serving at FP16, with 32-layer architecture, 32 attention heads of dimension 128, serving 8 concurrent conversations at 8,192 token context:

• Model weights: 13B × 2 bytes = 26GB
• KV cache: 2 × 32 × 32 × 128 × 8,192 × 8 × 2 bytes ≈ 34.4GB
• Activations and overhead: typically 2-4GB

Total: ~62-64GB. An A100 80GB fits this comfortably; an A100 40GB does not. An H100 80GB has additional headroom for longer contexts or more concurrent users.

The KV cache term is what catches teams out. Doubling the context length or the concurrent user count doubles the KV cache. Workloads that scale up users or extend context windows can run out of memory in ways the spec sheet didn't anticipate.

Estimating VRAM for training

The practical formula for training VRAM:

VRAM = Model weights + Optimizer state + Gradients + Activations + Communication buffers

For Adam-style optimizers in BF16 mixed precision:

• Model weights (BF16): parameters × 2 bytes
• Optimizer state (Adam): parameters × 12 bytes (FP32 momentum + FP32 variance + FP32 master copy of weights)
• Gradients (BF16): parameters × 2 bytes
• Activations: highly variable, can dominate for long sequences

A 7-billion-parameter model in BF16 mixed-precision training:

• Weights: 14GB
• Optimizer state: 84GB
• Gradients: 14GB
• Subtotal: 112GB

Plus activations, which depend heavily on batch size and sequence length. For meaningful batch sizes at typical sequence lengths, activations can add 20-50GB.

This is why 7B-parameter training is typically a multi-GPU workload even on 80GB cards, and why fine-tuning of larger models requires careful capacity planning.

Memory-saving techniques

Several techniques materially reduce the memory footprint of training and inference. The headline ones:

Matching VRAM to Civo's GPU range

Civo's Cloud GPU range covers the modern NVIDIA lineup. Matching workload memory requirements to the right hardware:

• L40s (48GB): Good for inference on models up to 13-20B at FP16, or larger models at INT8 or INT4 quantization, depending on context length and concurrency. Cost-efficient for inference and graphics workloads. From $1.29/hour.
• A100 40GB: Suitable for inference on 13B-class models, LoRA fine-tuning of 7-13B models, and training of smaller models. From $1.09/hour.
• A100 80GB: The workhorse for most fine-tuning workloads. Fits LoRA and parameter-efficient fine-tuning of 7B-class models comfortably; full fine-tuning of 7B-class models typically requires multiple A100 80GB GPUs. Strong for LoRA fine-tuning of 30-70B models and inference on 70B-class models at appropriate quantization. From $1.79/hour.
• H100 PCIe (80GB): Higher throughput than A100, similar memory budget. Strong for production inference. From $2.49/hour.
• H100 SXM (80GB): NVLink interconnect for multi-GPU training. From $2.99/hour.
• H200 SXM (141GB): Extended memory for larger models and longer contexts. Single-GPU inference on 70B-class models at INT8 becomes practical; FP16 serving of 70B-class models is possible only for short-context, low-concurrency workloads given the minimal headroom remaining after weights. From $3.49/hour.
• B200 Blackwell (192GB): Next-generation memory budget for the largest single-GPU workloads. Strong FP8 support for inference at scale.

The match between workload memory and GPU memory should leave headroom - typically 10-20% - for framework overhead, varying batch sizes, and operational variability. A workload that exactly fills the GPU's memory is fragile in production.

A planning workflow

For teams sizing GPU memory for a new LLM workload:

1. Estimate model weight memory based on parameter count and target precision
2. Estimate KV cache requirements based on context length, concurrent users, and model architecture
3. Add training overhead if applicable: optimizer state, gradients, activations
4. Apply memory-saving techniques appropriate to the workload (LoRA for fine-tuning, quantization for inference, mixed precision and gradient checkpointing for training)
5. Add 15-20% headroom for framework overhead and operational variability
6. Pick the smallest GPU that fits the resulting requirement, not the largest available

The last point matters for cost. An H100 running at 25% memory utilization on a workload that would fit comfortably on an A100 80GB is a meaningful overpay. The right-sized GPU produces better utilization and better cost per useful output.

What to do when the math doesn't fit

When the workload doesn't fit on a single GPU, the options:

• Move to a larger GPU: H200 (141GB) or B200 (192GB) extend single-GPU capacity
• Apply more aggressive memory-saving techniques: LoRA instead of full fine-tuning, INT4 quantization, gradient checkpointing with aggressive recomputation
• Move to multi-GPU: Tensor parallelism for inference, ZeRO/FSDP for training
• Consider model distillation: Train a smaller model that captures most of the larger model's capability

Multi-GPU is the most flexible answer but adds operational complexity and depends on having appropriate interconnect. Civo's Kubernetes GPU clusters support multi-GPU configurations with appropriate networking, and the latest hardware (H100 SXM, B200) includes NVLink for efficient gradient synchronization.

The practical takeaway

VRAM planning for LLM workloads breaks down into three things: knowing what consumes memory, estimating it accurately for the specific workload, and matching the result to the right GPU rather than the most powerful one available. The math is reliable once the components are clear; the cost savings from getting it right are substantial.

Civo's GPU range provides the flexibility to right-size at each point in the LLM lifecycle - small GPUs for experimentation, A100 80GB for the bulk of training and fine-tuning, H100 and H200 for production inference at scale, and B200 for the workloads that genuinely need it.

FAQs