Lumeir TOOLS HUB

LLM GPU VRAM, KV-Cache & vLLM Sizing Calculator

Size exact GPU memory for open-weight LLMs (weights + GQA/MHA KV-cache + CUDA activation headroom), match against RTX 4090, L40S, A100, H100, and H200 hardware, and generate a ready-to-run vLLM command.

Model & Inference Specs vLLM / llama.cpp Math
B
Total parameters in billions (supports up to 1,000B via numeric box).
Weight Quantization
Includes real quantization scale/group metadata overhead (e.g., ~4.5 bits effective for Q4_K_M).
k tok
Maximum sequence length per active request (e.g., 16k = 16,384 tokens).
seqs
Simultaneous full-context sequences held in PagedAttention KV-cache.
Attention & KV-Cache Format
Total VRAM Needed
0 GB
Includes safety buffer
Model Weights
0 GB
Precision
KV-Cache Memory
0 GB
Per batch
CUDA & Activations
0 GB
vLLM / PagedAttn
Memory Footprint Composition
Model Weights KV-Cache (Active Context) CUDA Context + Activations
GPU Hardware Fit & 24/7 Cloud Hosting Estimate Power-of-2 Tensor Parallel Sizing
Generated vLLM Launch Command

LLM GPU VRAM Sizing FAQ

How much VRAM is needed to run a 70B LLM in Q4 vs FP8 vs FP16?

A 70B parameter model needs roughly 130 GB for FP16 weights (2x 80GB A100/H100 GPUs), about 68 GB for FP8 weights (1x 80GB H100 with short context or 2x for long concurrent context), and roughly 39 GB for 4-bit Q4_K_M/AWQ weights (fits on a single 48GB L40S or 2x 24GB RTX 4090 GPUs plus KV-cache headroom).

Why does Grouped-Query Attention (GQA) drastically reduce KV-cache VRAM?

Modern architectures such as Llama 3, Qwen 2.5, and Mistral share Key and Value heads across groups of 8 Query heads (GQA 8:1). That reduces per-token KV-cache memory by 8x compared to legacy Multi-Head Attention (MHA), enabling 32k to 128k context windows and high concurrent batches without OOM kills.

Does FP8 KV-cache quantization work with vLLM?

Yes. Passing --kv-cache-dtype fp8 in vLLM on NVIDIA Ada (L40S / RTX 4090) and Hopper (H100 / H200) GPUs cuts KV-cache memory consumption in half with negligible perplexity degradation, allowing nearly 2x higher concurrent sequence throughput.

VRAM Formulas & Official Cloud GPU Pricing Sources (Verified Nov 2026)
Pricing & Liability Disclaimer: VRAM figures and cloud GPU hourly/monthly costs are architectural estimates based on published reference pricing (last reviewed November 2026) and standard transformer layer ratios. Actual memory usage varies by model architecture (MoE, sliding-window attention, vision encoders), CUDA driver version, and serving engine. Cloud GPU spot/on-demand rates are subject to change at any time without notice. Always verify pricing directly with your cloud provider and benchmark on target hardware. Lumeir is not responsible or liable if your actual GPU charges or memory requirements are higher or lower than estimated here.
Copied