LLM GPU VRAM Sizing FAQ
How much VRAM is needed to run a 70B LLM in Q4 vs FP8 vs FP16?
A 70B parameter model needs roughly 130 GB for FP16 weights (2x 80GB A100/H100 GPUs), about 68 GB for FP8 weights (1x 80GB H100 with short context or 2x for long concurrent context), and roughly 39 GB for 4-bit Q4_K_M/AWQ weights (fits on a single 48GB L40S or 2x 24GB RTX 4090 GPUs plus KV-cache headroom).
Why does Grouped-Query Attention (GQA) drastically reduce KV-cache VRAM?
Modern architectures such as Llama 3, Qwen 2.5, and Mistral share Key and Value heads across groups of 8 Query heads (GQA 8:1). That reduces per-token KV-cache memory by 8x compared to legacy Multi-Head Attention (MHA), enabling 32k to 128k context windows and high concurrent batches without OOM kills.
Does FP8 KV-cache quantization work with vLLM?
Yes. Passing --kv-cache-dtype fp8 in vLLM on NVIDIA Ada (L40S / RTX 4090) and Hopper (H100 / H200) GPUs cuts KV-cache memory consumption in half with negligible perplexity degradation, allowing nearly 2x higher concurrent sequence throughput.
VRAM Formulas & Official Cloud GPU Pricing Sources (Verified Nov 2026)
- VRAM Formula: Model weights =
params * bytesPerParam / 1024^3(FP16: 2.0B, FP8/Q8: 1.05B, Q6_K: 0.82B, Q4/AWQ: 0.56B). KV-Cache =2 * layers * kvHeads * headDim * contextTokens * batch * kvBytes / 1024^3. Sizing targets 90% usable VRAM (--gpu-memory-utilization 0.90) with power-of-2 tensor parallelism. - Reference Cloud GPU Sources: RunPod GPU Cloud Pricing, Lambda Cloud Pricing, vLLM Official Documentation, and NVIDIA Data Center GPU Specs.