LLM Token & GPU FinOps FAQ
How much does prompt caching reduce LLM API costs in production?
Prompt caching discounts repeated system prompts, RAG context, and tool schemas by 50% to 90% on input tokens depending on provider (50% on OpenAI, up to 90% on Anthropic Claude and DeepSeek cache hits). For agent workflows with 4,000+ input tokens and 70% cache hit rates, total monthly API spend typically drops 35% to 65%.
When does self-hosting vLLM on dedicated GPUs become cheaper than frontier APIs?
A single 80GB H100 GPU node costs roughly $1,800 to $2,400 per month on cloud GPU providers (or $3,600+ for HA multi-replica redundancy). Self-hosting generally breaks even against frontier flagship APIs above 150M to 300M tokens per month, while lightweight API models like Gemini Flash, GPT-4o-mini, or DeepSeek remain cheaper until billions of monthly tokens.
Why do output tokens dominate LLM API bills?
Most LLM providers price output tokens 3x to 5x higher than input tokens because autoregressive decoding is memory-bandwidth bound rather than compute bound. Limiting verbose reasoning, using structured JSON schemas, and routing simple classification tasks to flash-tier models usually produces the fastest bill reduction.
Pricing Sources & Live Validation Feed
- Live API Price Validation: Commercial token rates auto-sync in your browser from the public OpenRouter Live Models Pricing API on every page load (falling back cleanly to verified Nov 2026 baseline list rates if unreachable).
- Official Vendor Pricing Pages: OpenAI API Pricing, Anthropic Claude Pricing, Google Gemini API Pricing, DeepSeek API Pricing, and RunPod Cloud GPU Pricing.
- Self-Hosted vLLM Formula: Assumes 2.5x peak diurnal burst factor over average monthly output TPS, sized against 1,400 output tokens/sec sustained throughput per 80GB H100 GPU at 730 hours/month (24/7).