A single agentic task can consume 60 million tokens. At frontier model pricing, that is hundreds of dollars per run — for one security audit, one document analysis, one multi-step workflow. Scale that across a team running hundreds of tasks per day and the bill becomes untenable.

The instinct is to cap usage, switch to cheaper models, or restrict who can call the API. None of that works. Caps drive usage underground. Cheaper models degrade quality. Restrictions create Shadow AI.

The actual solution is architectural: make repeated computation free. That is what LLM caching does — and the difference between a naive implementation and an engineered one is the difference between a 5% cache hit rate and 84%.

The Three Caching Layers

LLM caching is not a single technique. It operates at three distinct layers, each matching different things and saving money through different mechanics.

Layer What’s Matched Where It Lives Savings on Hit Realistic Hit Rate
Prompt/Prefix Caching Exact token prefix Provider-side (OpenAI, Anthropic, Google) 90% off input tokens 60–87% after optimization
Semantic Caching Embedding similarity of the query Your infra (vector store + embeddings) 100% — LLM never called 20–45% in production
KV-Cache Optimization Attention states across requests Your serving stack (vLLM, SGLang) Higher throughput, lower latency Workload-dependent

These layers are complementary. Prompt caching makes repeated calls cheaper. Semantic caching eliminates repeated calls entirely. KV-cache optimization improves serving efficiency for self-hosted models. Deploy them in that order — each layer stacks on the previous one.

Layer 1: Prompt Prefix Caching

This is the highest-leverage layer and the one where most teams leave the most money on the table.

How It Works

When an LLM processes a prompt, it computes attention states (key-value matrices) for every token. These states capture the relationships between all tokens in the context. Prompt caching stores these computed states for a prefix of your prompt. When the next request starts with the exact same token sequence, the provider reuses the stored states instead of recomputing them.

The pricing reflects the savings:

Provider Cache Write Cache Read (Hit) Discount
Anthropic (5-min TTL) 1.25× base input 0.1× base input 90% off
Anthropic (1-hour TTL) 2× base input 0.1× base input 90% off
OpenAI (GPT-5.6+) 1.25× base input ~0.1–0.5× base input Up to 90% off
Google Gemini 2.5+ Implicit caching Discounted Automatic

Cache discounts stack with batch API discounts. Claude Sonnet 4.6 at $3/MTok input, combined with batch (0.5×) and cache read (0.1×), drops to $0.15/MTok — 95% off list.

The Prefix Sensitivity Rule

This is the critical constraint: prompt caching is strictly prefix-based. The provider matches from the first token forward. One different token at position N invalidates every token from N+1 onward.

If the first 100 tokens of your prompt change on every request — a timestamp, a session UUID, a randomized key order — 100% of the subsequent 10,000 tokens are invalidated. The cache is useless.

This is why most teams start at a 5% hit rate. They are not making a caching mistake. They are making a prompt architecture mistake.

The ProjectDiscovery Case Study: 7% to 84%

The most detailed public case study on prompt caching optimization comes from ProjectDiscovery’s Neo — an autonomous security testing platform that runs multi-agent, multi-step workflows averaging 26 steps and 40 tool calls per task. System prompts are 2,500+ lines of YAML, over 20K tokens per agent. A single complex task can consume 60 million input tokens.

Their journey from 7% to 84% cache hit rate is a masterclass in prompt architecture.

The Starting Point: Why 7%

Neo’s original prompt structure placed dynamic content — working memory, skills context, runtime context — between the static system prompt and the tool definitions. Working memory changes on nearly every step. This was silently killing every cache hit after the system prompt.

The Breakthrough: The Relocation Trick

The single most impactful optimization: move all dynamic content out of the cacheable prefix. ProjectDiscovery built a relocation function that finds dynamic system messages (working memory, skills context, runtime context), removes them from the middle of the prompt, and appends them as a <system-reminder> user message at the tail.

This single deployment took the cache hit rate from 7% to 74% overnight.

The Architecture: Three Breakpoints

They use three of Anthropic’s four available cache breakpoints:

Breakpoint 1 — Static System Prompt (1-hour TTL) Marks the last static system message. Walk backwards to skip any message containing dynamic headers. The 1-hour TTL (not the default 5 minutes) keeps the cache warm across users and tasks during business hours.

Breakpoint 3 — Tool Definitions (1-hour TTL) Tools are sorted alphabetically with static tools first, dynamic per-user subagents last. This creates a shared cache chain — [system prompt → BP1] [static tools → BP3] — that is identical across all users.

Breakpoint 2 — Conversation Sliding Window (5-minute TTL) Marks the last tool result in the conversation. Each new step only pays for messages added since the last breakpoint. The 5-minute TTL matches the rapid change rate of conversation state.

The Smaller Optimizations (74% to 84%)

Four additional optimizations closed the remaining gap:

  1. Stable template variables. System prompt YAML templates render with placeholders ([provided in Runtime Context]) instead of actual values. The real values arrive via the relocated tail message. This keeps the system prompt byte-identical across all users, all threads, all days.

  2. Frozen datetime. The datetime is frozen once per task run and formatted as date-only — no clock time. “Thursday, April 3, 2026” stays stable for an entire day. Including the current time would change every second.

  3. Provider routing for cache locality. Anthropic caches are provider-specific. A request to Anthropic Direct and a follow-up to Amazon Bedrock do not share caches even with identical prompts. ProjectDiscovery routes all traffic to Anthropic Direct first, falling back to Bedrock and Vertex only during outages.

  4. Tool message part-level marking. When an agent makes parallel tool calls, the SDK fans each tool response into a separate wire message. Marking the entire tool message would exhaust breakpoint slots. They mark only the last content part of the last tool message — one breakpoint consumed instead of N.

The Results

Week Starting Cache Rate Notes
Feb 2 4.2% Before optimization
Feb 9 7.6%
Feb 16 73.7% Relocation trick deployed
Feb 23 78.2%
Mar 9 80.2%
Mar 16 84.3% All optimizations in place
Mar 30 82.9% Steady state

The most revealing data point: cache rate scales with task complexity.

Steps Avg Cache Rate Avg Input Tokens
1 35.5% 47,518
2–3 30.0% 161,442
4–5 42.8% 253,880
6–10 53.6% 379,818
11–20 63.9% 745,685
20+ 74.0% 3,763,263

The single-step dip to 35% makes sense — no conversation history to cache yet, so only BP1 contributes. By step 20+, the sliding window is doing the heavy lifting across 3.7 million-token inputs. The most expensive tasks are the most cacheable. That is exactly the cost curve you want.

One extreme example: task c790f4 ran 67.5 million input tokens across 1,225 steps at 91.8% cache rate. A comparable pre-optimization task ran the same token volume at 3.2% cache rate — roughly 60 times the cost.

Layer 2: Semantic Caching

Prompt caching catches identical prefixes. Semantic caching catches semantically similar queries — “What is Kubernetes?” and “Explain Kubernetes to me” produce different tokens but the same answer.

How It Works

  1. Embed the incoming query using a lightweight model (384-dimension embedding, ~2–5ms)
  2. Search a vector store for previously answered queries above a cosine similarity threshold
  3. On hit, return the cached response — the LLM is never called

Real-World Hit Rates

Production semantic cache hit rates run 20–45%, not the 90–95% in vendor marketing. The 95% figure traces to match accuracy, not hit rate.

Source Traffic Type Hit Rate
Portkey gateway data Q&A / RAG ~20% at 99% accuracy
Walmart Tail search queries ~50% (expected 10–20%)
General open-ended chat Mixed 10–20%

Threshold Tuning

The similarity threshold is the core engineering problem:

  • 0.85: Aggressive. Higher hit rate but false-positive risk — may serve wrong cached answers
  • 0.92–0.95: Common sweet spot. Good balance of hits and accuracy
  • 0.98: Behaves like exact match. Minimal false positives but low hit rate

Start at 0.95 and tune down based on accuracy validation.

When to Use Semantic Caching

Good fit: High-volume, repetitive query traffic — product search, support FAQ, documentation Q&A, knowledge base lookups. Many distinct users asking semantically similar questions.

Bad fit: Agent workflows and internal pipelines where queries rarely repeat semantically. Creative generation where the same input should produce different outputs. Real-time data queries where cached responses go stale.

Layer 3: KV-Cache Optimization (Self-Hosted Only)

If you run your own inference with vLLM, SGLang, or TensorRT-LLM, the serving stack manages KV-cache blocks directly. Techniques include:

  • Paged attention — share KV blocks across requests with common prefixes
  • KV-cache quantization — compress cached states to fit more concurrent requests per GPU
  • Prefix-aware scheduling — route requests to the GPU that already holds the relevant cache
  • Session affinity — load balancers must pin conversations to the same worker, or the cache is lost every turn

If you call hosted APIs (OpenAI, Anthropic, Google), you cannot touch this layer directly. Prompt caching is your interface to it.

The Prompt Architecture Playbook

Getting from 5% to 60%+ cache hits is a prompt engineering problem, not an infrastructure problem. The rules:

Rule 1: Static-to-Dynamic Waterfall

Order every prompt from most permanent to most volatile:

┌─────────────────────────────────────┐
│  System instructions and role       │  ← Most stable. Cache gold.
├─────────────────────────────────────┤
│  Tool definitions (sorted, stable)  │  ← Stable within feature set.
├─────────────────────────────────────┤
│  Long-lived docs / few-shot examples│  ← Semi-stable.
├───── cache_control breakpoint ──────┤
│  Retrieved context / RAG            │  ← Changes per query.
├─────────────────────────────────────┤
│  User message / dynamic variables   │  ← Always unique.
└─────────────────────────────────────┘

Rule 2: Eliminate Non-Determinism from the Prefix

Before anything touches the API, verify:

  • No timestamps in the system prompt (freeze per-task, date-only format)
  • No request IDs, session UUIDs, or random nonces in the prefix
  • Tool definitions serialized with sorted keys, alphabetical order
  • Retrieved documents sorted by stable identifiers (file path, line number), not by similarity score
  • Template variables rendered with stable placeholders, not actual values

Rule 3: Use Explicit Cache Markers

On Anthropic, mark stable blocks with cache_control: {"type": "ephemeral"}. Use the 1-hour TTL for shared static content. Use the 5-minute TTL for session-specific conversation state. Budget your four breakpoints carefully.

On OpenAI, caching is automatic for prompts ≥1,024 tokens. Use prompt_cache_key for deterministic routing on GPT-5.6+ models. Keep traffic per key to ~15 requests per minute.

Rule 4: Keep Context Lean

A 5K-token prompt is far more likely to hit a cache than a 50K-token prompt. Every additional token that varies between requests reduces the cached prefix ratio.

  • Start fresh sessions when switching tasks
  • Scope file context to relevant sections, not entire documents
  • Disconnect unused tools from the prompt
  • Don’t compact mid-session — compaction creates a new prefix that matches nothing

Rule 5: Don’t Cache Writes You Won’t Read

Cache writes cost more than standard input tokens (1.25× on both Anthropic and OpenAI). If a prefix is only used once, caching adds cost. Enable caching when you expect repeated use — shared system prompts, multi-turn conversations, common tool schemas. Disable it for one-off requests.

Measuring What Matters

Most teams measure latency. Cache-optimized teams measure these:

Metric Target Why It Matters
Cache hit rate by template 80%+ Below 80% is a regression
Wasted write cost Minimize Cache writes with 0 reads within TTL
Prefix stability 95%+ % of requests where first 1K tokens are byte-identical
Effective $/1M tokens Trending down Blended rate across cache reads, writes, and uncached
Cost per successful task Trending down Not cost per token — cost per unit of work

Set alerts when cache hit rate drops below 80%. A drop almost always means a code change introduced non-determinism into the prefix — a new timestamp, a reordered field, a template variable rendering with live values.

The Full Stack

The most cost-effective architecture layers all three caching mechanisms:

User Request
      │
      ▼
┌─────────────────┐     ┌──────────────────┐
│ L1: Exact Match │────▶│  Cache hit?      │──▶ Return (0 LLM cost)
│ (hash of prompt)│     │ Redis / in-memory│
└─────────────────┘     └────────┬─────────┘
                                 │ miss
                                 ▼
                    ┌────────────────────────┐
                    │  L2: Semantic Cache    │──▶ Return (0 LLM cost)
                    │ (embedding similarity) │
                    └────────────┬───────────┘
                                 │ miss
                                 ▼
                    ┌────────────────────────┐
                    │  L3: Provider Prefix   │──▶ 90% discount on
                    │  Caching (Anthropic/   │    cached prefix
                    │  OpenAI automatic)     │
                    └────────────────────────┘

Each layer catches what the previous one missed. L1 is free (zero model calls). L2 is nearly free (embedding cost is negligible). L3 is 90% cheaper on the cached portion. Only true misses pay full price.

The Numbers That Matter

Across production deployments, the pattern is consistent:

  • ProjectDiscovery: 7% → 84% cache hit rate, 59–70% cost reduction, 9.8 billion tokens served from cache
  • Notion: 90% cost reduction, 85% latency reduction via Anthropic prompt caching
  • OpenAI coding customer: 60% → 87% hit rate from a single prompt_cache_key routing parameter
  • Academic benchmarks: 45–80% input cost reduction across agentic evaluations

The common thread: the improvement came from prompt architecture, not infrastructure investment. The tools are free. The engineering is in how you structure what you send.

The Bottom Line

Cache misses are not random. They are a failure of prompt design.

Every timestamp in a system prompt, every randomized tool order, every dynamically rendered template variable in the cacheable prefix — those are architectural choices that destroy cache hit rates. Fix the prompt structure and the cache hit rate follows.

The deployment order is clear: enable provider prompt caching first (automatic on OpenAI and Gemini, one parameter on Anthropic), restructure prompts to be prefix-stable, measure at the per-template level, add semantic caching for repetitive traffic, and optimize the serving stack if you self-host.

The economics are not marginal. At 84% cache hit rate with a 90% cached-input discount, the effective input cost is roughly 15% of list price. Stack that with batch processing and you approach 95% off. That is the difference between an AI budget that scales linearly with usage and one that stays flat while usage grows exponentially.

The infrastructure exists. The pricing is documented. The case studies are public. The only variable is whether your prompts are architected to take advantage of it.

Deep research notes: [[LLM-Caching-Optimization-Deep-Dive-2026]]