Anthropic and OpenAI shipped prompt caching, and most teams are not using it correctly. The feature is real, the discount is large (up to ninety percent off cached input tokens at Anthropic, fifty percent at OpenAI), and the implementation cost is one or two lines of API change. The reason teams leave the savings on the table is not that the feature is hard to enable but that the cache is sensitive to a discipline most prompt-engineering pipelines do not enforce: stability of the prompt prefix.

Without caching, LLM costs scale linearly with traffic. A unit-economics-positive product at a hundred users becomes negative at ten thousand, because every request pays the full input bill regardless of how much of the prompt is shared with the previous request. With caching applied correctly, the static portion of the prompt (system instructions, tool definitions, few-shot examples, reference documents) is charged at the discounted rate after the first request, and only the variable tail (the user's actual query) pays the full price. The difference is a step change in the cost curve, not a percent-point optimisation.

Issue 001 covered the latency stages inside an LLM call; prompt caching reduces both the latency and the cost of those stages by serving the cached prefix from a fast path. This issue walks through how prompt caching actually works at Anthropic, OpenAI, and Google, the prefix discipline that decides whether a request hits or misses the cache, the rules for what to cache and what not to, and the cost math that makes the savings concrete.

How prompt caching actually works

Prompt caching is a provider-side optimisation. The model provider stores the model's intermediate state after processing a prefix of your prompt (specifically the KV cache, the tensor of attention key-value pairs the transformer needs to continue generating) and reuses that state on subsequent requests whose prompts begin with the same prefix. The user-visible benefit is that the cached prefix is charged at a steeply discounted input-token rate, and the request is also faster because the provider skips the compute that produced the cached state.

Three properties of the mechanism govern how a team should think about it.

The first property is that the cache is keyed by the literal token prefix. Two requests hit the same cache entry only if their tokens are byte-identical from the start of the prompt up to the cached length. A single space added to the system prompt, a different timestamp embedded in a few-shot example, or a tool definition reordered by a different version of a tool list will produce a miss. The cache is content-addressed, and the content has to match exactly.

The second property is that cache writes cost more than uncached input. Anthropic charges 1.25x the base input price for a five-minute cache write and 2x for a one-hour cache write. OpenAI's automatic caching does not charge a write premium but only engages above a 1024-token threshold. The implication is that a single-use prompt that is cached and never read again costs more than it would have without caching. Caching is only economical at request volumes high enough that the write cost is amortised across many subsequent reads.

The third property is that cache lifetime is short. Anthropic's default ephemeral cache lives for roughly five minutes from the last write; the optional one-hour cache costs more to write but lives correspondingly longer. OpenAI's automatic cache lives for roughly five to ten minutes under normal load, with the system retaining cached prefixes for up to an hour during off-peak periods; the TTL is not user-configurable in either case. Any pattern that depends on caching has to either keep the prefix hot with regular traffic or accept that idle periods reset the savings entirely.

How the three major providers implement it

The three major LLM providers have shipped prompt caching with different APIs and different default behaviour. Understanding the differences matters because the right code shape depends on which provider the application targets.

Anthropic prompt caching. Anthropic's caching is explicit and per-content-block. The developer marks any content block (the system prompt, a tool definition, a user message) with a cache_control field, and the API caches the prefix up to and including that block. The default cache TTL is five minutes; setting {"type": "ephemeral", "ttl": "1h"} extends the cache lifetime to one hour at a higher write cost. The minimum cacheable prefix is 1024 tokens for Sonnet models and 2048 tokens for Haiku, below which the cache is not engaged at all. Cache reads are charged at roughly one-tenth the base input price; cache writes at 1.25x or 2x depending on the TTL.

OpenAI prompt caching. OpenAI's caching is automatic for gpt-4o, gpt-4o-mini, and o1-family models, with no API change required from the developer. Any prompt above the 1024-token threshold is eligible; the system caches the longest matching prefix and serves cache reads at fifty percent off the base input price. The cache lifetime is roughly five to ten minutes under normal load and can extend to about an hour during off-peak periods, in either case not user-configurable. Because the caching is implicit, there is no per-block control: stability of the prefix is the only lever available to the application.

Google Gemini context caching. Gemini's caching is explicit and uses a different abstraction: the developer creates a named cache object with a defined TTL (minutes to hours), and references the cache by name in subsequent requests. The pricing model is also different: a per-token storage charge per hour, plus a discounted per-token read charge. The model fits well for long-lived shared context (a large knowledge base referenced by many requests over hours) and less well for short-burst shared context across a single chat session.

The diagram above shows the common path through prompt caching. On a hit, the provider skips the compute for the matching prefix and charges a discounted read rate for those tokens. On a miss, the provider does the full prefix compute and, if the prefix is above the length threshold, writes a new cache entry at a small premium. The only application-side decisions are what content to place inside the cacheable prefix and how to keep that prefix stable across requests.

What to cache, and what not to

Three categories of content reward caching, and one category does not.

The first category is the system prompt. A well-designed system prompt for a production assistant or chat feature runs from a few hundred to several thousand tokens: persona, role, behavioural rules, formatting guidance, refusal patterns, citation requirements. The system prompt rarely changes during a session and is repeated on every turn, which makes it the highest-value cache target in almost every application.

The second category is tool definitions and function schemas. An assistant with five to twenty tools, each with a JSON schema for arguments and a short description, is comfortably a few thousand tokens of stable content. The tools rarely change between requests but are repeated on every request. Cache the tool block either at the end of the system prompt or as its own cached segment.

The third category is large shared context. RAG-retrieved chunks that are identical across requests (the same product manual referenced by every customer in a support tool, the same legal disclaimer prepended to every contract analysis), few-shot examples used across many users, and large documents the user uploads once and refers to many times all qualify. The threshold is "the same content appears in many requests close together in time".

The category that does not reward caching is the variable tail. The user's specific query, per-request metadata, dynamic timestamps, conversational history that diverges per session: anything that changes per request must come after the cached blocks in the prompt order. A single variable token inside the cacheable region invalidates everything that follows it.

The practical pattern is to construct the prompt in a stable order from broadest to most specific: cached system prompt, cached tool definitions, cached shared context, then the variable conversation history and the user's current query. The cached blocks live at the start; the variable blocks live at the tail. Reorder the prompt to fit this pattern even when it feels awkward, because the discount only applies to the part that does not change.

The cost math, with a concrete example

The case for prompt caching is most persuasive with the numbers laid out. Consider a customer-support chat product running on Claude Sonnet (approximate late-2025 pricing of three dollars per million input tokens). The system prompt is 5,000 tokens, tool definitions are 3,000 tokens, and a typical user message plus short conversation history adds 2,000 tokens, for a 10,000-token request. Traffic is 10,000 requests per hour during business hours.

Without caching, the input bill is 10,000 requests times 10,000 tokens times three dollars per million, or 300 dollars per hour, or roughly 7,200 dollars per day at twenty-four hours of equivalent traffic.

With Anthropic ephemeral caching on the system prompt and tool definitions (8,000 tokens cacheable), the first request of each five-minute window pays a write cost of 8,000 tokens at 3.75 dollars per million, or three cents, plus 2,000 tokens at three dollars per million, or 0.6 cents, for the variable tail. The next roughly 832 requests in that window pay only 8,000 tokens at 0.30 dollars per million, or 0.24 cents for the cache read, plus the same 0.6 cents for the variable tail. Each five-minute window therefore costs roughly seven dollars, and across the hour (twelve five-minute windows) the input bill comes to approximately 84 dollars, a reduction of about 72 percent on input cost.

The savings on output tokens are unaffected by caching, and output usually dominates the bill for short-prompt-long-response workloads. The pattern reverses for long-prompt-short-response workloads (RAG, agents with large tool sets, code assistants with large file context), where input dominates and prompt caching is the single largest cost lever available. The hook's "cut your bill in half" claim is shorthand for input-dominated workloads with stable prefixes; the actual savings range from near zero (highly dynamic per-request prompts) to over ninety percent on input (long stable system prompts, high traffic, the long-TTL cache).

Common mistakes

Four mistakes recur in production prompt-caching implementations.

The first is putting variable content inside the cached prefix. A timestamp embedded in the system prompt ("Today is 25 March 2026"), a user identifier added at the top for personalisation, a session ID for logging: each invalidates the cache on every request. Audit the system prompt template for any non-static substitution and move it after the cached blocks.

The second is reordering tool definitions between requests. If the application generates the tool list from a registry that returns items in a non-deterministic order, two consecutive requests produce different prefixes even when the tools are the same. Sort the tool list deterministically (alphabetically by tool name is the simplest correct choice) before serialising it into the prompt.

The third is not measuring the cache hit rate. Providers expose cache-read and cache-write token counts in the response metadata. Without tracking the ratio of read to write tokens over time, a team has no way to tell whether their caching is paying off or whether a recent template change broke the hit rate silently. Wire the metrics into the observability stack from Issue 004 alongside tokens_in and tokens_out.

The fourth is caching content that is not actually shared. A unique RAG chunk inserted per query, a session-specific summary generated per turn, an LLM-generated paraphrase of the system prompt that varies subtly across sessions: each pays the cache-write premium without ever paying it back through reads. If a content block is not expected to be reused within the cache TTL, do not mark it for caching.

Summary

Prompt caching is a provider-side optimisation that discounts cached input tokens by fifty to ninety percent, is available on all three major LLM providers, and is small to implement. The discipline that decides whether a team captures the savings is prefix stability: the cached content must be byte-identical across requests, must live at the start of the prompt, and must be large enough to clear the provider's minimum threshold. System prompts, tool definitions, and large shared context are the high-value cache targets; variable per-request content must come after the cached blocks. Measure cache hit rate as a first-class metric, and audit the prompt template for non-static substitutions on every change.

Production checklist

  • Audit every system prompt template for non-static substitutions (timestamps, user identifiers, session IDs, anything generated per request). Move all non-static content out of the cached prefix.

  • Order prompt content from most stable to most variable: cached system prompt, cached tool definitions, cached shared context, then conversation history and the user's current query.

  • Mark the appropriate content blocks for caching using the provider's API. Anthropic uses cache_control per block; OpenAI caches automatically above the 1024-token threshold; Gemini uses named cache objects.

  • Choose the cache TTL based on traffic shape. Use the default short TTL (five minutes) for steady high-traffic workloads; use the longer one-hour TTL for bursty traffic at the higher write cost.

  • Sort tool definitions deterministically (alphabetically by name is sufficient) before serialising into the prompt.

  • Track cache-read and cache-write token counts as separate metrics, sliced by tenant and prompt template. The ratio of reads to writes is the headline cache-hit indicator.

  • Set a regression alert on the cache hit rate per template, so a prompt edit that breaks the prefix is detected before the bill spikes.

  • Re-evaluate the cacheable threshold and discount percentages quarterly. Providers tune the minimum-prompt-length and per-token rates periodically.

Further reading