Prompt caching is a design property, not a billing detail

By turn thirty of a coding session, an agent is sending its provider a prompt north of a hundred thousand tokens — and roughly 95% of those bytes are identical to the bytes it sent last turn. Prompt caching is the provider-side feature that notices: instead of re-processing the unchanged part of your prompt at full input price, the provider serves it from a cache of its own internal state, and bills the cached portion at a fraction of the normal rate. On Anthropic, a cache read costs about a tenth of the full input price; OpenAI and Gemini discount cached tokens steeply too.

A tenth. On the bulk of every request an agent ever makes. That’s not an optimization you sprinkle on at the end — it’s the difference between an agent that’s economically viable to run all day and one that isn’t. And here’s the part that took me longest to internalize while building efferent: whether you actually get that discount is not something you configure. It’s a property your whole architecture either has or doesn’t. One function in the wrong place — a timestamp interpolated into the system prompt, a transcript “cleaned up” in place — and you silently pay full price forever, with no error message anywhere.

This post is the tour I wish I’d had: why agents specifically live or die by this, the one mental model that explains all three major providers, and then each provider in turn — slowest on Anthropic, because it’s the only one where doing nothing means caching nothing.

The agent shape of the problem#

A chat application sends a prompt once and is done with it. An agent doesn’t. The agent loop — the thing that makes it an agent — is send the transcript, get tool calls back, run them, append the results, send the transcript again:

let history = [systemPrompt, userMessage]

while (true) {
  const res = await model.generate(history) // sends EVERYTHING, every turn
  history = [...history, ...res.tail]       // grows by a small tail
  if (res.finishReason !== 'tool-calls') return res
}

Look at what the provider sees across one task. Turn one: system prompt, tool definitions, user message — say 10k tokens, all new. Turn two: all of that again, plus an assistant message and some tool results — 14k tokens, of which 10k are bytes the provider processed seconds ago. By turn twenty the request is 80k tokens and the genuinely new part is the last 2k. The output of each turn is small; the input is the entire accumulated past, re-shipped every single time.

efferent’s engine loop has exactly this shape: each turn maps the persisted message buffer to a prompt, calls the provider-agnostic LanguageModel, and appends the response parts as the new tail. Without caching, a forty-turn session bills the early turns’ tokens forty times over. With caching, you pay full price for each token roughly once, the discounted rate thereafter — the cost curve goes from quadratic-ish in conversation length to something close to linear.

So the stakes are clear. The question is what the cache actually keys on, because every design decision in the rest of this post falls out of the answer.

One mental model: the byte-stable prefix#

All three providers cache the same way: on a byte-stable prefix. The cache key is derived from the exact rendered bytes of your request, in order, from the very first byte — tool definitions, then system prompt, then messages. A request hits the cache for the longest previously-seen prefix and pays full price only for what comes after it.

The brutal corollary: one changed byte invalidates everything after it. Not “the changed part” — everything downstream of the change. The cache doesn’t diff; it matches prefixes.

turn 1:  [tools][system][user]                          ← all cold, cache written
turn 2:  [tools][system][user][asst][tools'][user]      ← prefix hit, tail cold
turn 3:  [tools][system][user][asst][tools'][user][…]   ← longer hit, smaller tail
         ────────────── warm ──────────────^── cold ──

That picture is the whole theory. Everything else is consequences:

• Timestamps are poison. Current time: 14:32:07 at the top of a system prompt means byte ~20 differs on every request, so nothing ever hits. The most expensive line of code in agent-land is an innocent-looking new Date() in a prompt template.
• The early prompt must be frozen. Tool definitions render first; add, remove, or reorder one tool and the entire cache goes cold. Same for any conditional system-prompt section.
• Dynamic context belongs at the end. A message appended at turn twelve invalidates nothing before turn twelve. The tail is free real estate; the head is load-bearing.
• History is sacred. Rewriting a message from turn three — even to make it smaller — re-bills every token after turn three at full price.

The providers differ only in how you opt in and how you observe it. Efferent’s router resolves the provider at call time, then shapes each outgoing request for that provider’s cache. Three providers, three shapes.

OpenAI: automatic, plus a routing hint#

OpenAI is the low-ceremony end of the spectrum. Prompts past a modest size threshold are prefix-cached automatically — no parameter, no markers, no opt-in. Send the same prefix twice in close succession and the second request reports cached tokens in its usage block and bills them at a discount.

There’s one knob worth turning anyway. OpenAI’s cache is sharded across their fleet, and a request only hits cache entries on the machine it lands on. The prompt_cache_key parameter is a routing hint: requests carrying the same key get routed consistently, so they keep landing where their warm prefix lives.

A static key for the whole process works for one session and degrades when two run together: every conversation shares one routing lane, and one session’s long prompts evict the other’s warm prefix. Nothing errors; both sessions still “cache,” just worse. Efferent therefore keys the route per conversation, and the way that identity travels is a useful piece of Effect.

The key is a FiberRef — fiber-scoped ambient state — declared in the engine:

/**
 * The CURRENT conversation's cache identity, stamped by `runAgent` for the
 * duration of a run. A FiberRef, not a port: it's ambient call metadata,
 * not a capability — and it must flow through the loop's fibers without
 * threading a parameter through every seam.
 */
export const CurrentPromptCacheKey = GlobalValue.globalValue(
  '@xandreed/engine/CurrentPromptCacheKey',
  () => FiberRef.unsafeMake(Option.none<string>()),
)

runAgent stamps it once, around the whole run:

return yield* runLoop({ /* … */ }).pipe(
  // One conversation, one server-side cache lane: adapters that support
  // a prompt cache key read this for every call in the run.
  Effect.locally(CurrentPromptCacheKey, Option.some(String(conversationId))), 
)

And the OpenAI provider reads it at the other end. The crucial detail: the router builds the provider client per call, inside the run’s fiber — never at layer build — which is exactly why a FiberRef works here with zero parameter threading:

Effect.flatMap(FiberRef.get(CurrentPromptCacheKey), (cacheKey) =>
  OpenAiClient.make({ apiKey: key }).pipe(
    Effect.flatMap((client) =>
      OpenAiLanguageModel.make({
        model: selection.modelId,
        // One shared constant lane cross-evicts between conversations —
        // construction happens per call, inside the run's fiber.
        config: {
          prompt_cache_key: Option.getOrElse(cacheKey, () => 'efferent'), 
        },
      }).pipe(Effect.provideService(OpenAiClient.OpenAiClient, client)),
    ),
  ),
)

The OpenAI-compatible gateway path (efferent’s default provider) sends the same field in its request body — present when a run stamped an id, absent otherwise — and the colocated test pins both cases: “the engine’s cache identity rides as prompt_cache_key; absent when unstamped.” One conversation, one cache lane; parallel sessions stop evicting each other.

Gemini: implicit, visible only in the receipts#

Gemini calls its version implicit context caching, and it’s the same deal as OpenAI ergonomically: stable prefixes are cached automatically, cached tokens are billed at a discount, and there is nothing to send. What’s different is where you see it — Gemini reports a cachedContentTokenCount inside the response’s usage metadata, and if you don’t go looking for it you’ll never know whether your prefix discipline is working.

efferent normalizes every provider’s usage report into one TokenUsage shape, and the Gemini cache figure comes from exactly that field — dug out of the finish part’s metadata, where the adapter parks it:

export const extractUsage = (usage: unknown, content: ReadonlyArray<unknown>): TokenUsage => {
  // …resolve the finish part; Gemini's numbers ride its google metadata
  const cacheReadTokens =
    u.cachedInputTokens ?? fu?.cachedInputTokens ?? gm?.['cachedContentTokenCount'] ?? 0
  // …input/output/total resolved the same way — plus an Anthropic
  // correction that gets its own section below
  return { inputTokens, outputTokens, totalTokens, cacheReadTokens }
}

Gemini also offers an explicit caching API — you upload content as a named cachedContent resource and reference it by handle, paying storage instead of re-transmission. efferent doesn’t use it, for an honest reason: the @effect/ai-google adapter always sends the full contents array and has no way to suppress the system prompt and tools in favor of a cache handle, so wiring it up would be theater. Implicit caching is real and live; the explicit tier waits until the library can express it. Knowing which optimizations you’re actually getting beats pretending.

Anthropic: nothing caches until you say so#

Here’s the one that bites people. Anthropic caches nothing by default. No markers in the request, no caching — every turn re-reads the entire conversation at full input price, forever, silently. The API is perfectly happy to let you do this. Most hand-rolled Anthropic clients I’ve read do exactly this, and their authors don’t know, because the only symptom is a bill.

Opting in means attaching cache_control: { type: 'ephemeral' } to specific content blocks in the request. Each marked block is a breakpoint: a position where Anthropic will both write a cache entry for the prefix ending there and look back for an existing entry to read. “Ephemeral” names the default lifetime — an entry lives about five minutes, refreshed each time it’s read, so an active session keeps itself warm. The economics make the opt-in design legible: cache reads cost ~0.1× the base input price, but cache writes cost a premium — about 25% over base for the five-minute tier. Caching is a bet that the same prefix comes back. Anthropic makes you place the bet explicitly.

You get at most four breakpoints per request, so placement is a real decision. The pattern for conversational agents — straight from Anthropic’s own docs, implemented in efferent as a pure function the router applies to every Anthropic-bound request — uses three: the last system message, and the last two non-system messages.

One anchor, two moving markers#

export const withAnthropicCacheBreakpoints = (options) => {
  const messages = Prompt.make(options.prompt).content
  if (messages.length === 0) return options

  // 1. the LAST system message — anchors the static prefix
  const lastSystem = messages.reduce((acc, m, i) => (m.role === 'system' ? i : acc), -1)

  // 2. the LAST TWO non-system messages — the moving frontier
  const nonSystem = messages.flatMap((m, i) => (m.role === 'system' ? [] : [i]))
  const stampIdx = new Set([...(lastSystem >= 0 ? [lastSystem] : []), ...nonSystem.slice(-2)])
  // …stamping below
}

Why these three positions?

The last system message is the anchor. Anthropic renders requests in a fixed order — tools first, then system, then messages — and a breakpoint covers everything before it. So one marker on the final system block caches the entire static prefix: every tool definition plus the whole system prompt, the part of the request that is byte-identical for the lifetime of a session. One breakpoint, maximum coverage.

The last two non-system messages are the moving frontier, and this is the part that looks wrong until it clicks. The markers sit on this turn’s newest messages — but this turn’s newest messages have never been seen before, so what is there to read? The answer is that a breakpoint reads and writes at different points in time. This turn, the marker on the tail writes a cache entry for the whole prefix ending there. Next turn, the transcript has grown past it, and when Anthropic processes the new request it looks back from the new breakpoints for the longest previously-cached prefix — and finds the entry last turn’s marker wrote. The previous turn’s breakpoint becomes this turn’s cache hit; this turn’s breakpoint becomes the next turn’s. The frontier advances one turn behind the conversation, and everything behind it stays warm.

Why two tail markers instead of one? Insurance for the lookback. A tool-heavy turn can append a long run of blocks — assistant message, a dozen tool results, a new user message — and Anthropic’s backward search from a breakpoint scans a bounded window, not the whole transcript. Two markers keep a read point close to wherever the previous write landed even when the tail grows by more than one message. Three breakpoints spent, one left in the budget.

And the question that should be nagging you: the markers move every turn — doesn’t moving them change the request bytes and break the very prefix-match the whole scheme depends on? It would, except Anthropic deliberately excludes cache_control placement from the content hash. Markers are metadata about the bytes, not part of them. The function’s own doc comment states the invariant: marker placement is not part of Anthropic’s content hash, so the markers moving forward each turn never invalidates earlier prefixes. The entire pattern is legal only because of that one carve-out.

Stamp gently: explicit markers win#

The second half of the function is about what it doesn’t do:

const stamped = messages.map((msg, i) => {
  if (!stampIdx.has(i)) return msg
  const anthropic = msg.options['anthropic'] ?? {}
  if (anthropic.cacheControl !== undefined) return msg // an explicit marker wins
  return Prompt.makeMessage(msg.role, {
    ...msg,
    options: { ...msg.options, anthropic: { ...anthropic, cacheControl: { type: 'ephemeral' } } },
  })
})
return { ...options, prompt: Prompt.fromMessages(stamped) }

Two courtesies, both load-bearing. Provider options are merged, not replaced — a message already carrying other Anthropic metadata keeps it. And a message that already has an explicit cacheControl is left untouched, which matters because ephemeral isn’t the only tier: you can pass ttl: '1h' for an hour-long entry at a higher write premium, and a caller who deliberately opted a long-lived prefix into that tier should not have the router silently downgrade it to five minutes.

The router applies this to every Anthropic-bound call — the agent loop and the one-shot helper tier (titles, digests). That second half is a small known overpay: a helper prompt is sent once and never again, so its markers buy a write premium and read back nothing. In practice the helper tier defaults to gateway models and rarely lands on Anthropic, but the principle is worth naming — a marked prompt that never recurs is a guaranteed loss, and a cleverer router would decline to place that bet.

The contract the rest of the system signs#

Everything so far is request shaping. But the prefix-match invariant doesn’t care about your request-shaping code — it cares about the bytes, and the bytes come from the whole system. Stamping breakpoints onto a transcript that some other subsystem rewrites is paying the write premium to cache garbage. The cache is a contract, and every component that touches history has to sign it. Two clauses.

Clause one: history is append-only. A persisted message is never edited. Not to fix a typo, not to redact, not to shrink. The tempting move when a transcript bloats is exactly the forbidden one:

// the obvious fix for a bloated transcript — and a cache poisoner:
history[3].content = summarize(history[3].content)
// one rewritten byte at position ~12,000 → everything after it re-bills cold

// the cache-safe shape: shrink on the way IN, then never touch it
history.push(compress(res.tail))

Clause two: when history must shrink, shrink it as one deliberate prefix rebuild. Eventually a session outgrows its context window; the loaded history has to fold into a summary. That’s a prefix change — there’s no avoiding the cold miss. The discipline is to take the hit once, on purpose, at a turn boundary, and efferent’s within-run compaction is built around exactly that: when a finished turn’s input tokens cross the threshold, everything but the last few assistant turns folds into a summary — and the cut always lands on an assistant message, so a tool call is never split from its results. The summary is written as a checkpoint; the original rows are never modified (a :resume replays the full story from them). The next request goes out with a brand-new prefix — a handoff note plus summary first, the kept tail after — one cold request writes the new cache entries, and every turn after that is warm again. A cooldown (three turns between folds) keeps the fold from thrashing.

The failure mode this clause forbids is incremental rewriting — trimming a little here, rewording a little there, each edit a fresh full-price re-read of everything downstream. Fold rarely, fold completely, and let the frontier rebuild.

Neither clause lives in the caching code. They live in the store and the loop — which is the point. You can’t bolt this property on; the architecture has it or it doesn’t.

Watching it work: the receipt on every turn#

A property this silent needs instrumentation, because both failure modes — never caching, and quietly breaking the prefix — produce zero errors. efferent’s TUI puts the answer on every turn: each settled assistant message lands with a spend tag —

opencode:kimi-k2.6 · turn 15.6k in (14.3k cached) · 135 out

— where the cached figure is what the provider’s prefix cache absorbed of that turn’s input. Keeping it on each turn makes the exact prefix break visible. The session cost estimate prices those cached reads at 0.1× the fresh-input rate, so the discount is visible in dollars too.

Feeding that tag honestly requires un-warping one provider’s accounting. Anthropic’s input_tokens excludes cache reads and writes — on a fully cached turn it reports only the cold tail, and a naive gauge would claim a 150k-token conversation is using 2% of its context window. The raw cache figures ride in provider metadata on the finish part, and extractUsage folds them back in:

// Anthropic's input_tokens EXCLUDES cache reads/writes (both ride only in
// the raw usage on the finish part) — without folding them back, a
// fully-cached turn reads ~0 input.
const cacheRead = au['cache_read_input_tokens'] ?? 0
const cacheWrite = au['cache_creation_input_tokens'] ?? 0
const fullInput = inputTokens + cacheRead + cacheWrite 
return {
  inputTokens: fullInput,
  outputTokens,
  totalTokens: fullInput + outputTokens,
  cacheReadTokens: cacheRead,
}

What healthy looks like, live: turn one of a session shows no cached share — nothing existed to read; that turn pays the writes. Turn two jumps high immediately, and the share climbs asymptotically from there as the warm prefix grows relative to each turn’s tail. Late in a long session, 95–98% of the input arriving cached is normal and correct.

What broken looks like is just as legible. A cached share that craters mid-session means the prefix changed somewhere early: an edit to persisted history, a tool definition that shifted, a system prompt that picked up dynamic content. A share pinned at zero on Anthropic means the breakpoints never went out at all. Either way you find out on the turn it happens, from the transcript pane, instead of three weeks later from the invoice. If you build an agent and take one thing from this post, take the readout: you cannot keep a promise you can’t see yourself breaking.

What it costs#

Honest ledger, because none of this is free.

The ephemeral TTL is short. Five minutes, refreshed on each read. An active session keeps itself warm indefinitely, but step away for coffee and the next turn pays cold-write prices on the whole transcript again. The 1-hour TTL exists for exactly this, at roughly double the write premium — which means it needs more subsequent reads to pay for itself, and for an interactive session the five-minute tier usually wins. There’s no free tier of patience.

Writes cost more than not caching. The +25% premium means stamping breakpoints on a prompt that never recurs is a guaranteed loss — the helper-tier overpay named above. And all three providers enforce a minimum cacheable prefix — on the order of a thousand to a few thousand tokens depending on model — below which markers are silently ignored. Small prompts don’t cache, and shouldn’t.

The breakpoint budget is real. Four per request, three spent on the conversational pattern. A feature that wants its own breakpoint — a per-document marker for a RAG prefix, say — is competing for one remaining slot, and preserving explicit markers (as the stamping function does) is also how you avoid blowing the budget by double-stamping.

Caching constrains what you’re allowed to want. This is the deep cost. A frozen prefix means no timestamp in the system prompt, no per-request mode flags up top, no mid-session tool swaps, no “small cleanup” of old history — designs that are perfectly reasonable in a stateless API become cache poison in an agent. You end up routing every dynamic impulse to the tail of the prompt, which is sometimes awkward and always deliberate. The cache doesn’t just discount your architecture; it disciplines it.

Cross-provider means per-provider shaping, forever. One provider needs markers stamped, one needs a conversation-keyed routing hint, one needs nothing sent but its usage read from a different field — and one reports input tokens in a way that breaks your gauge unless you correct it. A multi-provider agent can’t have a single caching story; it has a router that knows three. And the caches are per-model: a :model switch mid-session is a full cold rebuild, priced accordingly.

The bill is a code review#

Here’s the reframe I’d put on a sticky note for anyone building an agent: your provider bill is reviewing your architecture whether you asked it to or not. Every property the cache demands — a frozen prefix, append-only history, folds taken deliberately instead of edits taken casually — is a property a well-designed transcript store wants anyway. Append-only history is what makes sessions replayable and auditable. A stable system prompt is what makes behavior reproducible. Deliberate folds are what make context management explainable. The cache pays you a 10× discount for discipline you should already have, and quietly fines you for every shortcut, in a currency you can read off the transcript.

Most teams discover prompt caching as a line item — someone notices the bill and asks why it’s so high, and the answer turns out to be a timestamp. Flip it. Treat the cache as a design constraint from day one, put the cached share where you can see it, and the billing detail takes care of itself. The discount was never the point. The architecture that earns it is.