Caching

Two different caches touch your traffic, and they behave nothing alike. The first is the gateway’s own response cache, which answers a request out of Fairmeter’s storage without calling a provider at all. The second is the provider’s prompt cache, which still calls the provider but charges you a discounted rate for the part of the prompt the provider already had in memory. A gateway hit costs you nothing. A provider prompt-cache read costs less than a fresh input token but is not free. Conflating the two is the single easiest way to misread your bill, so this page keeps them apart throughout.

Gateway response caching

When the gateway can serve a request from its own store, it does, and no upstream call happens. Three behaviours are live.

Single-turn traffic is matched by an exact-match layer first and a semantic-similarity layer second, so a near-identical prompt from the same team returns the stored response instead of paying for a fresh generation. Both layers are scoped per team and per requested model, so one team’s traffic never serves another’s responses and a response stored against one model id or alias is never replayed for a request that named a different one. Entries age out roughly seven days after they are written.

Multi-turn agent traffic is covered by a conversation-scoped layer, keyed on the whole transcript so far rather than on the last message, with a short TTL of five minutes by default and strict org isolation. This is the layer that makes a repeated agent loop cheap: an agent that re-derives the same state on the way to a slightly different next step pays for the new step, not for the whole prefix again.

Streaming requests hit the cache too. A cached streaming response is replayed frame by frame in the original wire shape, so your client sees a normal stream and cannot tell a hit from a fresh call other than by how fast it arrives.

A gateway cache hit is billed at zero. It is not exempt from your limits, though: the rate-limit and budget checks run ahead of the cache lookup, so a hit still counts against your per-minute request allowance. Size a replay-heavy workload against the request limit, not just the spend. Latency lands in the single-digit to low-double-digit millisecond range, and a measured production hit came back in about 11 ms. The request row records the hit, so you can see exactly which traffic was served from cache and what it saved you.

What still bypasses the gateway cache

A request that carries an image attachment is never looked up and never written. The cache key is built from text, so a content-blind key would let one image’s description answer a question about a different image. Error-shaped responses are refused too: a reply with neither text nor a tool call is never stored, so a single upstream failure cannot poison a key for the rest of its TTL. Gateway-generated notices are refused on the same grounds.

Turns that carry tool calls are not excluded. An agent step whose output is a function call is a cacheable turn like any other, because the alternative was that agent traffic structurally never hit the cache at all. If a step in your loop carries a side effect, make the tool call idempotent on your side and treat a cached replay the way you would treat any retry.

Invalidation

There is no manual purge surface. Entries age out on their own schedule, seven days for the single-turn layers and five minutes by default for the conversation-scoped layer, and are evicted under memory pressure. The conversation layer’s TTL is an org-level setting bounded at an hour, and setting it to zero turns that layer off for the org entirely.

The semantic layer matches on a fixed similarity cutoff rather than a per-org dial, so how close a paraphrase has to be to count as a hit is the same for every team. The cutoff is deliberately tight: a near-miss falls through to a fresh provider call rather than answering a different question.

Provider prompt caching

Provider prompt caching is a different mechanism owned by the upstream provider, not by Fairmeter. The gateway passes it through and accounts for it per request, splitting cached input tokens out from ordinary input tokens on every row. The economics are asymmetric: writing a prompt into the provider’s cache costs more than an uncached input token, and reading it back costs substantially less, so a cache that is written often and read rarely can cost you more than no cache at all.

The practical consequence is that a provider prompt-cache read shows up on your bill at a discounted rate, while a gateway cache hit shows up at zero. If you see cost on a request you believed was cached, check which cache you were counting on. The accounting detail, including how the four-way token split is reported and how to tell whether your cache strategy is paying for itself, lives on the billing page.

Caching and zero data retention

Turning on zero data retention does not disable response caching: cached entries remain active per your org’s configuration, stay org-isolated, and carry the short TTLs described above. The full boundary, including what is and is not persisted with ZDR on, is on the security page.