Semantic Caching for LLM Apps: Cut Latency and Cost Without Losing Quality

Ask an LLM product the same question twice and you pay for it twice. On any production application, a meaningful share of incoming prompts are near-duplicates: a resent message, a retried request after a timeout, a rephrased follow-up, a teammate testing the same documentation question. Each one bills full price for tokens you already generated.

Semantic caching converts that second payment into a lookup. This guide covers the architecture that holds up under real traffic, the threshold that decides whether a hit is safe, and the request types that must stay out of the cache.

The economics of a duplicate request

A cache helps two budgets at once, and teams usually only measure the first.

Money. A retrieval-augmented answer costs several times more than a plain completion, because you are billing for the system prompt, the retrieved passages, and the generated answer on every call. When the same question arrives three times in an hour, two of those calls are pure waste. In agentic applications the waste compounds, because each step re-sends the accumulated context.

Latency. Users do not perceive your provider's throughput. They perceive how long the spinner lasted. Removing a generation step is the fastest improvement available to you, faster than rewriting your prompt or switching models, because it removes work rather than doing the same work faster.

Headroom. Rate limits usually decide how many concurrent users you can serve. Hitting a limit does not degrade gracefully; it produces errors. Cutting redundant upstream calls expands the number of real users you support on the same quota.

Why exact-match caching is not enough

The first thing most teams try is a hash of the prompt string, keyed on the full message history. It works, and it is close to useless, because real traffic rarely repeats character for character. Session identifiers change, whitespace changes, a user adds one clarifying sentence, a model version string gets appended to the system prompt. Exact-match hit rates on chat-style traffic are low enough that the engineering cost rarely pays for itself.

Semantic caching replaces string equality with intent equality. You embed the incoming prompt, search your store for the nearest neighbours, and serve a previous response only when the closest match is close enough. The unit of retrieval becomes meaning rather than text, which is why it collapses the variants you would otherwise miss.

That shift introduces a risk the exact-match version never had. A near match is not always an equivalent request. The similarity threshold is the entire safety mechanism, and it deserves more attention than the vector store.

A five-step rollout

Step 1: Instrument before you cache

You cannot tune what you cannot see. For one week, log every request with its normalized prompt hash, intent label, token counts, latency, and whether the user accepted, edited, or retried the answer. Then answer three questions with data: which intents repeat, how soon they repeat, and how long answers stay acceptable.

That last number drives your time-to-live. A pricing calculation and a legal explanation have very different shelf lives, and one shared expiry value will be wrong for both.

Step 2: Choose the embedding model and store deliberately

Use a small, fast, purpose-built embedding model rather than the same heavy model that generates answers. Response time is dominated by the slowest step in your path, and a heavyweight embedder turns your low-latency cache into a medium-latency one.

For storage, an in-process vector index is enough up to a few hundred thousand entries and keeps lookups in the same region as your app. Past that, use a managed vector store, but keep the cache behind an interface you can swap. Caching infrastructure gets replaced more often than application infrastructure, and you do not want that decision wired into your business logic.

Step 3: Set the threshold and the time-to-live

Start conservative. A high threshold means fewer hits and no wrong answers, which is the correct failure mode for a launch. Lower it once you have a labelled evaluation set.

Treat the threshold as a per-intent value rather than a global constant. General knowledge questions tolerate a lower threshold because a slightly different phrasing rarely changes a correct answer. Anything involving numbers, names, dates, or permissions needs a much higher bar, because near misses there produce confident, plausible, wrong output. If your embeddings are normalized, tune against cosine similarity; if they are not, make that explicit in configuration, because mixing metrics silently corrupts your score distribution.

Step 4: Decide what you actually return

There are three defensible strategies, and mixing them is where systems break.

Return the stored response verbatim. Fastest and cheapest, and correct only for deterministic, side-effect-free work.
Return the stored response as context. You re-run generation with the cached answer supplied as a grounding reference. You pay for generation but keep consistency, which suits long-tail phrasings of a common answer.
Return the stored retrieval, not the stored answer. You keep the expensive part, the search over your corpus, and let the model write again. This is the safest option when your corpus changes hourly and your answers must reflect it.

Most teams end up using the third option for anything corpus-derived and the first option for everything else. The important step is deciding explicitly per route rather than letting a framework default decide.

Step 5: Measure quality, not just savings

A hit rate on its own is a trap. You can hit on ninety percent of requests and still degrade the product. Track four numbers together: hit rate by intent bucket, median and tail latency, cost per resolved request, and the correction rate on served-from-cache answers. That last one is your early warning system, because it degrades before users write angry tickets.

What must never enter the cache

Four categories stay out, regardless of hit rate.

Authenticated and personal data. Caching by prompt text alone will happily serve one user's answer to another. If your responses contain anything user-specific, the cache key must include the identity scope, and per-user data is usually better excluded from shared storage entirely.

Tool calls with side effects. Anything that charges a card, writes to a database, sends a message, or mutates state must bypass the cache. Cache the planning step, never the execution step.

Time-sensitive factual answers. Inventory levels, remaining appointment slots, live prices, and current account balances are wrong the moment they are cached. Give these routes a very short expiry and treat any positive TTL as a bug.

Content you cannot reproduce. If an answer depends on state you cannot snapshot, or on a passage you cannot re-retrieve, do not cache it. Debugging a stale answer nobody can reproduce costs far more than the tokens you saved.

Failure modes to expect

Cache drift across model versions. The same prompt produces a different answer after an upgrade. Include the model version and prompt version in your key so old entries expire naturally instead of competing with new ones.

Embedding model changes. Swapping embedders invalidates every stored vector. Version your index and rebuild it rather than mixing distance scores from two models in one space.

Poisoned early entries. Whatever answered first becomes the cached truth for a week. Seed the store only after you trust the output, and test your bulk invalidation path.

Over-caching trivia. If a cached answer is factually correct but contextually wrong, users lose trust in the whole feature. When unsure whether an intent is cacheable, exclude it.

How to know it worked

A healthy semantic cache typically shows up as a large reduction in cost per resolved request alongside flat or improving correction rates. Latency should drop most in the tail, where long generations were previously the worst case. If savings arrive while correction rates climb, your threshold is too low, and tightening it is cheaper than any other fix.

Treat the cache as a product surface with its own metrics, dashboard, and rollback. That framing separates a cache that improves unit economics from one that quietly erodes answer quality.

Frequently Asked Questions

What similarity threshold should I start with for semantic caching?

Start high, around 0.95 cosine similarity, then lower it once you have a labelled evaluation set. High thresholds produce fewer hits and essentially no wrong answers, which is the right failure mode at launch. Lower thresholds only become safe when your intents are well separated, and even then you should use a stricter threshold for answers involving numbers, names, or dates. Treat the threshold as per-intent configuration rather than a single global constant, and revisit it whenever you change models.

Should I cache the generated answer or just the retrieval results?

It depends on how quickly your corpus changes. Caching the full answer is fastest and cheapest, and works for stable, deterministic questions. Caching only the retrieval keeps your expensive index lookup but regenerates the text, which is the safer choice when your source material changes frequently. Many production systems use the second approach for anything corpus-derived and full-answer caching for everything else, decided explicitly per route rather than left to a framework default.

How do I stop a cached answer from being wrong after a model upgrade?

Include the model version and prompt version in your cache key. When you upgrade, every entry written by the previous configuration becomes unreachable and expires naturally, instead of competing with fresh answers generated by a different model. The same versioning discipline applies to your embedding model: version the vector index and rebuild it rather than mixing distance scores computed by two different embedders in a single space.