LLM Cost Optimization Checklist for Production Apps

LLM cost is the one number in an AI product that moves in the wrong direction every time usage grows. Traffic is a feature, invoices are not. Teams that treat cost optimization as a quarterly cleanup usually discover that the expensive part is baked into the architecture: the wrong model tier for the job, the wrong context size, the wrong retrieval strategy, and no mechanism to tell any of it apart from noise.

The playbook below runs against a live application with production traffic and a bill you cannot ignore.## The Unit Economics Frame

Start by defining a cost unit before touching any code. For most applications the meaningful unit is not a token, it is a resolved task: a support answer that resolved a ticket, a document classified, a lead enriched. Expressing spend per resolved task rather than per request makes every decision easier, because it forces you to compare a cost reduction against a quality reduction.

The measurement requires three fields you may not currently log:

Tokens in, tokens out, and cached tokens, per request, tagged with the model identifier and the route that triggered the call.
An outcome signal — did the user accept the answer, did the ticket close — attached to the same request identifier.
A stable feature key so you can attribute spend to a product surface rather than a team guess.

Without the third field, cost optimization degenerates into blanket downgrading.

The Checklist

1. Route Every Task to the Cheapest Model That Passes Its Quality Bar

Model tiering is the highest-leverage move available, and it is usually the first one skipped. The mistake is treating the model as a global setting when it is really a per-task decision.

Build a routing layer with three tiers. A small, fast model handles classification, routing, extraction, entity normalization, and redaction. A mid-tier model handles summarization, rewriting, structured generation, and routine question answering over retrieved context. A frontier model handles tasks where a wrong answer is expensive: multi-step reasoning, ambiguous requests, and anything a customer escalates.

The quality bar must be a test, not a feeling. Take a fixed evaluation set of real requests — fifty to two hundred is enough to start — label the acceptable outputs by hand, and require each candidate model to clear a threshold before it serves traffic. When a task is routed to a cheaper model, keep a sampling percentage on the expensive tier so you can detect drift.

The most common failure is routing on input length alone. Length correlates weakly with difficulty: a short, ambiguous question can be harder than a long document containing the answer verbatim.

2. Shrink the Context Window Deliberately

Context is usually where the hidden cost lives, because it scales with usage in a way that feels free. Three techniques cut it directly.

Retrieval over concatenation. When your prompt includes documents, retrieve fewer and retrieve better ones. Sweep the number of retrieved chunks as a parameter and record the quality curve, not just the cost curve. The finding is usually that three well-ranked chunks match twenty mediocre ones.

Context trimming and compaction. Long multi-turn conversations carry full history forward on every turn. Summarize older turns into a running state object, keep only the last few turns verbatim, and preserve system instructions and tool results outside the compaction.

Structured output over prose. Asking a model to return a schema rather than an essay eliminates the extra tokens spent on explanation, preamble, and formatting repair, and it reduces parsing failures that trigger retries.

3. Get the Cache Working, Then Measure the Hit Rate

Prompt caching is the largest single cost lever in most production systems, and it is usually left partly on the table. The principle is that a large, stable prefix — system instructions, tool definitions, long reference documents — can be reused across calls if it is placed at the front of the prompt and stays byte-identical.

The mechanics punish sloppy ordering. Anything that varies per request — a timestamp, a user ID, a dynamically sorted list — must go after the cacheable prefix. Audit your prompt assembly code and confirm the prefix is genuinely static, including whitespace and system prompt version strings.

Instrument cache hit rate by feature. A hit rate far below what your traffic pattern should produce means the prefix is being invalidated — a bug with a large dollar value.

4. Stop Paying for Reasoning You Do Not Need

Extended reasoning modes and long thinking budgets have real uses, but they are not defaults. Route them to the cases that need them: ambiguous tasks, tasks where a wrong answer is costly, and tasks that decompose into multiple steps.

For everything else, set explicit token budgets for reasoning effort and cap them. Where the API exposes a reasoning-effort or thinking-budget parameter, treat it as a tunable with its own quality evaluation, not as a boolean.

Two further wins: ask for concise output when you only need the answer, and stop the model from restating the prompt. Both reduce output tokens without costing quality on extraction and classification tasks.

5. Batch and Cache at the Right Layer

Offline work should not be synchronous. Summarizing a document, enriching a lead record, pre-classifying a backlog — none need a sub-second response. Move them to a batch path with lower per-token pricing and accept the latency.

Cache application-level results. If a user asks the same question twice, a semantic cache in front of the model — matching on embedding similarity, not exact text — can serve the second request without touching the API. Use a high similarity threshold, an expiry window, and never cache responses containing permission-sensitive data.

6. Retire the Silently Expensive Code Paths

Audit for patterns that multiply token consumption invisibly. Retry loops without a token ceiling. Recursive summarization that re-summarizes a summary. A "improve this" button that refines the same output three times. Vector queries returning far more context than the prompt consumes. Conversation history replayed in full forever.

Each of these is invisible in a code review and obvious in a token histogram. Build the histogram.

7. Make Spend Observable and Budgeted

Instrumentation is what turns this checklist from a one-time project into an operating practice.

Per-feature cost dashboards, with a cost-per-resolved-task line, not just total spend.
Budgets with alerts, set at feature level so the alert fires before the invoice does.
Anomaly detection on tokens per request. A shift in the median often means a prompt change, a retrieval regression, or a loop.
A per-request cost ceiling in code, so one pathological input cannot generate an unbounded bill.

Failure Modes and Recovery

Quality collapse after a tier change. The cheaper model was cleared against a narrow evaluation set. Widen the set, re-run the comparison, and route by measured accuracy.

Cache hit rate near zero. Something in the prefix varies. Diff two consecutive assembled prompts byte by byte and find the moving part.

A runaway loop. A recursive agent or a retry storm. Cap tokens per request, cap recursion depth, and add a circuit breaker that degrades to a cheaper path.

The Measurement Loop

Run the checklist quarterly against a re-run of the evaluation set, and monthly against production cost data. Each cycle should produce a ranked list of cost drivers by feature, a quality-adjusted cost-per-task figure per feature, and one prioritized change with an expected saving and a quality impact.

Treat every optimization as an experiment with a quality gate, not a pure cost decision. The teams that sustain cost discipline made the quality bar explicit and measurable before they started cutting.

Summary

Production LLM cost is controlled in four places: model routing, context size, cache utilization, and batch versus synchronous execution. Instrument tokens by feature and cost per resolved task, route each task to the cheapest model that passes a written quality bar, shrink context deliberately, make the cacheable prefix static, cap reasoning effort, and move offline work off the synchronous path. Add per-feature budgets, per-request token ceilings, and anomaly detection so regressions surface in hours rather than in the next billing cycle.

Frequently Asked Questions

How do you decide which tasks can move to a cheaper model? Start from an evaluation set of real requests with hand-labelled acceptable outputs, then compare candidate models on quality and cost together. The routing rule usually follows task shape — classification, extraction, and formatting move first, because their failure modes are objective and cheap to check. Keep a percentage of traffic on the expensive tier as a control, and treat any accuracy drop there as a trigger to re-evaluate the route.

Does prompt caching actually make a meaningful difference at moderate volume? It usually does, because the savings compound on the stable prefix rather than on your whole request. The benefit scales with how large and how static that prefix is — long system instructions, tool definitions, and reference documents. The prerequisite is a byte-identical prefix, so audit how the prompt is assembled and confirm nothing variable sits in front of the cacheable region.

How should you handle a spike in usage from a single customer or feature? Measure cost per feature and per account before reacting, so you can tell a genuine feature problem from legitimate traffic growth. Then set two controls: a per-request token ceiling in code, and a per-feature budget with alerts. If the traffic is legitimate, price it or shape it with a tier; if it is pathological, the ceiling stops it.