LLM cost optimization: 8 levers that cut inference spend 50 to 90%
LLM cost optimization means paying for fewer tokens per request without lowering answer quality. Stacked, the levers in this guide typically cut 50 to 90% of spend: batch APIs take 50% off, prompt caching bills repeated input at 2.5 to 10% of the normal price, and model tiers differ up to 100× in price. In RAG and agent apps input tokens dominate, so measure first, cut irrelevant context, then cache, route and cap output.
- Input tokens usually dominate cost in RAG and agent apps because every request re-sends retrieved chunks, tool output and history while the answer stays short.
- Measure before changing anything: cost per request, input and output tokens per request, and the share of context that actually influenced the answer.
- The levers, in order of impact for context-heavy apps: cut input context, cache the stable prefix (reads at 2.5 to 10% of input price), route by difficulty (tiers differ up to 100×), cap output (5× the input price), batch offline work (50% off).
- Stacked on an illustrative support agent at Sonnet 5.5 list prices, context cuts, caching, output caps and routing take the monthly bill from $20,000 to about $4,185 (−79%).
- Every reduction needs a quality constraint. Track answer quality on an evaluation set alongside spend, and treat quality regressions as a failed change.
- Token optimization starts with lossless cleanup: minified tool output without null and empty fields cut a real tool result from 256 to 140 tokens.
- Reduce token spend without blindly reducing context: score and select what reaches the model instead of shrinking top-k or truncating history.
The LLM cost equation
The cost of one LLM request is input tokens times the input price, plus output tokens times the output price, with cached input tokens billed at a discount. Every optimization technique works by shrinking one of those four terms or by moving a request to a cheaper price list.
Output tokens are priced several times higher than input tokens on most frontier models, which leads many teams to focus on shorter answers. That instinct is right for chat products with short prompts. It is wrong for RAG, agent and tool-using applications, where a request can carry 10,000 to 50,000 input tokens and produce 200 to 800 output tokens (illustrative ranges). At that ratio, input spend exceeds output spend even after the price difference.
A worked example
Take an illustrative support agent with 18,000 input tokens and 400 output tokens per request, priced at $3 per million input tokens and $15 per million output tokens. Input cost is 18,000 × $3 / 1,000,000 = $0.054. Output cost is 400 × $15 / 1,000,000 = $0.006. Input is 90% of the bill. Halving output length saves $0.003 per request. Removing 60% of the input context saves $0.032 per request, more than ten times as much.
LLM API prices in 2026: what each token costs
LLM prices differ by up to 80× between model tiers, output costs about 5× input, and cached input costs a small fraction of normal input. Those three ratios decide which lever pays off. Anthropic list prices from the Claude pricing page, checked November 2026, in USD per million tokens:
| Model | Input | Output | Cached input (read) | Typical work |
|---|---|---|---|---|
| Claude Haiku 5.5 (prompts up to 100K tokens) | $0.10 | $0.50 | $0.01 | Classification, extraction, routing, short summaries |
| Claude Sonnet 5.5 | $2 | $10 | $0.10 | Most RAG answers, routine coding, standard agent steps |
| Claude Opus 5.5 | $4 | $20 | $0.20 | Harder reasoning and long agent tasks |
| Claude Fable 5.2 | $8 | $40 | $0.20 | Frontier reasoning, planning, high-stakes work |
The Batch API halves every price in the table, input and output, for work that can wait. Every Claude model, with a monthly cost calculator, is on the Claude API pricing page, Google's tiers on Gemini API pricing, and DeepSeek, which halves every price off-peak instead of offering a Batch API, on DeepSeek API pricing. Compare them on your own prompts with the token counter.
What routing is worth: 1 million classification requests
Take 1 million requests a month, each with 1,000 input tokens and a 20-token label. That is 1 billion input tokens and 20 million output tokens:
| Model | Input cost | Output cost | Monthly total |
|---|---|---|---|
| Claude Fable 5.2 | $8,000 | $800 | $8,800 |
| Claude Sonnet 5.5 | $2,000 | $200 | $2,200 |
| Claude Haiku 5.5 | $100 | $10 | $110 |
Same task, 80× apart. If the small model passes your evaluation set on this request type, routing it there is the largest single saving available for that slice. If it does not, the retries and wrong labels cost more than the tokens saved. The model routing docs show how Contextwaise suggests a cheaper model only when the switch pays back the prompt cache it loses.
Why input tokens dominate in RAG and agent apps
Input tokens dominate because retrieval and orchestration layers are built to be generous, and the model is charged for everything they hand over. Each source of context adds tokens that were never checked for relevance to the current request.
| Context source | How tokens pile up | Typical share of input (illustrative) |
|---|---|---|
| Retrieved chunks (RAG) | Top-k returns weakly related chunks alongside the useful ones | 30 to 60% |
| Tool and API output | Full JSON payloads are pasted into the prompt as-is | 10 to 40% |
| Conversation history | Every prior turn is re-sent on every request | 10 to 30% |
| System prompt and instructions | Grows with every edge case a team patches | 5 to 15% |
| Web search results | Whole pages are included when a few passages matter | 0 to 30% |
The shares above are illustrative ranges, and your own distribution will differ. The point is structural: none of these sources decides what is worth reading. A vector database returns the nearest k vectors. A tool returns its whole response. A chat framework appends the whole transcript. The generation model is the first component that evaluates relevance, and by then the tokens are already paid for.
Anthropic's guidance on effective context engineering frames context as a finite resource with diminishing returns: past a point, more tokens do not improve answers and can degrade them. That makes context reduction a quality lever as well as a cost lever, provided the reduction removes noise rather than evidence.
Step one: measure cost per request
Before optimizing anything, instrument every LLM call to record input tokens, cached tokens, output tokens, model and cost, then aggregate by feature, agent and customer. Without per-request numbers, teams optimize the lever they understand rather than the lever that matters.
- Log token counts from the provider response on every call. Do not estimate from string length.
- Tag each call with the feature, agent step and customer that triggered it.
- Break input tokens down by source: system prompt, retrieved context, tool output, history, user message.
- Compute cost per request and cost per successful outcome (a resolved ticket, a completed task), not just cost per token.
- Sample requests and ask a simple question of each context item: did this item influence the final answer? The share that did not is your addressable waste.
The last step is the one most teams skip. It is also the one that reveals whether the problem is volume, model choice or context bloat. If 70% of retrieved chunks never influence the answer, no amount of prompt tightening will beat fixing the retrieval-to-prompt boundary.
Lever one: cut input context before inference
The single highest-impact lever for context-heavy applications is to optimize context before inference: score every context item against the current request, drop what is irrelevant or redundant, and fit the rest into a token budget. This is different from retrieving less, which throws away results before anyone has looked at them.
Retrieve broadly, send narrowly
Lowering top-k is the naive version of this lever. It cuts tokens, but it also cuts recall, because the useful chunk is sometimes ranked ninth. The better pattern is to retrieve a wide set of results, then run a cheap relevance decision per item and send only the survivors. Cheap here means a lightweight classifier, cross-encoder or embedding score, not another call to the generation model.
- Relevance filtering: keep an item only if it plausibly helps answer this request, not just if it is semantically near the query.
- Deduplication: collapse near-duplicate chunks, repeated tool results and restated history into one representative.
- Prioritization: order survivors by expected contribution so the strongest evidence appears first.
- Token budgets: pack items by value per token into a fixed budget per agent or request. See token budgets for how per-agent limits keep cost predictable.
- Sufficiency checks: after the response, check whether the kept context was enough, and loosen the relevance threshold for later requests when it was not. See quality guard.
Worked example
Take the same illustrative agent: 18,000 input tokens, of which 12,000 are retrieved chunks, 3,500 are tool output, 1,500 are history and 1,000 are the system prompt. Suppose relevance filtering keeps 4 of 12 chunks (4,000 tokens), tool output is reduced to the relevant records (900 tokens), and history is trimmed to the turns that matter (500 tokens). New input is 1,000 + 4,000 + 900 + 500 = 6,400 tokens, a 64% reduction. At $3 per million input tokens, cost per request drops from $0.054 to $0.019 on the input side. Across 500,000 requests a month that is $27,000 down to $9,600, before any other lever.
These numbers are arithmetic on assumed inputs, not a benchmark. The actual reduction depends on how bloated your context is today, which is exactly what the measurement step tells you. Tools like Contextwaise implement this layer as a single call between retrieval and inference and report the before and after tokens per request, but the technique works with any relevance scorer you can run cheaply.
For the methods themselves (trimming, extractive selection, LLMLingua) and how they interact with caching, see prompt compression. For source-specific detail, see RAG context compression for retrieval pipelines and agent context optimization for multi-step agents where tool output and state accumulate.
Lever two: cache stable prefixes
Prompt caching lets the provider reuse the processed form of a prompt prefix across requests, billing the cached portion at a fraction of the normal input price. It pays off when a large block of tokens is identical across many calls, which is true of system prompts, tool definitions and long reference documents. Both Anthropic and OpenAI document their caching rules and discounts. For Claude, our guide to Anthropic prompt caching works through the break-even math and what breaks the cache. If you also trim context, trim only what comes after the cached prefix: cache-safe context optimization explains how Contextwaise keeps that prefix byte for byte.
Caching has two constraints that limit how far it goes. First, only the stable prefix can be cached, so the prompt has to be ordered with static content first and per-request content last. Second, retrieved context, tool output and the user message change on every request, and those are usually the largest part of a RAG or agent prompt. Caching removes the cost of the parts that were already cheap to reason about, and leaves the variable context untouched.
Where it applies, the discount is steep. On Claude a cache read costs 0.1× base input on most models, 0.05× on Opus 5.5 and Sonnet 5.5, and 0.025× on Fable 5.2, while a 5-minute cache write costs 1.25×. Across OpenAI, Anthropic and Google, the Don't Break the Cache evaluation measured 41 to 80% lower API cost and 13 to 31% faster time to first token on long-horizon agent tasks.
Cutting context and caching are complementary. Cut the variable context first, then cache the stable prefix. Applying caching to a bloated prompt locks in the bloat at a discount.
Levers three to eight: routing, output, batching, agent loops, semantic caching, prompt hygiene
The remaining levers each save a meaningful but smaller share, and each carries a distinct quality risk. Apply them after context is under control, because a smaller model reading 18,000 tokens of noise is still an expensive, unreliable model.
Model routing
Routing sends easy requests to a cheaper model and hard ones to a frontier model. The saving equals the price gap times the share of traffic that can safely be routed down. The risk is misclassification: an easy-looking request that needed the stronger model produces a wrong answer that costs more than the tokens saved. Routing works best when the router has a clear signal, such as request type, and when the cheaper model is evaluated on that slice specifically.
Output limits
Setting a maximum output length and instructing the model to answer concisely reduces the highest-priced token type. Savings are bounded by how long answers are today. For agents, the larger output saving usually comes from reducing the number of steps, since each step is another full request with the entire context re-sent.
Batching
Batch APIs process requests asynchronously at a discounted price, 50% on input and output at OpenAI, Anthropic and Google, and the discount stacks with prompt caching. This applies only to work that can wait, such as backfills, classification jobs and nightly summaries. Interactive traffic cannot use it. Code, limits and the stacking math are in the OpenAI Batch API guide.
Agent loops and reasoning tokens
Agents multiply cost: every step is another full request that re-sends the accumulated context, and reasoning models bill their hidden thinking tokens as output, the most expensive token type. Cap revision and review rounds, stop when a pass no longer improves the result, pass diffs and compact state instead of full transcripts, and try a lower reasoning-effort setting before a bigger model. A supervisor agent in the Stop Wasting Your Tokens study cut token use on the GAIA benchmark by 29.68% on average without lowering success rate. Long transcripts also hurt quality: models showed an average 39% performance drop in multi-turn conversations across 200,000+ simulations in LLMs Get Lost in Multi-Turn Conversation. More in agent context optimization.
Semantic caching
Semantic caching returns a stored answer when a new question is similar enough to an old one, and skips the model call entirely. On a hit it saves 100% of the call, which is more than any other lever, but it is also the riskiest: "cancel my order" and "cancel my subscription" can look alike to an embedding, and cached answers go stale when prices or policies change. Use it only for stable, read-only answers, set a strict similarity threshold, and shadow-test it against the live model before serving from it. Unlike prompt caching, which still generates a fresh answer, a semantic cache hit never reaches the model.
Prompt hygiene
System prompts accumulate instructions for edge cases and rarely lose any. Reviewing them quarterly, removing dead instructions and moving rarely used reference material into retrieval typically trims a few hundred to a few thousand tokens per request. The saving is small per request but applies to every request and improves instruction following.
Comparing the levers
For applications where input tokens dominate, cutting context offers the largest saving for moderate effort, while routing and caching offer meaningful savings with narrower applicability. The table below gives the published or measured number behind each lever; your result depends on the starting point.
| Lever | Effort | Impact on spend | Quality risk | Applies to |
|---|---|---|---|---|
| Cut input context | Medium | 45 to 71% fewer input tokens in our measured examples; the biggest lever when context is bloated | Low with guardrails, high if done by truncation | RAG, agents, tools, long chats |
| Prompt caching | Low | Cached reads at 2.5 to 10% of input price on Claude; 41 to 80% lower cost on agent tasks in one study | None | Long system prompts, tool definitions, static references |
| Model routing | Medium | Tiers differ up to 100× in price; saving is the price gap times the routable share | Medium, from misrouted hard requests | Mixed-difficulty traffic |
| Output limits | Low | Output costs 5× input on Claude; small for RAG, larger for verbose chat | Low to medium, from truncated answers | All |
| Batching | Low | Flat 50% off input and output; stacks with caching | None | Offline and async jobs |
| Agent loop control | Medium | 29.68% fewer tokens on GAIA in one study, with no drop in success rate | Medium, from stopping too early | Multi-step agents |
| Semantic caching | Medium | 100% of the call on a cache hit | High, from wrong or stale answers | Repeated read-only questions |
| Prompt hygiene | Low | A few hundred to a few thousand tokens per request | Low, often improves quality | All |
Read the table as an ordering, not a forecast. The measurement step tells you which row is large for your workload. A chat product with short prompts and long answers should start with output limits and routing. A RAG or agent product should start with context.
Stacking the levers: a worked monthly bill
Take the support agent from the cost equation at Claude Sonnet 5.5 list prices ($2 input, $10 output, $0.20 cached read per million tokens): 500,000 requests a month, 18,000 input tokens and 400 output tokens each, with a 1,000-token system prompt. Apply the levers in order:
| Step | Monthly cost | vs baseline |
|---|---|---|
| Baseline | $20,000 | — |
| Cut input context (18,000 → 6,400 tokens, as in lever one) | $8,400 | −58% |
| Cache the 1,000-token system prompt | $7,500 | −63% |
| Cap output (400 → 250 tokens) | $6,750 | −66% |
| Route 40% of requests to Haiku 5.5 | $4,185 | −79% |
Cache writes are left out because they are negligible at this volume, and batching does not apply because the agent is interactive. For an offline job, the Batch API would halve the remaining bill again. The order matters: caching a bloated prompt or routing it to a cheaper model saves less than removing the tokens first.
Token optimization: 8 ways to reduce token usage
Token optimization means getting the same answer from fewer tokens. Most of the waste is not in your prompt but in what your code adds to it: raw tool output, whole web pages, near-duplicate chunks and old conversation turns. These eight techniques reduce token usage in roughly the order they pay off for RAG and agent apps.
- Clean tool output. Minify JSON and drop fields that are null or empty. APIs return pretty-printed objects built for humans and debuggers, and the model pays for every space and empty field.
- Keep only the fields the task needs. Request a field list from the API, or map the response to the few fields the agent uses before it reaches the model.
- Strip web pages to their text. Scripts, styles, navigation and cookie banners are often more tokens than the article. Extract the main content before sending it.
- Deduplicate. Retrieval and repeated tool calls return the same fact several times. Send one copy.
- Filter by relevance, not by top-k. Retrieve broadly, then keep the items that help answer this request, as in lever one above.
- Trim conversation history. Keep recent turns and the ones the current request depends on. An agent should not resend every old tool result in full on every step.
- Put stable content first and cache it. A byte-identical prefix is billed at a fraction of the price, as in lever two above.
- Cap output. Set a maximum output length and ask for a concise or structured format. Output tokens cost several times more than input tokens.
Worked example: one tool result
This is a real invoice lookup from our demo: two charges, a refund status and a few empty fields, the kind of result a support agent reads on every billing question. Counted with the o200k tokenizer:
| Same tool result, sent as | Tokens | Saved |
|---|---|---|
| Pretty-printed JSON, as the API returned it | 256 | — |
| Minified JSON | 170 | 34% |
| Minified, null and empty fields removed | 140 | 45% |
Nothing the model needed was removed, so this saving costs no quality. An agent that calls tools ten times per task repeats it ten times per task. To see what your own prompts cost, paste them into the token counter.
A 30-day LLM cost optimization plan
A realistic plan spends the first week measuring, the second and third weeks on the largest lever, and the last week locking in savings with an evaluation gate. Each step is reversible and produces a number you can report.
- Days 1 to 5: instrument. Log tokens and cost per call, tagged by feature, agent and customer. Break input down by source. Build a baseline dashboard.
- Days 6 to 8: build an evaluation set. Collect 100 to 300 real requests with graded answers. This is the quality constraint every later change must pass.
- Days 9 to 12: sample for waste. For 50 requests, mark which context items influenced the answer. Quantify addressable input tokens by source.
- Days 13 to 20: cut context. Add a relevance filter and token budget between retrieval and inference on the highest-volume feature. Compare tokens, cost and eval scores against the baseline. Ship if quality holds.
- Days 21 to 24: cache and reorder. Move static content to the prompt prefix and enable caching. Confirm the cached share in provider responses.
- Days 25 to 28: route and cap. Pilot a cheaper model on one well-defined request type with its own eval slice. Set output limits where answers run long.
- Days 29 to 30: report. Publish before and after cost per request, tokens per request, and eval scores. Set alerts on both cost and quality so regressions surface early.
Teams that skip the evaluation set tend to ship the cost saving and discover the quality regression from customers. Building the set on days 6 to 8 is what makes the rest of the plan safe to execute quickly.
Scale the plan to the bill. If your monthly API spend is smaller than a week of engineering time, enable caching, set output limits and stop there; a routing layer or custom context pipeline will not pay for itself until volume grows.
Measuring savings with a quality constraint
A cost optimization is only real if answer quality on a fixed evaluation set stays within an agreed tolerance. Report savings and quality together, per change, so nobody can claim one without the other.
The minimum report has four numbers per feature: input tokens per request before and after, cost per request before and after, an answer-quality score on the evaluation set before and after, and a groundedness or faithfulness score before and after. Groundedness matters because removing noise often improves it, which is evidence that the reduction removed the right tokens. A savings report that lists removed tokens and the reason for each removal makes the cost side routine; pair it with your own eval run for the quality side.
Set the tolerance deliberately. A small drop may be acceptable for an internal tool and unacceptable for a customer-facing legal assistant. The point of optimizing for cost subject to a quality constraint is that the constraint is explicit, agreed and enforced. For the broader discipline of deciding what reaches the model, see context engineering.
Frequently asked questions
What is token optimization?
Token optimization means getting the same answer from an LLM while sending and generating fewer tokens. The main techniques are cleaning tool output, stripping web pages to their text, removing duplicate and irrelevant context, trimming conversation history, caching the stable prompt prefix and capping output length. Because providers bill per token, every token removed without losing needed information is a direct saving.
What is the biggest lever for LLM cost optimization?
For RAG, agent and tool-using applications, the biggest lever is reducing input context: filtering irrelevant and redundant retrieved chunks, tool output and history before the model call. Input tokens usually make up most of the bill in these apps, so cutting them by half or more outweighs shorter answers or a cheaper model. For short-prompt chat products, output limits and routing matter more.
How much can LLM cost optimization save?
Typically 50 to 90% when the levers are stacked, depending on the workload. Batch APIs take a flat 50% off, prompt caching bills repeated input at 2.5 to 10% of the normal price on Claude, routing easy requests to a small model can cut their cost up to 100×, and cutting irrelevant context removed 45 to 71% of input tokens in our measured examples. Our worked support-agent bill drops from $20,000 to about $4,185 a month (−79%). Measure your own before and after on real traffic.
Why are LLM costs so high?
Because every request re-sends its full context: retrieved chunks, tool output, conversation history and the system prompt. Agents multiply that with every step, and reasoning models bill hidden thinking tokens as output, the most expensive token type. Bills grow with token volume, so they can rise even while per-token prices fall.
What is the cheapest way to run an LLM in production?
Send the fewest tokens to the smallest model that passes your evaluation set. Trim irrelevant context, cache the stable prefix, route easy requests to a small tier such as Claude Haiku 5.5 ($0.10 per million input tokens), cap output, and send work that can wait through a batch API for 50% off. Self-hosting open-weight models pays off only at high, steady volume, because GPU and operations costs do not shrink when traffic drops.
Does model routing actually save money?
Yes, when a meaningful share of traffic is easy and the cheaper model is evaluated on that slice. One million short classification requests cost about $8,800 a month on Claude Fable 5.2 and $110 on Haiku 5.5. The saving shrinks with misroutes that trigger retries, the cost of running the router, and lost cache hits, because prompt caches do not carry across models.
What is the difference between prompt caching and semantic caching?
Prompt caching reuses the processed prefix of a prompt; the model still generates a fresh answer, so quality is unchanged and only the price drops. Semantic caching returns a stored answer for a similar question without calling the model. It saves the whole call but can return wrong or stale answers, so use it only for stable, read-only questions.
Should I optimize cost per token or cost per outcome?
Cost per outcome. A cheaper request that fails and triggers a retry, an escalation or a second agent step costs more than a slightly more expensive request that succeeds. Track cost per resolved ticket or completed task alongside cost per request, and let quality gate every change.
Keep reading

RAG Context Compression vs Reranking
RAG context compression removes redundant, off-topic and low-value chunks before the model call. How it differs from retrieval and reranking, with math.

Agent Context Optimization: Stop Token Bloat
Agent context grows every step. Where waste comes from (tool output, memory, stale state) and patterns that cut input tokens without hurting task success.

Context Engineering: The Practical Guide
Context engineering decides what information reaches an LLM on each call. Definition, principles, workflow, anti-patterns and metrics.

OpenAI Batch API: 50% Off, Code and Limits
The OpenAI Batch API runs requests within 24 hours at half price and stacks with caching. Working code, limits, and one job priced on Claude and Gemini.