Contextwaise
LLM context optimization

How to optimize the context you send to an LLM: a 12-step checklist

How to optimize LLM context, in one sentence: send the smallest set of tokens that still lets the model answer correctly: measure what each part of the prompt costs, give each part a token budget, keep only the relevant retrieved passages and tool output, remove duplicates and machine noise, put fixed content first so it caches, and check answer quality on every change. The 12 steps below are the full checklist, in the order that saves the most for the least risk, with what you can do by hand and what Contextwaise automates.

Updated November 16, 202614 min readBy the Contextwaise team
Key takeaways
  • Context is what you pay for: input tokens usually outnumber output tokens many times over, and in agents every tool result is resent on every later step.
  • Most of the waste is not subtle: retrieved passages that don't answer the question, duplicates, progress bars, empty JSON fields and old tool output.
  • Decide what goes into the context once, when it arrives, and never edit it afterwards. Editing history breaks the prompt cache.
  • Keep text verbatim where you can. Selecting and trimming is safer than summarizing, which can change numbers and conditions.
  • Measure answer quality next to token counts. A smaller prompt that answers wrong is not a saving.
  • Contextwaise automates steps 3 to 7, 9 and 11 before each request, logs every removal with its reason, and checks the kept context was enough.

The checklist at a glance

#StepWhat it savesRiskContextwaise
1Measure what is in your contextShows where the tokens goNoneSavings report per agent
2Give every part a token budgetStops context from growing uncheckedLowBudgets per agent and per tool
3Retrieve broadly, send narrowlyOften half of a RAG prompt or moreMedium: needs evalsRelevance scoring in ~100 ms
4Remove duplicatesRepeated passages and messagesLowYes, automatic
5Shape machine outputUp to 90%+ of logs and test outputLowFree local shapers
6Keep text verbatimPrevents silent errorsLowers riskExtractive only, never rewritten
7Trim tool output when it arrivesThe biggest agent costLow if done onceAgents SDK, at the moment a tool returns
8Manage conversation historyLong chatsMediumYou summarize; Contextwaise keeps new content stable
9Order for the prompt cacheUp to 90 to 95% on the fixed prefixNoneDecisions replayed byte for byte
10Order for attentionAccuracy on long promptsNoneYour prompt template
11Keep a way backMakes aggressive cuts safeLowers riskexpand tool
12Measure quality, not only tokensCatches bad cutsLowers riskQuality guard

1. Measure what is in your context

You can't optimize what you haven't measured. Log the token count of every part of the prompt for a day of real traffic: system instructions, tool definitions, conversation history, retrieved passages, tool results and the user's message. Sort by size. In most RAG apps the retrieved passages dominate; in agents it is tool output, because each result is resent on every later step.

Python
parts = {"system": system, "tools": json.dumps(tools), "history": history_text,
         "retrieved": "\n".join(chunks), "tool_output": tool_text, "user": question}
log.info({name: count_tokens(text) for name, text in parts.items()})
Use the provider's token counting endpoint: the same text counts differently on each provider.

With Contextwaise: every request reports tokens before and after, and the savings report adds them up per agent, so you see which source is worth working on. To size a single prompt, paste it into the token counter.

2. Give every part of the prompt a token budget

Without a budget, context only grows: one more retrieved chunk, one more field in the API response, one more turn of history. Set a ceiling per part, and fill each part with its most valuable content until the ceiling is reached.

Part of the promptExample budget (16K request)
System instructions and tool definitions3,000 (fixed, cached)
Conversation history2,000
Retrieved passages5,000
Tool output3,000
User message500
Room for the answer2,500

These numbers are an example, not a recommendation. A question that needs no retrieval shouldn't spend 5,000 tokens on it: budgets are ceilings, not targets. With Contextwaise: set a token budget per agent, or per tool in the agents SDK (budgets={"read_file": 1500}), and it fills it with the most relevant content.

3. Retrieve broadly, send narrowly

Top-k retrieval is built for recall: k is set high enough that the right passage is almost always in the list, so most of the list is there just in case. Retrieve generously, then score each passage against the question and send only the ones that clear a threshold, within the budget. A reranker or a relevance model does the scoring; a fixed top-k does not.

With Contextwaise: one call to the context optimization API scores every candidate against the question and keeps what fits the budget. On our example support request it kept 3 of 13 retrieved items and cut tokens by 71%, in 510 ms end to end (our own example, not a customer workload). The trade-offs between retrieval tuning, reranking and compression are in RAG context compression.

4. Remove duplicates

Duplicates are free to remove and never help: the same paragraph from three versions of a document, a FAQ answer that is also in the policy page, a tool called twice with the same arguments. Remove exact duplicates with a hash, and near duplicates with a similarity threshold, keeping the most recent or most authoritative copy.

With Contextwaise: duplicates are removed automatically, and with a conversation id, a new item that repeats something already sent earlier in the conversation is dropped too.

5. Shape machine output before the model reads it

Logs, test runs, API responses and web pages are mostly formatting: progress bars, color codes, hundreds of passing tests, repeated log lines, empty and null JSON fields, HTML markup. Plain code removes all of it, deterministically, without a model. Keep every error and say what you removed, so the model doesn't assume something is missing.

With Contextwaise: the agents SDK runs these shapers locally and for free before anything else: it strips color codes and progress bars, drops passing tests, collapses repeated log lines, removes empty JSON fields and turns HTML into text. Errors are never removed. On our demo agent, a 7,209-token test run became 156 tokens, and 28,309 tokens of application logs became 564.

6. Keep text verbatim: select, don't rewrite

There are two ways to shrink text. Extractive methods keep whole sentences, passages or lines and drop the rest, so what the model reads is the original. Abstractive methods ask a model to summarize, which costs a call and can change a number, a date or an exception. Token-level compressors such as LLMLingua sit in between: high ratios, but the output is telegraphic and can break IDs, code and JSON.

Prefer extractive selection wherever facts, citations or code matter. The five families of methods and when each fits are compared in our prompt compression guide. With Contextwaise: kept text is always the original text, byte for byte. Nothing is paraphrased.

7. In agents, trim tool output once, when it arrives

In an agent loop, every tool result becomes part of the history and is resent on every later step. A 20,000-token log read at step 3 of a 20-step task is paid for about 17 more times. That makes tool output the largest and fastest-growing cost in most agents.

Trim each result at the moment the tool returns, before it is first sent, and store the trimmed version in the history. Trimming it later, once it is already in the history, edits a message the provider has cached, and every token after that point is billed in full again. Tool schemas and lists of actions should always reach the model whole, because the next call must follow them exactly.

With Contextwaise: the agents SDK gives the model a short view of each tool result, once, when the tool returns. Your code keeps the raw output. On our demo agent over five questions, tools returned 52.6K tokens and the model read 9.1K (−83%), and the run cost $0.057 instead of about $0.129. More in agent context optimization.

8. Manage conversation history

LLM APIs are stateless, so every turn resends the whole conversation. Three habits keep it in check:

  • Keep the last few turns verbatim; they carry the current thread.
  • Replace older turns with a short running summary of decisions, facts and open questions, updated incrementally rather than regenerated every turn.
  • Store stable facts (the user's plan, preferences, IDs) as structured memory and inject only what the current turn needs.

Summarizing rewrites history, so do it at a boundary (every N turns), not on every turn: each rewrite invalidates the cached conversation after that point. With Contextwaise: history summarization stays in your code. Contextwaise works on the new content each turn (the retrieved passages and tool results) and, with a conversation id, replays its earlier decisions exactly so the history never changes behind your back. See conversations.

9. Order the prompt for the cache

Prompt caching bills a repeated prefix at a fraction of the input price: on Claude Sonnet 5.5, $0.10 per million cached tokens instead of $2. It only matches a byte-identical start. Put everything fixed first (instructions, tool definitions, examples, a reference document), everything that changes last, and keep timestamps, request IDs and per-user data out of the prefix. The rules are the same at Anthropic and OpenAI.

Formula
[ system instructions ]  fixed  → cached
[ tool definitions    ]  fixed  → cached
[ reference document  ]  fixed  → cached
[ history             ]  grows only at the end
[ retrieved passages  ]  new each turn
[ user message        ]  new each turn

Don't compress the fixed prefix on every request: cached, it is already 95% off on Sonnet 5.5, and a prefix that changes never caches. The math is in prompt compression vs caching and the Anthropic specifics in prompt caching. With Contextwaise: each item's decision is made once and replayed byte for byte, so optimizing context never breaks your prompt cache.

10. Order the prompt for attention

Models use long inputs less reliably than short ones, and a fact in the middle of a long context is used less reliably than one near the start or the end. Within the variable part of the prompt, put the most relevant passages first, and restate the question or the key constraint at the end. The research behind this is in context rot.

11. Keep a way back

Every cut is a bet that the model won't need what was removed. Make the bet cheap to lose: give the model a tool that fetches the removed part on demand, and count how often it is called. Rarely is right. On most requests means you are cutting too hard.

With Contextwaise: the model gets an expand tool and can ask for any trimmed part back with a question. A removed passage costs one extra call, not a wrong answer, and the dashboard counts that call against the savings.

12. Measure quality, not only tokens

Cutting 70% of tokens means nothing if answers get worse. Keep a labelled set of 50 to 200 real requests, including the hard ones (exact numbers, multi-step questions, rare terms), and run it with and without each change: same model, temperature 0. Track input tokens, cache hit rate, latency, answer accuracy and cost per correct answer, and re-run the set when the model or the data changes.

With Contextwaise: every removal is logged with its reason, and the quality guard checks after each response whether the kept context was enough to answer, so the next request keeps more if it was not.

Do it by hand, or use a tool?

Steps 1, 2, 8, 9 and 10 are design decisions: do them yourself, once. Steps 3 to 7 and 11 run on every request and are where the code piles up: scoring, budgets, shapers per tool, deduplication across turns, cache-safe replay, an expand tool and its bookkeeping. Build them if context selection is core to your product or you already have a tuned retrieval layer. Otherwise Contextwaise does them in one call, with any model provider, through a REST API or the TypeScript and Python SDKs. The demo replays a real agent run with and without it.

Frequently asked questions

How do I optimize the context I send to an LLM?

Send the smallest set of tokens that still lets the model answer correctly. Measure each part of the prompt, give each part a token budget, keep only relevant retrieved passages, remove duplicates and machine noise, trim tool output when it arrives, put fixed content first so it caches, and check answer quality on a labelled set after every change.

What is LLM context optimization?

LLM context optimization is the step before a model call that decides what goes into the prompt: which passages, tool results and history to keep, in what order, within a token budget. Its goals are lower cost and latency and equal or better answers.

How much can context optimization save?

It depends on how noisy the input is. Machine output such as logs and test runs often shrinks by over 90%; retrieved context typically by half or more. On our own demo agent, the model read 83% fewer tool-output tokens and the run cost 55% less.

Does optimizing context break prompt caching?

It can. Caching needs a byte-identical prefix, so trimming or summarizing history after it was sent invalidates the cache from that point on. Decide what goes into the context once, when it first arrives, and resend it unchanged afterwards.

Is context optimization the same as prompt compression?

Prompt compression is one part of it: making a given text shorter. Context optimization also decides what enters the prompt at all, the budget per part, the order, and how history and tool output are handled across turns.

What tools help optimize LLM context?

Rerankers (Cohere Rerank and others) for retrieval, LLMLingua for token-level prompt compression, LiteLLM's compress() for chat history, provider prompt caching for fixed prefixes, and Contextwaise, which scores, deduplicates, shapes and budgets retrieved passages and tool output in one call, with an expand tool and a quality guard.

Keep reading

Cut your LLM costs
without cutting quality.

One layer between retrieval and inference removes the context your model never needed, in milliseconds. Fewer input tokens, lower cost, a prompt cache that keeps hitting, and every removal logged with its reason.

See it live
Keep your model, retriever and prompts. Remove one call to roll back.