Prompt compression: send the model fewer tokens without losing the answer
Prompt compression reduces the number of tokens in a prompt before it reaches the model, by removing noise, keeping only the relevant passages, or dropping low-information tokens with a small model such as LLMLingua. It lowers the input bill and the latency of every call, and it often improves answers on long inputs. The catch most guides skip: compress the part of the prompt that changes on every request, and cache the part that doesn't, because compressing a cacheable prefix can cost more than caching it.
- There are five families of prompt compression: structural trimming, extractive selection, token-level pruning (LLMLingua), abstractive summaries, and soft prompts. Only the first four work with closed APIs.
- Structural trimming and extractive selection keep the text verbatim, so code, IDs, numbers and citations survive. Start there.
- LLMLingua-2 drops individual tokens and reaches high ratios, but its output is telegraphic text. Protect what must stay exact with
force_tokens, and keep it away from JSON and code. - Compression and prompt caching compete for the same tokens. On Claude Sonnet 5.5 a 10,000-token prefix costs $0.001 per request when cached and $0.0067 when compressed 3× on every request. Cache the fixed prefix, compress the variable part.
- Measure compression on your own labelled questions. A smaller prompt that answers wrong is not a saving.
What prompt compression is
Prompt compression is any step that makes a prompt shorter before the model reads it while keeping what the model needs to answer. The input can be retrieved documents in a RAG app, tool output in an agent, a long chat history, or a pasted log. The output is a smaller prompt sent to the same model.
It matters for three reasons. Input tokens are billed one by one, and in most production apps the input is many times larger than the output. Time to first token grows with input length. And models use long inputs less reliably than short ones, an effect known as context rot, so the tokens you remove were often hurting the answer as well as the bill.
What it is not: rewriting your instructions to be shorter is good hygiene, but it saves a few hundred tokens once. The volume is in the data you put in the prompt on every call, and that is where compression pays.
The five prompt compression methods, compared
The research splits methods into hard prompts, which produce text, and soft prompts, which produce learned vectors (Li et al., 2024 survey). For an app calling a hosted API, the practical split is how much of the original text survives.
| Method | What it does | Text stays verbatim | Extra cost | Best for |
|---|---|---|---|---|
| Structural trimming | Removes formatting noise: progress bars, repeated log lines, empty and null JSON fields, whitespace, boilerplate | Yes | Near zero, deterministic code | Tool output, logs, API responses |
| Extractive selection | Scores sentences, chunks or messages for relevance and keeps the best ones within a token budget | Yes, for what is kept | A fast scoring step (BM25, embeddings, a reranker) | RAG context, chat history, long documents |
| Token-level pruning | A small model drops individual low-information tokens (LLMLingua, LLMLingua-2) | No, telegraphic | A local model pass, GPU helps | Long prose where ratio matters more than exact wording |
| Abstractive summary | An LLM rewrites the input as a shorter summary | No, paraphrased | A full model call, and it can invent details | Old turns of a long conversation |
| Soft prompts | Encodes the prompt into learned vectors (gist tokens, embeddings) | No, not text at all | Training, access to model internals | Self-hosted models only |
The order of the table is the order to try them in. Each step down gains ratio and loses control over what the model sees.
Start with structural trimming and extractive selection
Most of the waste in a production prompt is not subtle. A test run prints hundreds of passing tests to report two failures. An API response carries dozens of empty fields. Ten retrieved chunks repeat the same paragraph from three versions of a document. None of that needs a model to detect.
Extractive selection is the next step: score each unit (a chunk, a sentence, a message) against the current question and keep the best ones until a token budget is full. BM25 is fast and good with exact terms such as error codes; embeddings catch paraphrases; a cross-encoder reranker is the most accurate and the slowest. The kept text is the original text, so citations and code still match their source. For retrieval pipelines in depth, see RAG context compression.
LiteLLM's compress() is a ready-made extractive compressor for chat history. It scores messages with BM25 (plus embeddings if you configure a model), always keeps the system message and the last user and assistant messages, and replaces the rest with short stubs. It also returns a litellm_content_retrieve tool, so the model can fetch a dropped message back. By default it only runs once a conversation passes 200,000 tokens, so set compression_trigger lower if your costs start earlier.
LLMLingua and LLMLingua-2: token-level compression
LLMLingua is Microsoft Research's open-source prompt compression library (MIT license). It uses a small language model to decide which tokens of a prompt carry little information and drops them. The family has three main versions:
| Version | Published | How it decides | Claim in the README |
|---|---|---|---|
| LLMLingua | EMNLP 2023 | Perplexity from a small causal model (GPT-2 small, LLaMA-7B): predictable tokens are dropped | Up to 20× compression with minimal performance loss |
| LongLLMLingua | ACL 2024 | Question-aware: keeps what is relevant to the question, reorders documents against lost in the middle | Up to 21.4% better RAG performance using 1/4 of the tokens |
| LLMLingua-2 | ACL 2024 Findings | A BERT-size token classifier trained on GPT-4 compressions; task-agnostic | 3× to 6× faster than LLMLingua |
Those are the authors' benchmark numbers. Your ratio depends on how redundant your text is, so measure it on your own prompts before you plan around 20×.
Where token-level compression breaks
- Exact strings. Dropping tokens inside an ID, a version number, a URL or a function name changes its meaning. Keep those out of the compressor or protect them with
force_tokens. - Code and JSON. The output is no longer valid syntax. Compress the prose around code, never the code.
- Instructions. A telegraphic system prompt is easier to misread. Instructions are short and fixed, so cache them instead.
- Latency on CPU. The compressor is a model. LLMLingua-2 is the light one, but budget for the extra step and run it on a GPU at volume.
Prompt compression vs prompt caching: which to use
Both lower the cost of input tokens, and they work on different tokens. Prompt caching discounts a prefix that is identical across requests: on Claude Sonnet 5.5 a cached input token costs $0.10 per million instead of $2, a 95% discount. Compression removes tokens, but whatever is left is billed at full price, and if the compressed text changes from one request to the next, the cache never matches.
Take a support bot that sends the same 10,000-token product manual with every question, 100,000 questions a month, on Claude Sonnet 5.5. Cache writes are left out to keep the comparison simple.
| The 10,000-token manual is… | Input tokens billed | Cost per request | Per month |
|---|---|---|---|
| Sent as is, no cache | 10,000 at $2/M | $0.020 | $2,000 |
| Compressed 3× per question (question-aware, so never cached) | 3,333 at $2/M | $0.0067 | $667 |
| Cached | 10,000 at $0.10/M | $0.001 | $100 |
| Compressed once, offline, then cached | 3,333 at $0.10/M | $0.00033 | $33 |
Compressing the manual on every request saves two thirds of the bill, but caching it untouched saves 95%. The rule that falls out:
- Fixed content (instructions, tool definitions, a manual, few-shot examples): put it first and cache it. If you compress it, do it once offline and cache the result.
- Variable content (retrieved chunks, tool output, the newest messages): compress it on every request. It can't be cached anyway, so every token removed is billed at nothing instead of the full price.
- Work nobody waits on (evals, backfills, labelling): send it through a batch API for 50% off every token, then compress and cache on top.
- Never compress inside the cached prefix at request time. One changed token at the start invalidates everything after it.
How to measure whether compression hurt the answers
A compression ratio on its own says nothing about quality. Measure both on the same labelled set before you ship, and keep measuring after.
- Collect 50 to 200 real requests with a known good answer, including the hard ones (multi-hop questions, exact numbers, rare terms).
- Run each one twice, uncompressed and compressed, with the same model, temperature 0 and the same instructions.
- Score the answers: exact match for facts and numbers, an LLM judge for open answers, and citation checks for RAG.
- Log input tokens and end-to-end latency for both runs, including the compression step itself.
- Pick the most aggressive setting whose accuracy stays within your tolerance, often within one or two points of the baseline, and re-run the set when the model or the data changes.
Keep a way back. When the model can ask for what was removed, through a retrieval tool as in LiteLLM's compress(), a bad cut costs one extra call instead of a wrong answer. Count how often that tool is called: if it fires on most requests, you are compressing too hard.
What compression saves on real inputs
Ratios depend almost entirely on how noisy the input is. These are measured on our own demo agent and example requests, not on customer workloads, with structural trimming and extractive selection only:
| Input | Before | After | Reduction |
|---|---|---|---|
| Test run output (2 failures among passing tests and progress bars) | 7,209 tokens | 156 tokens | −98% |
| Application logs | 28,309 tokens | 564 tokens | −98% |
| Support RAG request (13 retrieved items → 3) | 901 tokens | 264 tokens | −71% |
Machine output compresses the most, because most of it is repetition. Prose written by people compresses the least, and that's where token-level methods like LLMLingua earn their place. To size your own case, paste a real prompt into the token counter and price it per month.
A prompt compression checklist
- Log the token count of every part of your prompt for one day and sort by size. Compress the biggest variable part first.
- Move fixed content to the start and turn on prompt caching before you compress anything.
- Add structural trimming for tool output and logs. It's code, it's deterministic and it can't paraphrase.
- Add extractive selection for retrieved context and long histories, with a token budget.
- Try LLMLingua-2 only on long prose that is still too big, with
force_tokensfor anything exact. - Run the labelled set, compare accuracy and cost, and keep a retrieval path for what was removed.
Frequently asked questions
What is prompt compression?
Prompt compression is the step that shortens a prompt before the model reads it, keeping what the model needs to answer. Methods range from removing formatting noise and selecting the most relevant passages to dropping individual low-information tokens with a small model such as LLMLingua.
What is LLMLingua?
LLMLingua is an open-source prompt compression library from Microsoft Research. A small language model scores each token and drops the predictable ones. LLMLingua-2, a faster task-agnostic version, uses a BERT-size classifier. Install it with pip install llmlingua.
How much can prompt compression save?
It depends on the input. Repetitive machine output such as logs and test runs can shrink by over 90%. Retrieved documents typically shrink by half or more with extractive selection. LLMLingua's authors report up to 20× on their benchmarks. Measure on your own prompts.
Does prompt compression reduce answer quality?
It can. Token-level methods can break exact strings, code and JSON, and any method can remove a fact the answer needed. Test on a labelled set of real questions, compare with the uncompressed prompt, and give the model a way to retrieve what was removed.
Is prompt compression better than prompt caching?
They solve different parts of the prompt. Caching discounts a fixed prefix by up to 90 to 95%, but only if it is identical across requests. Compression removes tokens from content that changes. Cache the fixed part and compress the variable part; compressing a cacheable prefix on every request costs more than caching it.
Does prompt compression reduce latency?
Usually, because the model processes fewer input tokens before the first output token. The compression step adds its own time, though: near zero for structural trimming, milliseconds for BM25 or embedding selection, more for a model-based compressor like LLMLingua. Measure end to end.
Keep reading

LLM Cost Optimization: 8 Levers That Cut 50–90%
LLM cost optimization in 8 levers with 2026 prices: cut input context, prompt caching (90% off), batching (50% off), routing and output caps. Worked examples.

RAG Context Compression vs Reranking
RAG context compression removes redundant, off-topic and low-value chunks before the model call. How it differs from retrieval and reranking, with math.

Anthropic Prompt Caching (Claude): What It Costs
Anthropic prompt caching bills a repeated prefix at 10% of the input price or less. How cache_control works, when it pays off, and what silently breaks it.

Agent Context Optimization: Stop Token Bloat
Agent context grows every step. Where waste comes from (tool output, memory, stale state) and patterns that cut input tokens without hurting task success.