Contextwaise
Prompt compression

Prompt compression: send the model fewer tokens without losing the answer

Prompt compression reduces the number of tokens in a prompt before it reaches the model, by removing noise, keeping only the relevant passages, or dropping low-information tokens with a small model such as LLMLingua. It lowers the input bill and the latency of every call, and it often improves answers on long inputs. The catch most guides skip: compress the part of the prompt that changes on every request, and cache the part that doesn't, because compressing a cacheable prefix can cost more than caching it.

Updated November 16, 202612 min readBy the Contextwaise team
Key takeaways
  • There are five families of prompt compression: structural trimming, extractive selection, token-level pruning (LLMLingua), abstractive summaries, and soft prompts. Only the first four work with closed APIs.
  • Structural trimming and extractive selection keep the text verbatim, so code, IDs, numbers and citations survive. Start there.
  • LLMLingua-2 drops individual tokens and reaches high ratios, but its output is telegraphic text. Protect what must stay exact with force_tokens, and keep it away from JSON and code.
  • Compression and prompt caching compete for the same tokens. On Claude Sonnet 5.5 a 10,000-token prefix costs $0.001 per request when cached and $0.0067 when compressed 3× on every request. Cache the fixed prefix, compress the variable part.
  • Measure compression on your own labelled questions. A smaller prompt that answers wrong is not a saving.

What prompt compression is

Prompt compression is any step that makes a prompt shorter before the model reads it while keeping what the model needs to answer. The input can be retrieved documents in a RAG app, tool output in an agent, a long chat history, or a pasted log. The output is a smaller prompt sent to the same model.

It matters for three reasons. Input tokens are billed one by one, and in most production apps the input is many times larger than the output. Time to first token grows with input length. And models use long inputs less reliably than short ones, an effect known as context rot, so the tokens you remove were often hurting the answer as well as the bill.

What it is not: rewriting your instructions to be shorter is good hygiene, but it saves a few hundred tokens once. The volume is in the data you put in the prompt on every call, and that is where compression pays.

The five prompt compression methods, compared

The research splits methods into hard prompts, which produce text, and soft prompts, which produce learned vectors (Li et al., 2024 survey). For an app calling a hosted API, the practical split is how much of the original text survives.

MethodWhat it doesText stays verbatimExtra costBest for
Structural trimmingRemoves formatting noise: progress bars, repeated log lines, empty and null JSON fields, whitespace, boilerplateYesNear zero, deterministic codeTool output, logs, API responses
Extractive selectionScores sentences, chunks or messages for relevance and keeps the best ones within a token budgetYes, for what is keptA fast scoring step (BM25, embeddings, a reranker)RAG context, chat history, long documents
Token-level pruningA small model drops individual low-information tokens (LLMLingua, LLMLingua-2)No, telegraphicA local model pass, GPU helpsLong prose where ratio matters more than exact wording
Abstractive summaryAn LLM rewrites the input as a shorter summaryNo, paraphrasedA full model call, and it can invent detailsOld turns of a long conversation
Soft promptsEncodes the prompt into learned vectors (gist tokens, embeddings)No, not text at allTraining, access to model internalsSelf-hosted models only

The order of the table is the order to try them in. Each step down gains ratio and loses control over what the model sees.

Start with structural trimming and extractive selection

Most of the waste in a production prompt is not subtle. A test run prints hundreds of passing tests to report two failures. An API response carries dozens of empty fields. Ten retrieved chunks repeat the same paragraph from three versions of a document. None of that needs a model to detect.

Python
import json, re

def trim_json(value):
    """Drop null and empty fields; keep 0 and False, which carry meaning."""
    if isinstance(value, dict):
        kept = {k: trim_json(v) for k, v in value.items()}
        return {k: v for k, v in kept.items() if v not in (None, "", [], {})}
    if isinstance(value, list):
        return [trim_json(v) for v in value]
    return value

def trim_log(text):
    """Keep failures and their context; collapse progress bars and passing lines into a count."""
    lines = text.splitlines()
    passed = sum(1 for l in lines if re.search(r"\bPASS(ED)?\b", l))
    kept = [l for l in lines if not re.search(r"\bPASS(ED)?\b|\d+%\|", l)]
    return f"[{passed} passing lines removed]\n" + "\n".join(kept)

compact = json.dumps(trim_json(api_response), separators=(",", ":"))
Structural trimming: deterministic, no model, and nothing the model needs is rewritten. Say what you removed, so the model doesn't assume it is missing.

Extractive selection is the next step: score each unit (a chunk, a sentence, a message) against the current question and keep the best ones until a token budget is full. BM25 is fast and good with exact terms such as error codes; embeddings catch paraphrases; a cross-encoder reranker is the most accurate and the slowest. The kept text is the original text, so citations and code still match their source. For retrieval pipelines in depth, see RAG context compression.

LiteLLM's compress() is a ready-made extractive compressor for chat history. It scores messages with BM25 (plus embeddings if you configure a model), always keeps the system message and the last user and assistant messages, and replaces the rest with short stubs. It also returns a litellm_content_retrieve tool, so the model can fetch a dropped message back. By default it only runs once a conversation passes 200,000 tokens, so set compression_trigger lower if your costs start earlier.

LLMLingua and LLMLingua-2: token-level compression

LLMLingua is Microsoft Research's open-source prompt compression library (MIT license). It uses a small language model to decide which tokens of a prompt carry little information and drops them. The family has three main versions:

VersionPublishedHow it decidesClaim in the README
LLMLinguaEMNLP 2023Perplexity from a small causal model (GPT-2 small, LLaMA-7B): predictable tokens are droppedUp to 20× compression with minimal performance loss
LongLLMLinguaACL 2024Question-aware: keeps what is relevant to the question, reorders documents against lost in the middleUp to 21.4% better RAG performance using 1/4 of the tokens
LLMLingua-2ACL 2024 FindingsA BERT-size token classifier trained on GPT-4 compressions; task-agnostic3× to 6× faster than LLMLingua

Those are the authors' benchmark numbers. Your ratio depends on how redundant your text is, so measure it on your own prompts before you plan around 20×.

Python
# pip install llmlingua
from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
    use_llmlingua2=True,
)

result = compressor.compress_prompt(
    retrieved_context,          # compress the data, not your instructions
    rate=0.33,                  # keep about a third of the tokens
    force_tokens=["\n", "?"],   # tokens that must always survive
)

print(result["origin_tokens"], "->", result["compressed_tokens"], result["ratio"])
prompt = f"{instructions}\n\n{result['compressed_prompt']}\n\nQuestion: {question}"
LLMLingua-2 with the model and arguments from the README. The result also includes `saving`, an estimate in dollars.

Where token-level compression breaks

  • Exact strings. Dropping tokens inside an ID, a version number, a URL or a function name changes its meaning. Keep those out of the compressor or protect them with force_tokens.
  • Code and JSON. The output is no longer valid syntax. Compress the prose around code, never the code.
  • Instructions. A telegraphic system prompt is easier to misread. Instructions are short and fixed, so cache them instead.
  • Latency on CPU. The compressor is a model. LLMLingua-2 is the light one, but budget for the extra step and run it on a GPU at volume.

Prompt compression vs prompt caching: which to use

Both lower the cost of input tokens, and they work on different tokens. Prompt caching discounts a prefix that is identical across requests: on Claude Sonnet 5.5 a cached input token costs $0.10 per million instead of $2, a 95% discount. Compression removes tokens, but whatever is left is billed at full price, and if the compressed text changes from one request to the next, the cache never matches.

Take a support bot that sends the same 10,000-token product manual with every question, 100,000 questions a month, on Claude Sonnet 5.5. Cache writes are left out to keep the comparison simple.

The 10,000-token manual is…Input tokens billedCost per requestPer month
Sent as is, no cache10,000 at $2/M$0.020$2,000
Compressed 3× per question (question-aware, so never cached)3,333 at $2/M$0.0067$667
Cached10,000 at $0.10/M$0.001$100
Compressed once, offline, then cached3,333 at $0.10/M$0.00033$33

Compressing the manual on every request saves two thirds of the bill, but caching it untouched saves 95%. The rule that falls out:

  • Fixed content (instructions, tool definitions, a manual, few-shot examples): put it first and cache it. If you compress it, do it once offline and cache the result.
  • Variable content (retrieved chunks, tool output, the newest messages): compress it on every request. It can't be cached anyway, so every token removed is billed at nothing instead of the full price.
  • Work nobody waits on (evals, backfills, labelling): send it through a batch API for 50% off every token, then compress and cache on top.
  • Never compress inside the cached prefix at request time. One changed token at the start invalidates everything after it.

How to measure whether compression hurt the answers

A compression ratio on its own says nothing about quality. Measure both on the same labelled set before you ship, and keep measuring after.

  1. Collect 50 to 200 real requests with a known good answer, including the hard ones (multi-hop questions, exact numbers, rare terms).
  2. Run each one twice, uncompressed and compressed, with the same model, temperature 0 and the same instructions.
  3. Score the answers: exact match for facts and numbers, an LLM judge for open answers, and citation checks for RAG.
  4. Log input tokens and end-to-end latency for both runs, including the compression step itself.
  5. Pick the most aggressive setting whose accuracy stays within your tolerance, often within one or two points of the baseline, and re-run the set when the model or the data changes.

Keep a way back. When the model can ask for what was removed, through a retrieval tool as in LiteLLM's compress(), a bad cut costs one extra call instead of a wrong answer. Count how often that tool is called: if it fires on most requests, you are compressing too hard.

What compression saves on real inputs

Ratios depend almost entirely on how noisy the input is. These are measured on our own demo agent and example requests, not on customer workloads, with structural trimming and extractive selection only:

InputBeforeAfterReduction
Test run output (2 failures among passing tests and progress bars)7,209 tokens156 tokens−98%
Application logs28,309 tokens564 tokens−98%
Support RAG request (13 retrieved items → 3)901 tokens264 tokens−71%

Machine output compresses the most, because most of it is repetition. Prose written by people compresses the least, and that's where token-level methods like LLMLingua earn their place. To size your own case, paste a real prompt into the token counter and price it per month.

A prompt compression checklist

  1. Log the token count of every part of your prompt for one day and sort by size. Compress the biggest variable part first.
  2. Move fixed content to the start and turn on prompt caching before you compress anything.
  3. Add structural trimming for tool output and logs. It's code, it's deterministic and it can't paraphrase.
  4. Add extractive selection for retrieved context and long histories, with a token budget.
  5. Try LLMLingua-2 only on long prose that is still too big, with force_tokens for anything exact.
  6. Run the labelled set, compare accuracy and cost, and keep a retrieval path for what was removed.

Frequently asked questions

What is prompt compression?

Prompt compression is the step that shortens a prompt before the model reads it, keeping what the model needs to answer. Methods range from removing formatting noise and selecting the most relevant passages to dropping individual low-information tokens with a small model such as LLMLingua.

What is LLMLingua?

LLMLingua is an open-source prompt compression library from Microsoft Research. A small language model scores each token and drops the predictable ones. LLMLingua-2, a faster task-agnostic version, uses a BERT-size classifier. Install it with pip install llmlingua.

How much can prompt compression save?

It depends on the input. Repetitive machine output such as logs and test runs can shrink by over 90%. Retrieved documents typically shrink by half or more with extractive selection. LLMLingua's authors report up to 20× on their benchmarks. Measure on your own prompts.

Does prompt compression reduce answer quality?

It can. Token-level methods can break exact strings, code and JSON, and any method can remove a fact the answer needed. Test on a labelled set of real questions, compare with the uncompressed prompt, and give the model a way to retrieve what was removed.

Is prompt compression better than prompt caching?

They solve different parts of the prompt. Caching discounts a fixed prefix by up to 90 to 95%, but only if it is identical across requests. Compression removes tokens from content that changes. Cache the fixed part and compress the variable part; compressing a cacheable prefix on every request costs more than caching it.

Does prompt compression reduce latency?

Usually, because the model processes fewer input tokens before the first output token. The compression step adds its own time, though: near zero for structural trimming, milliseconds for BM25 or embedding selection, more for a model-based compressor like LLMLingua. Measure end to end.

Keep reading

Cut your LLM costs
without cutting quality.

One layer between retrieval and inference removes the context your model never needed, in milliseconds. Fewer input tokens, lower cost, a prompt cache that keeps hitting, and every removal logged with its reason.

See it live
Keep your model, retriever and prompts. Remove one call to roll back.