Context rot: why LLMs get worse as the prompt gets longer

What context rot is

A context window is a hard limit: the most tokens a model will accept. Context rot is about what happens well inside that limit. As input grows, the model's attention is spread over more tokens, and its ability to find, connect and follow the right ones gets worse. Nothing errors. The request succeeds, the answer reads fluently, and it is a little less likely to be right.

The term was popularised by Chroma's 2025 report, which showed that models do not use their context uniformly: the same fact, asked the same way, is found less reliably at 100K tokens than at 1K. Three related problems are often mixed up with it.

The practical consequence: a model advertised with a 1M-token window does not give you 1M tokens of equally good attention. The window says what you are allowed to send, not what the model will use well.

What the research shows

Chroma, 2025: 18 models, all of them degrade

Chroma tested 18 models from Anthropic, OpenAI, Google and Alibaba on deliberately simple tasks, varying only the input length. The point of simple tasks was to rule out the excuse that long inputs are just harder problems.

• Less obvious matches fail first. When the question and the answer share few words (low semantic similarity), accuracy fell much faster as input grew. Real questions are usually like this: users do not phrase them the way your documents are written.
• Distractors compound the loss. Adding one passage that looked relevant but did not answer the question lowered accuracy; four lowered it further. Models differed in how they failed: Claude models tended to abstain, GPT models tended to answer confidently with the distractor.
• Focused beats full. On 306 LongMemEval questions, models scored markedly better given only the relevant ~300 tokens than given the full ~113K-token conversation history that also contained them.
• Coherent text hurt more than shuffled text. Across all 18 models, a haystack of randomly shuffled sentences was searched more accurately than the same text in its logical order. Structure that reads naturally can pull attention away from the one sentence that matters.
• Even copying breaks. Asked to repeat a long run of a common word with one different word inserted, models made more mistakes as length grew: wrong positions, wrong lengths, random output.

Lost in the middle, 2023: position matters

Liu et al. gave models 20 to 30 retrieved documents with the answer in one of them, and moved that document around. Accuracy followed a U-shape: highest when the answer was first or last, lowest in the middle. In some settings, a model with the answer buried mid-context did worse than the same model with no documents at all. Chroma's needle tests found position mattered less in its setup, which is a useful reminder that both effects exist and that your prompt shape decides which one bites.

Agents, 2026: giving up before the window is full

Xia et al. studied search agents on long-horizon tasks across four flagship models. As the agent's context filled with search results, models increasingly gave up or returned uncertain wrong answers long before running out of window, including on questions they could solve. Context management that kept peak context lower redirected the models back to searching. For agents, context rot shows up less as wrong facts and more as lost persistence.

Why long context degrades

There is no single accepted cause, but three mechanisms explain most of what the studies observe.

• Attention is a budget. Each token's attention weights sum to one across everything in context. More tokens means each relevant token competes with more irrelevant ones, and weak signals get diluted. Anthropic's context engineering post describes context as a finite resource with diminishing returns for exactly this reason.
• Similar-looking text competes. The model retrieves by resemblance. A passage that shares vocabulary with the question but answers a different one (last year's price, the wrong customer, a deprecated API) is a better match on surface features than the right answer phrased differently.
• Long examples are rarer in training. Models see far more short sequences than long ones, and long-range positions are learned from less data. Stretching the window does not stretch the experience at the same rate.

All three point the same way: the cost of an irrelevant token is not zero. It takes attention from relevant ones, and the more it resembles the answer, the more it takes.

Where context rot shows up in production

Benchmarks build long contexts on purpose. Production systems build them by accident, one reasonable decision at a time.

The common thread is that context is added because it might help and rarely removed because nobody checks whether it did. Each addition looks harmless on its own. Together they push every request further down the degradation curve, and the cost goes up at the same time: the same unneeded tokens that dilute attention are billed on every call. Paste a real request into the token counter to see how much of it is history, tool output and retrieved text.

How to measure context rot in your own app

Published benchmarks tell you context rot exists; they do not tell you where your prompt sits on the curve. The cheapest test copies Chroma's LongMemEval setup: take questions you already know the answers to, and run each one twice, once with only the passages that answer it and once with the full context your app would really send.

# For each labelled case: the question, the passages that answer it,
# and the full context your app actually builds for it.
def accuracy(cases, build_context):
    correct = 0
    for case in cases:
        answer = ask_model(build_context(case), case["question"])
        correct += grade(answer, case["expected"])  # exact match, or an LLM judge
    return correct / len(cases)

focused = accuracy(cases, lambda c: c["answer_passages"])
full = accuracy(cases, lambda c: c["full_context"])
print(f"focused {focused:.0%}  full {full:.0%}  gap {focused - full:+.0%}")

• A gap of a few points or more means the extra context is costing you accuracy, not just money. That is the signal to cut.
• Plot accuracy against input tokens. Bucket production-like cases by context size and look for where the curve bends. That is your practical budget, whatever the model's window says.
• Add distractors on purpose. Insert a passage that resembles the answer but is wrong (an old price, a different customer) and see how often the model uses it. That is the failure your users will meet.
• For agents, track steps, not just answers. Rising step counts, repeated tool calls and early give-ups on long runs are context rot even when the final answer is fine.

How to prevent context rot

Every fix comes down to the same idea: give the model the smallest set of tokens that answers the question, arranged so the important ones are easy to find. In rough order of impact:

1. Retrieve wide, send narrow. Keep top-k high so recall is not the bottleneck, then rerank and apply a relevance threshold so only passages that earn their place reach the prompt. The full method is in RAG context compression.
2. Remove duplicates and near-duplicates. Overlapping chunks and copies of the same page are pure distractors: they repeat a fact without adding one, and they pull attention toward whichever version is stale.
3. Clear stale tool output. Once an agent has acted on a file listing or a test log, the raw output rarely needs to stay. Replace it with a one-line result, or drop it. See agent context optimization.
4. Compact long sessions. Summarise older turns into the decisions and facts still in force, and start fresh when the task changes. Coding agents do this with commands like /compact and /clear.
5. Split work across clean contexts. A sub-agent that searches with its own context and returns a short answer keeps the main context small. Each context stays near the top of the curve.
6. Put the question and key instructions at the edges. Start or end of the prompt, not between two long documents. For long documents, ask the question after them.
7. Set a token budget per request. A ceiling turns "add it, it might help" into a choice about what to drop. Measure where your accuracy bends and set the budget below it.
8. Let the model abstain. Tell it to say when the context does not contain the answer. An honest "not found" is easier to catch than a confident answer built on a distractor.

Will bigger context windows fix it?

Not by themselves. Windows have grown from 4K to over 1M tokens, and every model in Chroma's study, including the long-context ones, still degraded with length. A larger window raises the ceiling on what you can send; it does not flatten the curve below it. Newer models do degrade more slowly, so the bend moves right, but the shape stays.

There is also the bill. Long inputs cost more per call, and the big three providers also charge a higher rate per token past a threshold: Gemini 3.1 Pro doubles its input price above 200K tokens, GPT-6.1 Sol above 272K, and Claude Haiku 5.5 bills input at 5× above 100K (see Gemini API pricing, Claude API pricing and the LLM pricing tool). Sending less is the one change that improves accuracy and cost at once. The rest of the discipline is in the context engineering guide.

Frequently asked questions