Guides

Conversations and prompt caching

In a conversation, what you already sent matters as much as what you send next. Here is how to save tokens without losing your provider's prompt cache.

Why conversations are different#

LLM APIs are stateless: every turn resends the whole conversation. Providers make that affordable with prompt caching: when a request starts with exactly the same bytes as an earlier one, that start is billed at a fraction of the price, often a tenth. For Claude's rules and prices, see the guide to Anthropic prompt caching.

The cache only works while the beginning of the conversation is byte-identical. If an earlier turn changes, even one chunk dropped or added, everything after that point is billed in full again. On newer Claude models, editing earlier turns also invalidates the model's saved reasoning.

Don't re-optimize the history

Passing the whole conversation through any context filter on every turn changes earlier turns, and the cache misses. A filter that saves 40% of tokens can end up costing more than it saves.

The rule: new content, once#

Optimize each piece of context once, when it first enters the conversation: the chunks retrieved for this turn, the tool result that just returned.
Store the optimized version in your history, and resend it unchanged on later turns.
Never pass the conversation's own messages (user questions, assistant answers) as candidates.

The recommended pattern#

chat.ts

// Turn N: only the chunks retrieved for this question go through Contextwaise.
const chunks = await vectorStore.search(question);
const context = await optimizeContext(question, toCandidates(chunks), "support-chat", MODEL);

// Store the optimized version. From now on it is part of the history.
messages.push({ role: "user", content: render(question, context) });

// Earlier messages are resent exactly as stored: never passed through Contextwaise again.
const reply = await llm.chat({ model: MODEL, system: SYSTEM_PROMPT, messages });
messages.push({ role: "assistant", content: reply.text });

Each turn, only new content is scored, so requests stay small and fast, and the conversation only ever grows at the end: exactly what prompt caching needs.

Cache-safe mode#

Some applications can't follow the pattern above: they rebuild the context from the original sources on every turn, and the history isn't stored as sent. For them, pass a stable conversation id:

Request

{
  "query": "When will I get the duplicate charge refunded?",
  "candidates": [ /* every item you send this turn, old and new */ ],
  "agent": "support-chat",
  "conversation": "chat_8f2k"
}

How it behaves#

Every decision is remembered for that conversation. An item seen before gets its first decision again: it is not re-scored against the new question.
Reused items are marked "reused": true in report.items, and don't count against budget, which applies to new items.
A new item that repeats something already sent is dropped as duplicate.
report.conversation returns { id, reused, protected }. protected: false means decisions couldn't be stored for this request, so it isn't guaranteed to match earlier ones.
A conversation's decisions are kept for 7 days after its last request.

The result: each turn's context starts with the previous turn's context, byte for byte, and only grows at the end. The live demo shows this turn by turn, against plain trimming.

When to use it#

Your app…	Do this
Stores the context it sent, and resends history as is	Optimize new content once. No conversation id needed.
Rebuilds all context from sources on every turn	Send everything with a stable `conversation` id.
Uses model routing in a chat	Pass a `conversation` id so the model is chosen once per conversation.

Use one id per conversation, made of letters, digits, _ - . : (up to 128 characters), for example your chat or thread id.

Tools your provider runs#

Tools executed by the provider itself, such as hosted web search or web fetch, never pass through your code, so Contextwaise can't trim them. Cap them with the provider's own settings instead (maximum uses, maximum content tokens, allowed domains). Tools you run are covered by the API or the Agent SDK.