Token counter

What is a token?

A token is the unit a language model reads and bills: a word, part of a word, a punctuation mark or a space. In English, a token averages about 4 characters, or three quarters of a word, on OpenAI and Gemini models.

Models never see letters. A tokenizer cuts the text into pieces from a fixed vocabulary, and the model works on those pieces. Common words are one token; rare words, names and numbers are split into several.

Code, JSON, URLs and languages other than English use more tokens per word, because the vocabulary has fewer ready-made pieces for them.

Why Claude, GPT and Gemini count differently

Each provider trains its own tokenizer, so the same text is a different number of tokens on each model. Compare prices per token only after counting on that model.

• OpenAI publishes its tokenizers. This counter runs o200k_base, the one behind GPT-4o and GPT-5. GPT-6's has not been published, so the count is very close rather than guaranteed.
• Claude's tokenizer is not public. Anthropic says 1M tokens is about 2.5M characters on the tokenizer it introduced with Claude Opus 4.7, which uses noticeably more tokens than older Claude models for the same text.
• Gemini: Google says a token is about 4 characters.
• DeepSeek: about 0.3 tokens per English character and 0.6 per Chinese character.

Count tokens in your code

For exact numbers in production, each provider has an endpoint that counts a request the way the model will see it, including the formatting around messages and tools.

Claude, with the Anthropic SDK:

const { input_tokens } = await anthropic.messages.countTokens({
  model: "claude-opus-5-5",
  messages: [{ role: "user", content: prompt }],
});

OpenAI, with the input token count endpoint:

curl https://api.openai.com/v1/responses/input_tokens \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-6.1-sol", "input": "Your prompt"}'

Or offline in Python with tiktoken:

import tiktoken
tokens = len(tiktoken.get_encoding("o200k_base").encode(text))

Gemini, with the Google Gen AI SDK:

const { totalTokens } = await ai.models.countTokens({
  model: "gemini-3.1-pro-preview",
  contents: prompt,
});

What a prompt really costs

A token count turns into a bill three ways, and the counter above shows all three.

1. Per call: tokens × the input price. A 20,000-token prompt on a $2 per 1M model is 4 cents.
2. At volume: the same 4 cents at 1,000 calls a day is $1,200 a month.
3. In an agent: each step resends everything before it, so the same context is billed again and again. Set the agent steps above to 20 and watch the monthly figure.

Prompt caching softens the third: after the first call, the repeated prefix is billed at 3 to 10% of the input price, depending on the model. The prompt caching guide covers what breaks a cache, and the LLM cost guide covers the rest of the bill.

How to use fewer tokens

In most production prompts, the instructions are small and the context is large: retrieved documents, search results, tool output and chat history. That is where the tokens are, and most of it is never used by the answer.

• Send fewer, better retrieved chunks instead of the top 20.
• Trim tool output before it enters an agent's history, since it is resent on every later step.
• Keep the stable part of the prompt first and unchanged, so the cache keeps hitting.