About GPT-2 and Davinci Token Counter
This page opens on r50k_base, used by the first GPT-3 models (davinci, curie, babbage, ada). GPT-2 used a very similar byte-pair encoding, so the counts are close for it, but this is not the GPT-2 tokenizer itself. It is free, needs no sign-up and runs in your browser, so nothing you enter is uploaded.
The example mixes several scripts and an emoji, which shows how the older vocabulary breaks other languages into many small pieces.
How to use GPT-2 and Davinci Token Counter
- Paste your prompt or text into the box, or tap a sample.
- Choose the encoding. o200k_base is for GPT-4o and newer, cl100k_base for GPT-4 and GPT-3.5 Turbo.
- Read the exact token count, characters, words and characters per token.
- Look at the coloured tokens below. Turn on Show token ids to see the numbers.
- To see an estimate for Claude, Gemini or Llama, pick the model in the estimate panel. It is labelled as an estimate.
- To work out a cost, type your price per 1 million tokens from your provider's pricing page.
Features
- Exact token counts with o200k_base, cl100k_base, p50k_base and r50k_base.
- Coloured token view, with the id of each token on hover and a switch to list all ids.
- Characters, words and characters-per-token ratio.
- Estimates for Claude, Gemini and Llama, labelled as estimates with the ratio shown.
- Cost calculator that uses the price you enter, so it never goes out of date.
- Handles other scripts and emoji, and groups tokens that split one character.
- The tokenizer loads on demand, so the page itself stays light.
Worked examples
hello world gives 2 tokens
আমি ভাত খাই। gives 22 tokens
Privacy: how it works
- Counting happens in your browser. Your text is not sent to Ulkaa or to any provider, and no model file is downloaded.
Tips
- Use the encoding that matches your model. The wrong one can be off by a large margin on non-English text.
- Trailing spaces and line breaks are tokens too. Trim a prompt before counting if you will trim it before sending.
- Keep a margin below a context limit, because the answer also uses tokens.
- A rule of thumb for English is about four characters or three quarters of a word per token, but check with the real count.
Frequently asked questions
Is it really free?
Yes. There is no price, no trial and no credit card. Ulkaa Tool does the work with your own device, so there is no server bill to pass on to you.
Do I need an account?
No. There is no sign-up and no login. Open the page and use it. An Ulkaa account is not needed for anything on this tool.
Does it run in my browser?
Yes. Your text are processed by code running in your browser tab, on your own device.
Is my data uploaded?
No. Your text are never sent to Ulkaa or to anyone else. You can check this yourself by watching the Network tab in your browser developer tools while you use the tool.
Is r50k_base the same as the GPT-2 tokenizer?
They are very close but not identical, so treat counts for GPT-2 as near-exact. For GPT-3 davinci and its siblings, r50k_base is the encoding.
More Token Counter pages
- Token CounterPaste a prompt to get an exact token count for OpenAI models, with every token shown.
- GPT-4o Token CounterAn exact token count for GPT-4o prompts, using the o200k_base encoding.
- ChatGPT Token CounterPaste a conversation and see how many tokens it uses, with cl100k_base.
- GPT-3 Token CounterToken counts for text-davinci-003 and Codex, using the p50k_base encoding.
- Claude Token CounterA labelled estimate of Claude tokens, next to an exact OpenAI count for comparison.
- Gemini Token CounterA labelled estimate of Gemini tokens for mixed-script text.
- Llama Token CounterA labelled estimate of Llama tokens for code and prose.