ai-architecture
ranked
11 Sept 2026
Eight ways to count tokens before you send them, ranked by how far each drifts
Measured across 1.17 million characters of prose, TypeScript, lockfile JSON and four languages: where each pre-flight count lands against the tokenizer that actually bills you.
By Purvansh Parmar
Checked 2026-09-11
Read 7 min
Entries 8
You are about to ship a chunker, and something has to decide whether a document fits. The cheap answer is length divided by four. Anthropic's own pricing FAQ suggests it. I ran eight pre-flight counting methods over 1.17 million characters of real material, English prose and TypeScript and a 686 KB lockfile and Wikipedia in four languages, and the divide-by-four rule turned out to be the safest thing on this list for English and the most dangerous thing on it for JSON. The gap is not small. On the lockfile it under-counts by 36 percent.
How this was checkedI built a 1.17 million character corpus out of text already on this machine: hand-written English technical prose, a project's TypeScript, a 686 KB package-lock.json, 400 lines of git log, and parallel Wikipedia articles on photosynthesis in English, German, Hindi and Japanese. Then I counted all of it eleven different ways on 2026-09-11. gpt-tokenizer, js-tiktoken and @dqbd/tiktoken returned identical counts on all nine files. Every other pair in the set disagreed, and two of them disagreed by more than 2x.
What I compared on
- Drift against the tokenizer the model actually bills under, measured per content type
- Throughput on a 261 KB mixed payload, averaged over three runs after a warm-up
- Cold import time, because on serverless that lands on a real request
- Whether it needs a network call before it can answer
A network call that returns the exact input token count for the model ID you pass it.
Wins
Closed models. No Claude tokenizer is published, so this is the only number that matches the invoice. It is free on Anthropic, rate limited to 5,000 requests per minute on the Start tier and 20,000 on Scale, and those limits are tracked separately from message creation.
Loses
It costs a round trip, so it cannot sit inside a chunking loop that runs a thousand times per document. The docs are also careful to call the result an estimate that may differ from the billed count by a small amount, which means it is exact enough to budget with and not exact enough to assert on.
5,000 RPM on Start20,000 RPM on Scale$0 per call
Pure JavaScript BPE with the OpenAI encodings compiled in, no WASM step and no download.
Wins
OpenAI models, counted in process. It chewed through my 261 KB mixed payload at 8.35 MB/s, roughly four times js-tiktoken, and it was also the quickest to load at 187 ms cold.
Loses
It knows o200k_base and cl100k_base and nothing else. Point it at a Llama or Mistral deployment and you get a confident wrong answer with no warning. 27.2 MB unpacked is a lot of disk for a function that returns an integer.
8.35 MB/s187 ms cold import27.2 MB unpacked
WASM bindings over OpenAI's own Rust tiktoken, same encodings behind a different runtime.
Wins
You would rather run the upstream implementation than a reimplementation of it. Its counts matched gpt-tokenizer on all nine of my corpora, down to the token, and it imported in 213 ms.
Loses
2.99 MB/s, about a third of gpt-tokenizer's throughput for an identical answer. The WASM blob also has to survive whatever bundler you are using, and the encoder holds memory until you call free() on it.
2.99 MB/s213 ms cold import23.6 MB unpacked
The pure JavaScript sibling of tiktoken, built for runtimes that will not load a WASM binary.
Wins
Edge runtimes, and anywhere the WASM path is closed to you. It agreed with the other two on every file I threw at it, so swapping it in when a bundler rejects the binary changes your numbers by exactly zero.
Loses
Slowest of the three OpenAI options at 2.01 MB/s, and by far the worst cold start at 673 ms. On a serverless function that 673 ms is paid by a user, on the first request after every scale-up.
2.01 MB/s673 ms cold import22.4 MB unpacked
Pulls the real tokenizer.json for any open-weight model off the Hub and runs it locally.
Wins
Open-weight serving, where the exact tokenizer your server uses is downloadable. I loaded Llama 3, Gemma 2, Qwen2.5, Mistral and DeepSeek V3, and the spread between them was wider than I expected: Mistral billed 27 percent more tokens for the same TypeScript than Llama 3 did, and 24 percent more for the same JSON.
Loses
First use fetches the tokenizer over the network and caches it, so a cold container makes a Hub round trip before it can count a single character. At 1.50 MB/s it is the slowest exact option here, and 735 ms to initialise even with the cache already warm.
1.50 MB/s735 ms warm cache27% Mistral against Llama 3
06 A different model's tokenizer
Counting with cl100k_base because the old code did, then serving a model that uses o200k_base.
Wins
Same family, same generation. cl100k_base and o200k_base land within 3 percent of each other on English prose and on minified JSON, so if your corpus genuinely is only those two things, the stale counter will not hurt you this quarter.
Loses
Everything else. Hindi counted with cl100k_base comes to 18,723 tokens where o200k_base says 6,462, so a budget built on the older encoding is 2.9 times too large and every routing decision downstream of it is wrong. German drifts 15 percent, Japanese 30 percent. Anthropic shipped the same trap in the other direction: models from Claude 4.7 onward use a newer tokenizer that produces about 30 percent more tokens for identical text.
2.9x on Hindi15% on German30% on Japanese
07 Length divided by four
One line, zero dependencies, and the rule of thumb the vendor documentation itself quotes.
Wins
English prose, where it fails safe. Over 78 KB of hand-written English it predicted 19,468 tokens against a real 16,912, over-booking by 15 percent, which keeps you under the ceiling rather than through it. As a rough context guard on chat turns that is a defensible choice.
Loses
Structured data, where it fails the other way. On a 686 KB package-lock.json it predicted 171,537 tokens against 267,587 real ones, short by 36 percent. Japanese came out short by 53 percent and a file of git hashes by 42 percent. Density on one encoding ran from 4.81 characters per token for English Wikipedia down to 1.87 for Japanese, so the constant is wrong by construction on anything mixed.
+15% on English prose-36% on lockfile JSON-53% on Japanese
08 Word count times 1.33
Split on whitespace and multiply by a constant somebody measured on English once, years ago.
Wins
English prose, and strictly nothing else. It came within 8 percent on hand-written English, which is the one corpus where splitting on whitespace corresponds to anything a tokenizer does.
Loses
It is not measuring text. It is measuring spaces. Minified JSON has almost none, so it predicted 47,270 tokens against 267,587, low by 82 percent. Japanese puts no spaces between words at all and it came in low by 93 percent. Even TypeScript, which is full of whitespace, was low by 51 percent.
-82% on lockfile JSON-93% on Japanese-51% on TypeScript
Side by side
| Exact for | Throughput | Worst drift measured |
|---|
| count-tokens endpoint | the model you name | one round trip | vendor calls it an estimate |
| gpt-tokenizer | OpenAI encodings | 8.35 MB/s | 0% in family, unbounded out |
| @dqbd/tiktoken | OpenAI encodings | 2.99 MB/s | 0% in family, unbounded out |
| js-tiktoken | OpenAI encodings | 2.01 MB/s | 0% in family, unbounded out |
| HF AutoTokenizer | open-weight models | 1.50 MB/s | 0%, needs a Hub fetch |
| another model's tokenizer | nothing | varies | 190% on Hindi |
| length / 4 | nothing | instant | 53% on Japanese |
| words x 1.33 | nothing | 103 MB/s | 93% on Japanese |
Verdict
Use gpt-tokenizer if you serve OpenAI models, AutoTokenizer if you serve open weights, and the count-tokens endpoint if you serve Claude, because for Claude there is no local option at all. Length over four keeps one job. It is a cheap guard on English chat turns, where over-booking is the safe direction to be wrong in. The moment your input is JSON, source code, or anything outside Latin script, that heuristic stops being an estimate and becomes a bug with a 36 percent head start.
Sources