devtools
guide
18 Oct 2026
Five Python type checkers against ten planted bugs, and the one that found none
Same file, same ten errors, default settings. Two found all ten, one found zero, and the zero is not a bug in the tool. It is the preset it ships with.
By Purvansh Parmar
Checked 2026-10-21
Read 8 min
Entries 5
Ten type errors in one small module, none of them the obvious kind. No integer added to a string. Nothing a reviewer catches in a second: a dataclass attribute that does not exist, a function that falls out of its only branch, a coroutine passed where the value was wanted, a dict returned from something annotated as returning a list. Then five checkers at their out of the box settings, because that is how almost everyone runs them. The spread was not what I expected: the quickest tool was also the most accurate, and the one that found nothing found nothing on purpose.
How this was checkedPlanted ten type errors in a 91 line inventory module, the kind that survive review rather than the kind you spot instantly, then stripped every marker from the file the checkers actually read. Ran five checkers at their defaults. ty and mypy found all ten with no false positives, in 89 and 240 milliseconds. pyrefly found zero, because its default preset is basic, and the same file under its strict preset produced ten errors.
What I compared on
- how many of the ten planted bugs are reported at default settings
- false positives, counted as diagnostics on lines with nothing planted
- wall clock on a single file
Astral's Rust type checker, still pre 1.0, run with no configuration file present.
Wins
It is the only tool here that is both the fastest and jointly the most accurate, and its diagnostics carry context blocks pointing at the declaration site, which is the part you need when the error is reported somewhere other than where you typed it.
Loses
It is version 0.0.90. The output format is not stable, those helpful context blocks are easy to miscount if you are parsing rather than reading, and nobody should be pinning a release process to it yet.
10/100 false positives89 ms
The reference implementation and the one most projects already have in their pre-commit config.
Wins
It finds everything at its defaults, explains itself better than anything else here, and is the tool that every error code, every stub package and every piece of advice on the internet is written against.
Loses
It is 2.7x slower than ty on one small file, and that gap grows on a real codebase where its caching behaviour and import following become the thing you spend configuration effort on.
10/100 false positives240 ms
Microsoft's checker, the engine behind Pylance and therefore the one most VS Code users are already running.
Wins
If you use VS Code you are already getting these nine findings as you type, with no pipeline step and no configuration, which is worth more in practice than a tenth finding in CI.
Loses
Slowest of teh three modern checkers by a factor of eight, it needs a Node runtime in whatever image runs it, and its basic mode is noticeably more permissive about optional values flowing into strict positions.
9/100 false positives1,938 ms
Google's inference based checker, which runs the code's types rather than only reading the annotations.
Wins
Its inference finds things in unannotated code that annotation readers cannot, which is the case it was built for and still the reason to reach for it on a legacy codebase with no type hints at all.
Loses
Ten seconds for one small file is 110x ty's time, and on a fully annotated module that inference bought nothing: it scored lowest of the four tools that found anything.
8/109,767 ms110x slower than ty
Meta's Rust checker, which selects a preset when it cannot find a configuration file and tells you which one.
Wins
It is honest about it. The line saying which preset it chose is printed on every run, so the zero is explained in the output rather than left to be discovered later by a type error in production.
Loses
A checker that reports success on a file containing ten type errors is going to be trusted by someone who did not read the second line of output. Shipping the permissive preset as the no-config default is the single most consequential decision in this comparison.
0/10 at default10 errors under strict260 ms
Side by side
| Found | False positives | Time |
|---|
| ty 0.0.90 | 10/10 | 0 | 89 ms |
| mypy 2.4.0 | 10/10 | 0 | 240 ms |
| pyright | 9/10 | 0 | 1,938 ms |
| pytype 2024.10.11 | 8/10 | 0 | 9,767 ms |
| pyrefly 1.4.0 (basic) | 0/10 | 0 | 260 ms |
Verdict
Run mypy in CI today and watch ty closely, because ty matched it exactly on this file at a third of the time and is the only thing here that might eventually make the choice interesting. The result to actually act on is pyrefly's. It reported a clean file containing ten type errors, not because it cannot find them, since its strict preset finds all ten, but because its no-config default does not look for them. Whatever checker you use, run it once against a file you have deliberately broken before you believe a green result from it, and check which preset it told you it had chosen.
Sources