False-positive benchmark
A false positive is a warning shown on a site that is actually fine, and the false-positive rate is the share of legitimate sites a detector wrongly warns about. It is the single number that decides whether a warning tool is worth keeping switched on.
What a false positive is
A phishing warner earns its place by staying quiet on the millions of ordinary sites you visit and speaking up only when something is genuinely off. If it cries wolf on normal sites, you quickly learn to click past its warnings, and then it cannot help you on the day one of them is real. Precision matters more than a big scary number: a tool that flags everything catches every scam and is also useless.
So we measure how often Fish Catcher warns about good sites, publish the result, and make it reproducible. If it is not measured, it is just a claim. This page is the long version of the number quoted on the transparency page: the corpus, the method, the exact command, the domains that were flagged, and the history so far.
The corpus
To measure false positives you need a large set of sites that are known to be legitimate. We use the Majestic Million, a public, freely available list of the most-linked domains on the web, ranked by how many separate networks link to them. The top of that list is dominated by long-established, widely referenced sites, which makes it a fair stand-in for normal, legitimate browsing. The benchmark takes the first 100,000 domains from it.
The same cached copy of the list is used by the project's model-training script, so a single download serves both and every rerun is fully offline. No list of legitimate sites is perfect: a popularity ranking can, in principle, contain a parked or briefly compromised domain. Any such case counts against Fish Catcher here, not for it, so if anything this method is slightly harsh on the false-positive number rather than flattering to it.
In short, the corpus is deliberately weighted toward sites that should never be flagged, which is what makes a low number here meaningful rather than easy to reach.
Method
The benchmark runs only the passive, address-only part of the shipped Fish Catcher engine: the exact same analyzer that runs inside the extension, with no page-content signals and no cloud lookups enabled. For each of the 100,000 domains it builds the address https://domain/ and asks the analyzer for a verdict. It then counts how many domains would be warned about at high or critical, the two levels that actually raise an alarm for a user. The quieter elevated and low levels do not interrupt you, so they are not counted here. The rate is the count of high and critical warnings divided by the number of domains scanned. Lower is better.
- Corpus: Majestic Million, first 100,000 domains.
- Engine: the shipped passive URL analyzer, with no page content and no cloud lookups.
- Counted: warnings at high or critical only.
- Metric: (high + critical warnings) / total domains scanned.
Because only the address is used, the run is reproducible from the domain list alone, with nothing fetched from or stored about any page you visit.
Latest result
Latest run, , over 100,000 legitimate sites:
| Measure | Value |
|---|---|
| Domains scanned | 100,000 |
| Warnings at high | 5 |
| Warnings at critical | 0 |
| False-positive rate | 0.005% (5 in 100,000) |
The flagged domains
Because the audit prints every domain it flags, the result is fully auditable rather than a number you have to trust. In the latest run all five were the same shape: very short, digit-heavy names on high-abuse domain endings, exactly the pattern that automated phishing kits produce by the thousand. Each scored 45, the very bottom of the high band, and each tripped the same four signals: a high-abuse ending, a digit-heavy name, a very short name, and the on-device model.
| Domain | Score | Level | Why it was flagged |
|---|---|---|---|
| hy315.cc | 45 | high | high-abuse ending, digit-heavy, very short, on-device model |
| 854854.cc | 45 | high | high-abuse ending, digit-heavy, very short, on-device model |
| god123.xyz | 45 | high | high-abuse ending, digit-heavy, very short, on-device model |
| 121644.cc | 45 | high | high-abuse ending, digit-heavy, very short, on-device model |
| cm883.cc | 45 | high | high-abuse ending, digit-heavy, very short, on-device model |
These are genuinely hard to tell apart from throwaway phishing domains by their address alone, which is why the engine leans cautious on them. Five in a hundred thousand, all at the lowest alarming score, is the price of catching the many similar-looking names that are not innocent.
Version history
We rerun this monthly and add a row each time, so the trend stays public and any regression is visible. Newest runs go on top.
| Date | Corpus | High | Critical | Rate |
|---|---|---|---|---|
| 2026-08-11 | Majestic Million, 100,000 | 5 | 0 | 0.005% |
Reproduce it
You do not have to take the number on faith. Clone the source repository and run the audit yourself; it prints the rate and every domain it flagged, so you can check both:
node scripts/fp-audit.mjsOn the first run it downloads the Majestic Million once and caches it, so every rerun afterwards is offline. The useful flags are:
--limit Nscans fewer domains for a quick sample.--selftestproves the harness on about ten famous domains that must stay calm.--cache path.csvreuses a corpus you have already downloaded.
The measurement always exits zero: it is a measurement, not a pass-or-fail test, so a non-zero finding never breaks a build. Only --selftest fails loudly, and only if the harness itself is broken.