False-positive benchmark
A false positive is a warning on a site that is fine. The false-positive rate is the share of legitimate sites a detector wrongly warns about, and it decides whether a warning tool is worth keeping on.
What a false positive is
A phishing warner earns its place by staying quiet on ordinary sites and speaking up only when something is off. If it cries wolf, you learn to click past it, and then it cannot help you when a warning is real. So we measure how often Fish Catcher warns about good sites and publish a reproducible result. The short version is on the transparency page.
The corpus
We use the Majestic Million, a public list of the most-linked domains on the web, ranked by how many separate networks link to them, and take the first 100,000. The same cached copy feeds the model-training script, so reruns are offline. A popularity list can contain a parked or briefly compromised domain; any such case counts against Fish Catcher here, not for it.
Method
For each domain the benchmark builds https://domain/ and asks the shipped analyzer for a verdict.
- Corpus: Majestic Million, first 100,000 domains.
- Engine: the shipped passive URL analyzer, no page content, no cloud lookups.
- Counted: warnings at high or critical only. Elevated and low do not interrupt you.
- Metric: (high + critical warnings) / domains scanned. Lower is better.
Latest result
Run of , 100,000 legitimate sites:
The flagged domains
The 20 August run flagged none. Earlier runs flagged five very short, digit-heavy names on high-abuse endings (hy315.cc, 854854.cc, god123.xyz, 121644.cc, cm883.cc), each at the bottom of the high band; the retrained on-device model now rates them elevated, the level that does not alarm. The audit still prints every domain it flags on each run, so the claim stays checkable.
Recall
The other side of the coin: how much of the live phishing traffic the engine catches from the address alone. The corpus is the three public feeds the threat registry packs every day (Phishing.Database active domains, URLhaus, OpenPhish), deduplicated to 392,233 hosts on . The same address-only analyzer runs with the bundled blocklist and the community list switched off, so the result measures the heuristics and the on-device model, not the list.
The most frequent reasons among the 285,129 flagged hosts:
| Reason | Hosts |
|---|---|
| On-device model (random-looking name) | 276,801 |
| Unusual mix of digits and hyphens | 56,828 |
| Brand name in the address, not the brand's domain | 27,351 |
| High-abuse domain ending | 40,521 |
| Phishing keyword in the address | 22,915 |
| Deep chain of subdomains | 20,063 |
| Very short, random-looking domain | 10,540 |
| Raw IP address | 5,467 |
| Misspelled brand | 493 |
Most misses are ordinary-looking names on free hosting (weebly.com, 000webhostapp.com, godaddysites.com, pages.dev) or plain compromised sites, which no address rule can tell apart from their legitimate neighbours. Those are the list's job, and the page checks add more once you open the site. Reproduce it with:
node scripts/recall-audit.mjsFlags are the same as the false-positive audit: --limit N, --selftest, --cache path.txt. The first run downloads the three feeds and caches the merged list for the day.
Version history
Rerun after engine changes and at least monthly. Newest on top.
| Date | Corpus | High | Critical | Rate | Recall (any / alarm) |
|---|---|---|---|---|---|
| 2026-08-20 | Majestic Million, 100,000; feeds, 392,233 | 0 | 0 | 0% | 73.0% / 10.9% |
| 2026-08-19 | Majestic Million, 100,000 | 5 | 0 | 0.005% | not measured |
| 2026-08-11 | Majestic Million, 100,000 | 5 | 0 | 0.005% | not measured |
What changed on 20 August: the recall audit was added; the engine is unchanged, so the false-positive numbers are the same as 19 August.
What changed on 19 August: hosting-platform suffixes (netlify.app, web.app, pages.dev, github.io and similar) are treated as separate registrable domains; 24 more brands, including major US and UK banks, Coinbase, Binance, MetaMask, USPS, FedEx, DHL, DocuSign and Okta; digit-homoglyph folding; softer weights on known-legitimate domains. Same result.
Reproduce it
Clone the repository and run the audit. It prints the rate and every flagged domain; the first run downloads and caches the corpus, later runs are offline.
node scripts/fp-audit.mjs--limit Nscans fewer domains for a quick sample.--selftestchecks the harness on about ten famous domains that must stay calm.--cache path.csvreuses a corpus you already downloaded.
The audit always exits zero; only --selftest fails, and only if the harness itself is broken.