99% of my website traffic is bots

The gist

• PatronView, a 1.5 million-page US philanthropy donor database built from IRS 990 forms and donor walls, served 1.28 million pages in one week while self-hosted Plausible recorded just 5,977 human pageviews, roughly 214 bot page loads per human.
• On April 22, 2026 the site took 3.62 million requests in one day from 361,844 unique IPs, nearly all Chinese, with Cloudflare's Managed Challenge absorbing 1.18 million in about ten hours before the author blocked China on April 23 and later Vietnam and Singapore.
• The author measured Claude-SearchBot at 35,000 pages crawled per visitor referred, versus Cloudflare's stated Anthropic average of about 3,000, citing 420,680 pages and 4.63 GB served in a week against 12 human visitors and 175 KB; after a 403 block, Anthropic's crawler fell from 60,000 requests a day to about 25.
• Crawl-to-referral ratios cited are Googlebot 46:1, Bingbot 406:1 (158,610 requests for 680 visitors), Claude-SearchBot 35,000:1, and Amzn-SearchBot infinite, with Amazon's crawler blocked two days before publication after hitting about 117,000 requests a day.
• Cloudflare's JavaScript Detections added 2,875 milliseconds on a mid-range phone versus 278ms for the site's own JavaScript; turning it off on August 5 lifted the mobile Lighthouse score from 58 to 99, but five hours later an Azure scraper pulled 23,000 pages in an hour from over 80 IPs.
• Unique IPs per day rose from a roughly 18,000 baseline to 124,000 on July 31 from residential botnets spoofing Chrome 118 to 120, prompting a rule that challenges Chrome 100 to 130 and old Firefox, since only 0.54% of real search visitors use browsers that old.

Discussion