AI Crawler Traffic Overwhelming a Small Developer Site
Press Space for next Tweet
total traffic for last 30 days on http://codebase.md - 9.7k human visitors - 14k agents (curl etc) - 771.5k crawlers (!) ai crawlers are blocked by cloudflare but amazonbot is hammering me with 408k requests a month
