AI Crawler Traffic Overwhelming a Small Developer Site
Press Space for next Tweet
total traffic for last 30 days on http://codebase.md - 9.7k human visitors - 14k agents (curl etc) - 771.5k crawlers (!) ai crawlers are blocked by cloudflare but amazonbot is hammering me with 408k requests a month
9
3
0
4
Topics
Read the stories that matter.The stories and ideas that actually matter.
Save hours a day in 5 minutesTurn hours of scrolling into a five minute read.