Soft directives don’t stop crawlers from ingesting deprecated | Tech Twitter

Finding signal on Twitter is more difficult than it used to be. We curate the best tweets on topics like AI, startups, and product development every weekday so you can focus on what matters.

Trendingwhat is spiking in the last 24 hours Time Travelreplay any day in the archive Topicsclusters the feed keeps returning to Profilesthe accounts we watch

//

Articleslong-form writeups pulled from the feed Threadspublished editions, start to finish

//

Agentshow machines read this site Skillsinstallable skills for your agent agents.mdthe plain-text brief

//News//About//

Focus

Press Space to continue

Built by Daniel Hunter

Soft directives don’t stop crawlers from ingesting deprecated | Tech Twitter

Press Space to continue

Soft directives don’t stop crawlers from ingesting deprecated content. Redirects for AI Training allows anybody on Cloudflare to redirect verified crawlers to canonical pages with one toggle and no origin changes. https://cfl.re/4mAGpW5

Redirects for AI Training enforces canonical content

Redirects for AI Training enforces canonical content

Previous tweet Next tweet More from @Cloudflare