Back to all articles
•Technology

Cloudflare Sets New Rules for AI Web Crawlers to Protect Publisher Content

View original source

Cloudflare has announced new guidelines to differentiate between traditional search web crawlers and those used for AI purposes, effective from September 15, 2026.

  • Default Settings Change: Cloudflare's settings will block "mixed-use" crawlers from accessing sites with ads unless site owners adjust these settings.
  • Impact on AI Providers: This move could hinder AI model providers' access to data necessary for training and agent services.
  • Protection of Intellectual Property: Many site owners want their content available via search and AI, but without compromising their intellectual property.
  • Google's Role: The world's largest search engine, likely Google, is mentioned for having more data access than other AI companies, causing some criticism. Google claims it allows site owners to opt-out from AI training without affecting search results through Google Extended bot.
  • Response to Non-Human Traffic: With non-human internet traffic surpassing human traffic, Cloudflare aims to create a sustainable ecosystem.
  • Commercial Opportunities for Publishers: The company introduces tools to manage AI scraping, including evolving models like "Pay Per Use," compensating publishers when AI companies extract value from their content.
  • Initial Partnerships: Cloudflare collaborates with Ceramic.ai and You.com to implement their new model.