Web giant Cloudflare to block AI bots from scraping content by default

Starting Tuesday, every new web domain that signs up to Cloudflare will be asked if they want to allow or block AI crawlers

The move could "hinder AI chatbots' ability to harvest data for training and purposes," one lawyer told CNBC

In this articleNET your favorite stocksCREATE FREE ACCOUNTJaque Silva | Nurphoto | Getty ImagesInternet firm Cloudflare will start blocking artificial intelligence crawlers from accessing content without website owners' permission or compensation by default, in a move that could significantly impact AI developers' ability to train their models

Starting Tuesday, every new web domain that signs up to Cloudflare will be asked if they want to allow AI crawlers, effectively giving them the ability to prevent bots from scraping data from their websites

The company will also allow publishers to charge AI crawlers for access using a new "pay per crawl" model

Cloudflare is what's called a content dery network, or CDN

It helps es der online content and applications faster by caching the data closer to end-users

They play a significant role in making sure people can access web content seamlessly every day

Roughly 16% of global internet traffic goes directly through Cloudflare's CDN, the firm estimated in a 2023 report. "AI crawlers have been scraping content without limits

Our goal is to put the power back in the hands of creators, while still helping AI companies innovate," said Matthew Prince, co-founder and CEO of Cloudflare, in a statement Tuesday. "This is safeguarding the future of a free and vibrant Internet with a new model that works for everyone," he added

AI crawlers are automated bots designed to extract large quantities of data from websites, databases and other sources of information to train large language models from the s of OpenAI and Google

Whereas the internet previously rewarded creators by directing users to original websites, according to Cloudflare, today AI crawlers are that model by collecting text, and images to generate responses to queries in a way that users don't need to visit the original source

This, the company adds, is depriving publishers of vital traffic and, in turn, revenue from online advertising

Stock hits recordTuesday's move builds on a tool Cloudflare launched in September last year that gave publishers the ability to block AI crawlers with a single click

Now, the company is going a step further by making this the default for all websites it vides services for

OpenAI says it declined to participate when Cloudflare previewed its plan to block AI crawlers by default on the grounds that the content dery network is adding a middleman to the system

The Microsoft-backed AI lab stressed its role as a pioneer of using robots

Txt, a set of code that prevents automated scraping of web data, and said its crawlers respect publisher preferences. "AI crawlers are typically seen as more invasive and selective when it comes to the data they consumer

They have been accused of overwhelming websites and significantly impacting user experience," Matthew Holman, a partner at U

Law firm Cripps, told CNBC. "If effective, the development would hinder AI chatbots' ability to harvest data for training and purposes," he added. "This is ly to lead to a short term impact on AI model training and could, over the long term, affect the viability of models. "WATCH: AI engineers are in high demand — but what is the job really