Fri, Sep 27, 2024 10:25

Here is a noteworthy product announcement from Cloudflare. To combat the increase of AI-related web crawlers, the company is introducing tools to assist companies in managing and monetizing their website data. According to their blog post, this also tackles the issue of AI services decreasing website traffic by users opting for LLM services instead of visiting websites.

The rise of AI Large Language Models (LLMs) and other generative tools created a murkier third category. Unlike malicious bots, the crawlers associated with these platforms are not actively trying to knock your site offline or to get in the way of your customers. They are not trying to steal sensitive data; they just want to scan what is already public on your site.

However, unlike helpful bots, these AI-related crawlers do not necessarily drive traffic to your site. AI Data Scraper bots scan the content on your site to train new LLMs. Your material is then put into a kind of blender, mixed up with other content, and used to answer questions from users without attribution or the need for users to visit your site. Another type of crawler, AI Search Crawler bots, scan your content and attempt to cite it when responding to a user’s search. The downside is that those users might just stay inside of that interface, rather than visit your site, because an answer is assembled on the page in front of them.

See also this news article on The Register.