#pay-per-crawl — Public Fediverse posts

Behold the AI bots that Cloudflare blocked from this blog

I don’t like writing for free–social media blatantly excepted–so when I watched a panel at Web Summit in mid-November about the effect of AI-model crawlers on news-site revenue and the Pay Per Crawl initiative that Cloudflare was proposing as a solution, I had to take notes.

Then a few weeks after I got home from Lisbon, I realized I could take action: While Pay Per Crawl remains in an invitation-only beta test, Cloudflare’s AI Crawl Control is open to the public and included in that Internet infrastructure firm’s free tier. Then I learned that it’s shockingly easy to add Cloudflare’s services to a WordPress.com blog.

Crawl Control comes with a preset list of bots to block and bots to allow, grouped by type: “AI Assistant” bots that take action in response to user requests are fine; “AI Search” bots that support “AI-driven search experiences” are also okay (contrary to Cloudflare CEO Matthew Prince’s discussion of them in that Web Summit panel); “AI Crawler” bots that collect content for training AI models are not.

I took a screenshot of this part of my Cloudflare dashboard at almost the same time each afternoon this week, and these are my totals:

• Huawei’s PetalBot was the highest-volume AI crawler, with Cloudflare reporting 224 “unsuccessful” request attempts from that Chinese tech giant’s AI crawler (Cloudflare doesn’t take direct credit for blocking bots in this interface), followed by Anthropic’s Claude-SearchBot, with 165 unsuccessful requests.
• Among AI assistants, the second-highest category by volume, OpenAI’s ChatGPT-User had 1,251 allowed requests, DuckDuckGo’s DuckAssistBot had 36 allowed, and Perplexity’s Perplexity-User had one unsuccesful request.
• The top bot in AI search came from an unlikely place: Apple’s Applebot, with 734 allowed. OpenAI’s OAI-SearchBot was far behind, with 128 allowed requests, while Perplexity’s PerplexityBot had all eight request attempts fail.

To put this in context, the top two search engine crawlers had exponentially higher numbers. Google’s Googlebot somehow racked up a little over 20,000 requests, more than 30 times the presumably-human traffic I see in my WordPress dashboard here for the last five days, and 23 failed requests. Microsoft’s Bingbot came in second with 3,003 allowed requests and two unsuccessful ones.

As Cloudflare’s CEO complained in that Web Summit panel, Googlebot feeds into both Google’s traditional search and the AI Overview search results that Web publishers now blame for dangerous declines in their search traffic. There’s nothing I can do about that from this side of the screen except hope that Cloudflare’s Pay Per Crawl efforts and other advocacy efforts stir some rethinking at Google.

But I can’t tell you how well Pay Per Crawl works, because almost three weeks after applying to join the private beta I’m still waiting for my invitation. I imagine I’ll be waiting much longer before an AI-crawler operator decides that my tiny contribution to the Web’s collective content is worth sending me some money.

#AI #AIBot #AICrawlControl #AICrawler #Amazon #Applebot #Bingbot #ChatGPT #Cloudflare #Huawei #OpenAI #PayPerCrawl #Petalbot

Weekly output: Mozilla Firefox CEO, AI crawlers vs. publishers and creators, teenage AI chatbot use, Android Live Emergency Video, PCMag’s best tech bought in 2025, World App

Somehow I’m down to the last full workweek of the year–and yet my writing and gift shopping seem to have more than a week’s worth of work remaining.

12/8/2025: Mozilla is doing a delicate dance with AI, Fast Company

I spoke with Mozilla CEO Laura Chambers at a Web Summit event for the second time this year. One thing Firefox’s management no longer needs to worry about, unlike when I met with Chambers at Web Summit Qatar in February: the threat of Google being forced to stop paying browser developers to keep its search engine as the default.

12/9/2025: AI Platforms Are Paying (Some) Big Publishers, Leaving Smaller Ones Behind, PCMag

This post began with me taking notes from a Web Summit panel featuring Cloudflare CEO Matthew Prince talking about that Internet infrastructure company’s Pay Per Crawl initiative to push AI providers to pay Web publishers for access to their content, then I did some follow-up reporting that included setting up Cloudflare’s AI Crawl Control bot-blocking filter on this blog, and then I had to update the post the morning it was published after the European Commission opened an investigation into how Google runs its AI Overview search feature.

12/9/2025: 28% of Teens Use Chatbots Daily. You Can Probably Guess Which One They Like Best, PCMag

The latest survey by the Pew Research Center surfaced some interesting statistics about how much teenagers use AI chatbots and which ones they use the most.

12/10/2025: Need Help? Android Phones Can Now Share Live Video With 911 Dispatchers, PCMag

Google is shipping this feature a year after Apple did, but its emergency live video implementation works on far more devices than Apple’s.

12/11/2025: The Best Tech PCMag Editors Bought in 2025, PCMag

I wrote a short graf lauding the compact, quick-charging (and Wirecutter-endorsed) USB-C charger that I bought after losing the considerably bulkier model that came with my laptop.

12/13/2025: App That Verifies Your Existence Adds Encrypted Messaging, PCMag

Tools for Humanity announced an update to its World App that adds an end-to-end-encrypted chat feature and expands its cryptocurrency tools. I took advantage of this news peg to try out the app’s ability to verify a “World ID” by scanning the NFC tag on my U.S. passport; that did not go well at all for me.

12/15/2025: Updated to add the PCMag best-tech package that I forgot to check for on Sunday.

#AIChatbot #AIOverview #AISearch #ChatGPT #Firefox #GoogleZero #Mozilla #PayPerCrawl #PewResearchCenter #ToolsForHumanity #WebSummit #WorldApp #WorldID

RSL is the missing layer for the AI era: set terms, get attribution, and get paid (per crawl or per inference). Open standard, collective leverage. If AI uses your work, it should respect your license. Time to take control.

https://hostvix.com/rsl-a-new-standard-to-make-ai-pay-for-the-content-it-consumes/

#RSL #ReallySimpleLicensing #AI #AIethics #AIsafety #AIdata #ContentRights #Licensing #OpenWeb #RobotsTxt #Publishers #Creators #Attribution #PayPerCrawl #PayPerInference #RSS #WebStandards #DigitalRights #CollectiveLicensing #Fastly

Bin gespannt, ob sich ein Geschäftsmodell wie "pay per crawl", das offenbar gerade von #Cloudflare wegen der grassierenden AI-Bots getestet wird, bestehen wird.

https://techcrunch.com/2025/07/01/cloudflare-launches-a-marketplace-that-lets-websites-charge-ai-bots-for-scraping/

#PayPerCrawl

Cloudflare Lets Websites Charge AI Crawlers For Scraping Content #cloudflare #internet #paypercrawl

https://www.lowyat.net/2025/359525/cloudflare-lets-websites-charge-ai-crawlers-scraping-content/

🚀 New #ThisWeekInNET episode!

Cloudflare’s Will Allen joins us to talk Pay-Per-Crawl and Content Independence Day (with a Matthew Prince cameo).

How can content creators get paid when AI bots crawl their work? Big shift to opt-in.

Watch here: https://youtu.be/Hp-uCchlgic

#PayPerCrawl

News Summary: Cloudflare Launches Pay Per Crawl for AI Scraping; Amazon Hits One Million Robots

You’ve heard, of course, of pay-per-view. And we are used to streaming revenue on a pay-per basis from the likes of Audible and Spotify. This week has seen the launch (admittedly at the moment in beta) of possibly the most transformative source of pay-per revenue…
https://selfpublishingadvice.org/cloudflare/

#AIscraping #Amazonrobots #Cloudflare #generativeAI #PayPerCrawl
@indieauthors

El lado del mal - CloudFlare Pay-Per-Crawl: Un servicio que ayuda a las webs a negociar el pago por acceso al contenido que hacen los crawlers de IA https://www.elladodelmal.com/2025/07/cloudflare-pay-per-crawl-un-servicio.html #iA #InteligenciaArtificial #PayPerCrawl #AI #CloudFlare #WAF

#Cloudflare lässt KI-Crawler auflaufen, wenn nicht für #Scraping bezahlt wird | heise online https://www.heise.de/news/Cloudflare-laesst-KI-Crawler-auflaufen-wenn-nicht-fuer-Scraping-bezahlt-wird-10467015.html #PayPerCrawl #ArtificialIntelligence #copyright #Urheberrecht

If you celebrate this, you are celebrating that a single company is able to turn on or off access to a substantial part of the web. That's why they can credibly offer such a "feature".

https://blog.cloudflare.com/introducing-pay-per-crawl/

#PayPerCrawl cloudflare

Cloudflare Launches Pay Per Crawl, Letting Publishers Charge AI for Content

#AI #Publishing #WebEconomy #PayPerCrawl #AIEthics #Cloudflare

https://winbuzzer.com/2025/07/01/cloudflare-launches-pay-per-crawl-letting-publishers-charge-ai-for-content-xcxwbn/

#Cloudflare launched a #marketplace called #PayperCrawl, allowing website owners to charge #AIbots for scraping their content: Cloudflare acts as an #intermediary, facilitating transactions between #publishers and #AIcompanies. https://techcrunch.com/2025/07/01/cloudflare-launches-a-marketplace-that-lets-websites-charge-ai-bots-for-scraping/?eicker.news #tech #media #news

Cloudflare introduces pay-per-crawl for AI bots

https://blog.cloudflare.com/introducing-pay-per-crawl/

#HackerNews #Cloudflare #AI #Cloud #Computing #PayPerCrawl #Bots