#aiscrapers — Public Fediverse posts

I *knew* #aiscrapers are incredibly abusive, I read all about how bad their traffic can get, and counted myself lucky that I was only seeing spikes of 50,000 requests per day, while still being utterly annoyed by how those spikes were messing with my logs and thus working hard to block them more effectively. Yet seeing for myself how suddenly the floodgates opened and I'm getting 30 requests per second is still an entirely different experience than reading about it. Just serving every single one of them a static 403 error page is more than doubling the server's CPU usage!

#fuckAI

I *knew* #aiscrapers are incredibly abusive, I read all about how bad their traffic can get, and counted myself lucky that I was only seeing spikes of 50,000 requests per day, while still being utterly annoyed by how those spikes were messing with my logs and thus working hard to block them more effectively. Yet seeing for myself how suddenly the floodgates opened and I'm getting 30 requests per second is still an entirely different experience than reading about it. Just serving every single one of them a static 403 error page is more than doubling the server's CPU usage!

#fuckAI

@pertho

I have used your #OpenBSD script to keep pf in tune with a list of ip addresses and converted it to FreeBSD, since ftp on FreeBSD is not the same as on OpenBSD.

Any other (dynamic) blocklist that could be added?

@grahamperrin @jo @BrideOfLinux

#noAI #AIscrapers

@pertho

I have used your #OpenBSD script to keep pf in tune with a list of ip addresses and converted it to FreeBSD, since ftp on FreeBSD is not the same as on OpenBSD.

Any other (dynamic) blocklist that could be added?

@grahamperrin @jo @BrideOfLinux

#noAI #AIscrapers

@pertho

I have used your #OpenBSD script to keep pf in tune with a list of ip addresses and converted it to FreeBSD, since ftp on FreeBSD is not the same as on OpenBSD.

Any other (dynamic) blocklist that could be added?

@grahamperrin @jo @BrideOfLinux

#noAI #AIscrapers

@pertho

I have used your #OpenBSD script to keep pf in tune with a list of ip addresses and converted it to FreeBSD, since ftp on FreeBSD is not the same as on OpenBSD.

Any other (dynamic) blocklist that could be added?

@grahamperrin @jo @BrideOfLinux

#noAI #AIscrapers

RE: https://xoxo.zone/@waxy/116893327438667104

but… how much money did Cloudflare make from #aiscrapers gobbling up the billions of cached web pages they got?

RE: https://xoxo.zone/@waxy/116893327438667104

but… how much money did Cloudflare make from #aiscrapers gobbling up the billions of cached web pages they got?

RE: https://xoxo.zone/@waxy/116893327438667104

but… how much money did Cloudflare make from #aiscrapers gobbling up the billions of cached web pages they got?

RE: https://xoxo.zone/@waxy/116893327438667104

but… how much money did Cloudflare make from #aiscrapers gobbling up the billions of cached web pages they got?

RE: https://xoxo.zone/@waxy/116893327438667104

but… how much money did Cloudflare make from #aiscrapers gobbling up the billions of cached web pages they got?

FYI: Fox buys Roku, Publicis and TTD end feud, UK publishers sue AI scrapers: Fox's $22bn Roku deal reshapes CTV, Publicis and The Trade Desk end dispute, UK publishers bill AI scrapers £500 per scraped article through county courts. https://ppc.land/fox-buys-roku-publicis-and-ttd-end-feud-uk-publishers-sue-ai-scrapers/ #FoxBuysRoku #CTV #Publicis #TheTradeDesk #AIscrapers

FYI: Fox buys Roku, Publicis and TTD end feud, UK publishers sue AI scrapers: Fox's $22bn Roku deal reshapes CTV, Publicis and The Trade Desk end dispute, UK publishers bill AI scrapers £500 per scraped article through county courts. https://ppc.land/fox-buys-roku-publicis-and-ttd-end-feud-uk-publishers-sue-ai-scrapers/ #FoxBuysRoku #CTV #Publicis #TheTradeDesk #AIscrapers

My PineNote blog posts were linked to from the main body of a discussion thread on Hacker News but due to the consistent and prolific AI scraping it's barely a blip on my traffic. Remember when humans used to hug sites to death? In the old days this would absolutely stick out.

The good news is a human emailed me and asked me for my experience one year in and that was nice (which is also how I found out about the HN post). I'll share my response below until I get around to fleshing out a blog post from it:

I should absolutely do a write-up a year in. I'll give you the highlights:

• I lost my original PineNote and after a few months of trying to get it back, I ended up getting a replacement, which should say something all in itself.
• I wish the device was a little bit smaller and lighter, but I have gotten used to it.
• I use it as a single function device at a time, mostly writing or reading, very occasional computing.
• However, I on a month-long trip and I only brought along my work laptop and the PineNote instead of bringing a personal laptop. I can do all the personal computing that I need on the PineNote and my phone. See my setup here: https://shom.dev/posts/20250406_a-pinenote-only-5-day-weekend/
• I wish Pine64 would update their images with the kernel optimization created by the community member hrdl, they do link to it from their official docs.
• There is also an interesting project called QuillOS, which provides a nice interface to the whole system, but it's not ready for primetime on PineNote.

#PineNote #HackerNews #AIScrapers #AIBots

My PineNote blog posts were linked to from the main body of a discussion thread on Hacker News but due to the consistent and prolific AI scraping it's barely a blip on my traffic. Remember when humans used to hug sites to death? In the old days this would absolutely stick out.

The good news is a human emailed me and asked me for my experience one year in and that was nice (which is also how I found out about the HN post). I'll share my response below until I get around to fleshing out a blog post from it:

I should absolutely do a write-up a year in. I'll give you the highlights:

• I lost my original PineNote and after a few months of trying to get it back, I ended up getting a replacement, which should say something all in itself.
• I wish the device was a little bit smaller and lighter, but I have gotten used to it.
• I use it as a single function device at a time, mostly writing or reading, very occasional computing.
• However, I on a month-long trip and I only brought along my work laptop and the PineNote instead of bringing a personal laptop. I can do all the personal computing that I need on the PineNote and my phone. See my setup here: https://shom.dev/posts/20250406_a-pinenote-only-5-day-weekend/
• I wish Pine64 would update their images with the kernel optimization created by the community member hrdl, they do link to it from their official docs.
• There is also an interesting project called QuillOS, which provides a nice interface to the whole system, but it's not ready for primetime on PineNote.

#PineNote #HackerNews #AIScrapers #AIBots

My PineNote blog posts were linked to from the main body of a discussion thread on Hacker News but due to the consistent and prolific AI scraping it's barely a blip on my traffic. Remember when humans used to hug sites to death? In the old days this would absolutely stick out.

The good news is a human emailed me and asked me for my experience one year in and that was nice (which is also how I found out about the HN post). I'll share my response below until I get around to fleshing out a blog post from it:

I should absolutely do a write-up a year in. I'll give you the highlights:

• I lost my original PineNote and after a few months of trying to get it back, I ended up getting a replacement, which should say something all in itself.
• I wish the device was a little bit smaller and lighter, but I have gotten used to it.
• I use it as a single function device at a time, mostly writing or reading, very occasional computing.
• However, I on a month-long trip and I only brought along my work laptop and the PineNote instead of bringing a personal laptop. I can do all the personal computing that I need on the PineNote and my phone. See my setup here: https://shom.dev/posts/20250406_a-pinenote-only-5-day-weekend/
• I wish Pine64 would update their images with the kernel optimization created by the community member hrdl, they do link to it from their official docs.
• There is also an interesting project called QuillOS, which provides a nice interface to the whole system, but it's not ready for primetime on PineNote.

#PineNote #HackerNews #AIScrapers #AIBots

My PineNote blog posts were linked to from the main body of a discussion thread on Hacker News but due to the consistent and prolific AI scraping it's barely a blip on my traffic. Remember when humans used to hug sites to death? In the old days this would absolutely stick out.

The good news is a human emailed me and asked me for my experience one year in and that was nice (which is also how I found out about the HN post). I'll share my response below until I get around to fleshing out a blog post from it:

I should absolutely do a write-up a year in. I'll give you the highlights:

• I lost my original PineNote and after a few months of trying to get it back, I ended up getting a replacement, which should say something all in itself.
• I wish the device was a little bit smaller and lighter, but I have gotten used to it.
• I use it as a single function device at a time, mostly writing or reading, very occasional computing.
• However, I on a month-long trip and I only brought along my work laptop and the PineNote instead of bringing a personal laptop. I can do all the personal computing that I need on the PineNote and my phone. See my setup here: https://shom.dev/posts/20250406_a-pinenote-only-5-day-weekend/
• I wish Pine64 would update their images with the kernel optimization created by the community member hrdl, they do link to it from their official docs.
• There is also an interesting project called QuillOS, which provides a nice interface to the whole system, but it's not ready for primetime on PineNote.

#PineNote #HackerNews #AIScrapers #AIBots

My PineNote blog posts were linked to from the main body of a discussion thread on Hacker News but due to the consistent and prolific AI scraping it's barely a blip on my traffic. Remember when humans used to hug sites to death? In the old days this would absolutely stick out.

The good news is a human emailed me and asked me for my experience one year in and that was nice (which is also how I found out about the HN post). I'll share my response below until I get around to fleshing out a blog post from it:

I should absolutely do a write-up a year in. I'll give you the highlights:

• I lost my original PineNote and after a few months of trying to get it back, I ended up getting a replacement, which should say something all in itself.
• I wish the device was a little bit smaller and lighter, but I have gotten used to it.
• I use it as a single function device at a time, mostly writing or reading, very occasional computing.
• However, I on a month-long trip and I only brought along my work laptop and the PineNote instead of bringing a personal laptop. I can do all the personal computing that I need on the PineNote and my phone. See my setup here: https://shom.dev/posts/20250406_a-pinenote-only-5-day-weekend/
• I wish Pine64 would update their images with the kernel optimization created by the community member hrdl, they do link to it from their official docs.
• There is also an interesting project called QuillOS, which provides a nice interface to the whole system, but it's not ready for primetime on PineNote.

#PineNote #HackerNews #AIScrapers #AIBots

AI scrapers are driving up your hosting costs while real users are left waiting in the digital lobby 🤖

It is time to take the pressure off your infrastructure by using robots.txt and cache normalization to manage those thirsty bots 💡

We are sharing how to set sane application limits so your site stays fast for humans and does not turn into a villain story for your budget/.

👉 https://developer.upsun.com/posts/insights/the-not-so-hidden-cost-of-ai-scrapers

#WebPerformance #DevOps #AIScrapers #TechInsights

AI scrapers are driving up your hosting costs while real users are left waiting in the digital lobby 🤖

It is time to take the pressure off your infrastructure by using robots.txt and cache normalization to manage those thirsty bots 💡

We are sharing how to set sane application limits so your site stays fast for humans and does not turn into a villain story for your budget/.

👉 https://developer.upsun.com/posts/insights/the-not-so-hidden-cost-of-ai-scrapers

#WebPerformance #DevOps #AIScrapers #TechInsights

AI scrapers are driving up your hosting costs while real users are left waiting in the digital lobby 🤖

It is time to take the pressure off your infrastructure by using robots.txt and cache normalization to manage those thirsty bots 💡

We are sharing how to set sane application limits so your site stays fast for humans and does not turn into a villain story for your budget/.

👉 https://developer.upsun.com/posts/insights/the-not-so-hidden-cost-of-ai-scrapers

#WebPerformance #DevOps #AIScrapers #TechInsights

AI scrapers are driving up your hosting costs while real users are left waiting in the digital lobby 🤖

It is time to take the pressure off your infrastructure by using robots.txt and cache normalization to manage those thirsty bots 💡

We are sharing how to set sane application limits so your site stays fast for humans and does not turn into a villain story for your budget/.

👉 https://developer.upsun.com/posts/insights/the-not-so-hidden-cost-of-ai-scrapers

#WebPerformance #DevOps #AIScrapers #TechInsights

AI scrapers are driving up your hosting costs while real users are left waiting in the digital lobby 🤖

It is time to take the pressure off your infrastructure by using robots.txt and cache normalization to manage those thirsty bots 💡

We are sharing how to set sane application limits so your site stays fast for humans and does not turn into a villain story for your budget/.

👉 https://developer.upsun.com/posts/insights/the-not-so-hidden-cost-of-ai-scrapers

#WebPerformance #DevOps #AIScrapers #TechInsights

RE: https://mastodon.social/@gamingonlinux/116560908559455886

#RPCS3, open-source PlayStation 3 emulator and debugger, says on Twitter:

> PSA: #Tencent is aggressively scraping the Internet to build yet another AI slop chatbot, DDoSing many websites in the process.
>
> We've found that, as of last week, their scraping bots can now solve Cloudflare challenges and behave like real users while ignoring robots.txt. In the last 24 hours alone, our website received more than 3 million successful requests from Tencent bot IP addresses, plus another 1 million that were blocked by Cloudflare challenges.
>
> These recurring DDoS attacks from Tencent have been going on for over a year, and we have been constantly adjusting our firewall rules to filter them while trying not to impact Tencent's real users. Because that is no longer possible, we're now fully blocking Tencent IP addresses, starting with ASN 132203. We recommend other sysadmins do the same.

https://x.com/rpcs3/status/2061946000734888017

#DDoS #AI #AIChatBot #AIScraping #AIScraper #AIScrapers #PlayStation #PlayStation3 #PS3

RE: https://mastodon.social/@gamingonlinux/116560908559455886

#RPCS3, open-source PlayStation 3 emulator and debugger, says on Twitter:

> PSA: #Tencent is aggressively scraping the Internet to build yet another AI slop chatbot, DDoSing many websites in the process.
>
> We've found that, as of last week, their scraping bots can now solve Cloudflare challenges and behave like real users while ignoring robots.txt. In the last 24 hours alone, our website received more than 3 million successful requests from Tencent bot IP addresses, plus another 1 million that were blocked by Cloudflare challenges.
>
> These recurring DDoS attacks from Tencent have been going on for over a year, and we have been constantly adjusting our firewall rules to filter them while trying not to impact Tencent's real users. Because that is no longer possible, we're now fully blocking Tencent IP addresses, starting with ASN 132203. We recommend other sysadmins do the same.

https://x.com/rpcs3/status/2061946000734888017

#DDoS #AI #AIChatBot #AIScraping #AIScraper #AIScrapers #PlayStation #PlayStation3 #PS3

RE: https://mastodon.social/@gamingonlinux/116560908559455886

#RPCS3, open-source PlayStation 3 emulator and debugger, says on Twitter:

> PSA: #Tencent is aggressively scraping the Internet to build yet another AI slop chatbot, DDoSing many websites in the process.
>
> We've found that, as of last week, their scraping bots can now solve Cloudflare challenges and behave like real users while ignoring robots.txt. In the last 24 hours alone, our website received more than 3 million successful requests from Tencent bot IP addresses, plus another 1 million that were blocked by Cloudflare challenges.
>
> These recurring DDoS attacks from Tencent have been going on for over a year, and we have been constantly adjusting our firewall rules to filter them while trying not to impact Tencent's real users. Because that is no longer possible, we're now fully blocking Tencent IP addresses, starting with ASN 132203. We recommend other sysadmins do the same.

https://x.com/rpcs3/status/2061946000734888017

#DDoS #AI #AIChatBot #AIScraping #AIScraper #AIScrapers #PlayStation #PlayStation3 #PS3

RE: https://mastodon.social/@gamingonlinux/116560908559455886

#RPCS3, open-source PlayStation 3 emulator and debugger, says on Twitter:

> PSA: #Tencent is aggressively scraping the Internet to build yet another AI slop chatbot, DDoSing many websites in the process.
>
> We've found that, as of last week, their scraping bots can now solve Cloudflare challenges and behave like real users while ignoring robots.txt. In the last 24 hours alone, our website received more than 3 million successful requests from Tencent bot IP addresses, plus another 1 million that were blocked by Cloudflare challenges.
>
> These recurring DDoS attacks from Tencent have been going on for over a year, and we have been constantly adjusting our firewall rules to filter them while trying not to impact Tencent's real users. Because that is no longer possible, we're now fully blocking Tencent IP addresses, starting with ASN 132203. We recommend other sysadmins do the same.

https://x.com/rpcs3/status/2061946000734888017

#DDoS #AI #AIChatBot #AIScraping #AIScraper #AIScrapers #PlayStation #PlayStation3 #PS3

RE: https://mastodon.social/@gamingonlinux/116560908559455886

#RPCS3, open-source PlayStation 3 emulator and debugger, says on Twitter:

> PSA: #Tencent is aggressively scraping the Internet to build yet another AI slop chatbot, DDoSing many websites in the process.
>
> We've found that, as of last week, their scraping bots can now solve Cloudflare challenges and behave like real users while ignoring robots.txt. In the last 24 hours alone, our website received more than 3 million successful requests from Tencent bot IP addresses, plus another 1 million that were blocked by Cloudflare challenges.
>
> These recurring DDoS attacks from Tencent have been going on for over a year, and we have been constantly adjusting our firewall rules to filter them while trying not to impact Tencent's real users. Because that is no longer possible, we're now fully blocking Tencent IP addresses, starting with ASN 132203. We recommend other sysadmins do the same.

https://x.com/rpcs3/status/2061946000734888017

#DDoS #AI #AIChatBot #AIScraping #AIScraper #AIScrapers #PlayStation #PlayStation3 #PS3

Why is #twitter not properly identifying itself as a bot when trying to scrape my website? (69.12.56.0/21 is AS63179 is Twitter)

Could it be cause they're a malicious party training an #aibot?

(This is extremely low-intensity, but based on the combination of this specific UA and the pages they're trying to reach, I've seen them before, coming in from residential proxies.)

The funny thing is that bots identifying as bots and observing robots.txt would actually be allowed to reach those particular pages.

#aiscrapers #scrapers

Why is #twitter not properly identifying itself as a bot when trying to scrape my website? (69.12.56.0/21 is AS63179 is Twitter)

Could it be cause they're a malicious party training an #aibot?

(This is extremely low-intensity, but based on the combination of this specific UA and the pages they're trying to reach, I've seen them before, coming in from residential proxies.)

The funny thing is that bots identifying as bots and observing robots.txt would actually be allowed to reach those particular pages.

#aiscrapers #scrapers

Why is #twitter not properly identifying itself as a bot when trying to scrape my website? (69.12.56.0/21 is AS63179 is Twitter)

Could it be cause they're a malicious party training an #aibot?

(This is extremely low-intensity, but based on the combination of this specific UA and the pages they're trying to reach, I've seen them before, coming in from residential proxies.)

The funny thing is that bots identifying as bots and observing robots.txt would actually be allowed to reach those particular pages.

#aiscrapers #scrapers

Why is #twitter not properly identifying itself as a bot when trying to scrape my website? (69.12.56.0/21 is AS63179 is Twitter)

Could it be cause they're a malicious party training an #aibot?

(This is extremely low-intensity, but based on the combination of this specific UA and the pages they're trying to reach, I've seen them before, coming in from residential proxies.)

The funny thing is that bots identifying as bots and observing robots.txt would actually be allowed to reach those particular pages.

#aiscrapers #scrapers

Why is #twitter not properly identifying itself as a bot when trying to scrape my website? (69.12.56.0/21 is AS63179 is Twitter)

Could it be cause they're a malicious party training an #aibot?

(This is extremely low-intensity, but based on the combination of this specific UA and the pages they're trying to reach, I've seen them before, coming in from residential proxies.)

The funny thing is that bots identifying as bots and observing robots.txt would actually be allowed to reach those particular pages.

#aiscrapers #scrapers

Czech publishers get new robots.txt shield against AI scrapers: SPIR on March 19 updated its standard for Czech online publishers to opt out of AI text and data mining, adding real-time response crawlers to the scope of the robots.txt framework. https://ppc.land/czech-publishers-get-new-robots-txt-shield-against-ai-scrapers/ #CzechPublishing #AIScrapers #RobotsTxt #DataMining #OnlinePrivacy

Czech publishers get new robots.txt shield against AI scrapers: SPIR on March 19 updated its standard for Czech online publishers to opt out of AI text and data mining, adding real-time response crawlers to the scope of the robots.txt framework. https://ppc.land/czech-publishers-get-new-robots-txt-shield-against-ai-scrapers/ #CzechPublishing #AIScrapers #RobotsTxt #DataMining #OnlinePrivacy

Czech publishers get new robots.txt shield against AI scrapers: SPIR on March 19 updated its standard for Czech online publishers to opt out of AI text and data mining, adding real-time response crawlers to the scope of the robots.txt framework. https://ppc.land/czech-publishers-get-new-robots-txt-shield-against-ai-scrapers/ #CzechPublishing #AIScrapers #RobotsTxt #DataMining #OnlinePrivacy

The issue of #aiscrapers

https://www.kasada.io/stopping-unauthorized-ai-data-scraping/

The issue of #aiscrapers

https://www.kasada.io/stopping-unauthorized-ai-data-scraping/

The issue of #aiscrapers

https://www.kasada.io/stopping-unauthorized-ai-data-scraping/

The issue of #aiscrapers

https://www.kasada.io/stopping-unauthorized-ai-data-scraping/

The issue of #aiscrapers

https://www.kasada.io/stopping-unauthorized-ai-data-scraping/

Traffic sources to my #SelfHosted #Gitea instance. You can clearly see where the real visits are and where the AI scrapers are. Last time I checked, they weren’t triggering any analytic events. They are definitely improving.

#aiscrapers #ai #llm #LLMs #aislop #homelab #selfhost #selfhosting

Traffic sources to my #SelfHosted #Gitea instance. You can clearly see where the real visits are and where the AI scrapers are. Last time I checked, they weren’t triggering any analytic events. They are definitely improving.

#aiscrapers #ai #llm #LLMs #aislop #homelab #selfhost #selfhosting

Traffic sources to my #SelfHosted #Gitea instance. You can clearly see where the real visits are and where the AI scrapers are. Last time I checked, they weren’t triggering any analytic events. They are definitely improving.

#aiscrapers #ai #llm #LLMs #aislop #homelab #selfhost #selfhosting

Traffic sources to my #SelfHosted #Gitea instance. You can clearly see where the real visits are and where the AI scrapers are. Last time I checked, they weren’t triggering any analytic events. They are definitely improving.

#aiscrapers #ai #llm #LLMs #aislop #homelab #selfhost #selfhosting

Traffic sources to my #SelfHosted #Gitea instance. You can clearly see where the real visits are and where the AI scrapers are. Last time I checked, they weren’t triggering any analytic events. They are definitely improving.

#aiscrapers #ai #llm #LLMs #aislop #homelab #selfhost #selfhosting

@grumpybozo

30% of web search traffic goes through #AI now.

The same folk who pontificate about lost web traffic will gleefully tell you they are blocking "#Aiscrapers"

@grumpybozo

30% of web search traffic goes through #AI now.

The same folk who pontificate about lost web traffic will gleefully tell you they are blocking "#Aiscrapers"

@grumpybozo

30% of web search traffic goes through #AI now.

The same folk who pontificate about lost web traffic will gleefully tell you they are blocking "#Aiscrapers"

#AIBots may lead to the end of the internet as we know it

In recent weeks, #OpenDemocracy’s website has been repeatedly brought down by an army of bots. We’re not the only ones

Matthew Linares
20 February 2026

Excerpt: "Slater explained that 'the traffic often arrives through anonymous residential IPs', referring to residential proxy networks that route internet traffic through intermediary servers using IP addresses assigned by internet service providers to real homeowners. This, he said, makes it 'hard to distinguish ‘normal users’ from automated collection'. [That's not right and needs to be changed!!!]

" 'We're being forced into permanent defence mode. #ResidentialProxyNetworks let #AIScrapers hide in plain sight, rotate identities, and extract data at scale. That shifts real costs onto projects that exist to serve people, not feed training pipelines."

Read more:
https://www.opendemocracy.net/en/ai-chatbots-scraper-bots-chatgpt-website-offline-change-internet/

#AISucks #AI #DataMining #Internet #Websites #TechNews #AI #ArtificialIntelligence #BigTech #TechBros