#webcrawlers — Public Fediverse posts

FYI: Google rewrites Googlebot's rulebook: 2MB limits, IP moves, and what crawlers really are: Google today published two blog posts revealing Googlebot's true architecture as a shared SaaS platform, a 2MB fetch limit, and a new IP ranges directory path. https://ppc.land/google-rewrites-googlebots-rulebook-2mb-limits-ip-moves-and-what-crawlers-really-are/ #Googlebot #SEO #WebCrawlers #DigitalMarketing #SaaS

Google rewrites Googlebot's rulebook: 2MB limits, IP moves, and what crawlers really are: Google today published two blog posts revealing Googlebot's true architecture as a shared SaaS platform, a 2MB fetch limit, and a new IP ranges directory path. https://ppc.land/google-rewrites-googlebots-rulebook-2mb-limits-ip-moves-and-what-crawlers-really-are/ #Google #Googlebot #SEO #WebCrawlers #DigitalMarketing

Google-Agent joins the crawler list as AI browsing gets an official identity: Google on March 20 added Google-Agent to its user-triggered fetchers list, formalizing a new user agent for AI systems like Project Mariner that navigate the web on behalf of users. https://ppc.land/google-agent-joins-the-crawler-list-as-ai-browsing-gets-an-official-identity/ #GoogleAgent #AIBrowsing #WebCrawlers #ArtificialIntelligence #ProjectMariner

FYI: Anthropic clarifies what its three web crawlers do - and how to block them: Anthropic today updated its crawler documentation, detailing ClaudeBot, Claude-User, and Claude-SearchBot - what each collects and what blocking them means for site visibility. https://ppc.land/anthropic-clarifies-what-its-three-web-crawlers-do-and-how-to-block-them/ #WebCrawlers #SEO #DataPrivacy #ClaudeBot #SiteVisibility

FYI: Anthropic clarifies what its three web crawlers do - and how to block them: Anthropic today updated its crawler documentation, detailing ClaudeBot, Claude-User, and Claude-SearchBot - what each collects and what blocking them means for site visibility. https://ppc.land/anthropic-clarifies-what-its-three-web-crawlers-do-and-how-to-block-them/ #WebCrawlers #SEO #DataPrivacy #ClaudeBot #SiteVisibility

FYI: Anthropic clarifies what its three web crawlers do - and how to block them: Anthropic today updated its crawler documentation, detailing ClaudeBot, Claude-User, and Claude-SearchBot - what each collects and what blocking them means for site visibility. https://ppc.land/anthropic-clarifies-what-its-three-web-crawlers-do-and-how-to-block-them/ #WebCrawlers #SEO #DataPrivacy #ClaudeBot #SiteVisibility

ICYMI: Anthropic clarifies what its three web crawlers do - and how to block them: Anthropic today updated its crawler documentation, detailing ClaudeBot, Claude-User, and Claude-SearchBot - what each collects and what blocking them means for site visibility. https://ppc.land/anthropic-clarifies-what-its-three-web-crawlers-do-and-how-to-block-them/ #Anthropic #WebCrawlers #SEO #ClaudeBot #DigitalMarketing

ICYMI: Anthropic clarifies what its three web crawlers do - and how to block them: Anthropic today updated its crawler documentation, detailing ClaudeBot, Claude-User, and Claude-SearchBot - what each collects and what blocking them means for site visibility. https://ppc.land/anthropic-clarifies-what-its-three-web-crawlers-do-and-how-to-block-them/ #Anthropic #WebCrawlers #SEO #ClaudeBot #DigitalMarketing

ICYMI: Anthropic clarifies what its three web crawlers do - and how to block them: Anthropic today updated its crawler documentation, detailing ClaudeBot, Claude-User, and Claude-SearchBot - what each collects and what blocking them means for site visibility. https://ppc.land/anthropic-clarifies-what-its-three-web-crawlers-do-and-how-to-block-them/ #Anthropic #WebCrawlers #SEO #ClaudeBot #DigitalMarketing

Anthropic clarifies what its three web crawlers do - and how to block them: Anthropic today updated its crawler documentation, detailing ClaudeBot, Claude-User, and Claude-SearchBot - what each collects and what blocking them means for site visibility. https://ppc.land/anthropic-clarifies-what-its-three-web-crawlers-do-and-how-to-block-them/ #Anthropic #WebCrawlers #ClaudeBot #SearchEngine #DigitalMarketing

Anthropic clarifies what its three web crawlers do - and how to block them: Anthropic today updated its crawler documentation, detailing ClaudeBot, Claude-User, and Claude-SearchBot - what each collects and what blocking them means for site visibility. https://ppc.land/anthropic-clarifies-what-its-three-web-crawlers-do-and-how-to-block-them/ #Anthropic #WebCrawlers #ClaudeBot #SearchEngine #DigitalMarketing

Anthropic clarifies what its three web crawlers do - and how to block them: Anthropic today updated its crawler documentation, detailing ClaudeBot, Claude-User, and Claude-SearchBot - what each collects and what blocking them means for site visibility. https://ppc.land/anthropic-clarifies-what-its-three-web-crawlers-do-and-how-to-block-them/ #Anthropic #WebCrawlers #ClaudeBot #SearchEngine #DigitalMarketing

Facebook's Fascination with My Robots.txt

https://blog.nytsoi.net/2026/02/23/facebook-robots-txt

#HackerNews #Facebook #RobotsTxt #SocialMedia #TechNews #WebCrawlers

NiemanLab: News publishers limit Internet Archive access due to AI scraping concerns. “When The Guardian took a look at who was trying to extract its content, access logs revealed that the Internet Archive was a frequent crawler, said Robert Hahn, head of business affairs and licensing. The publisher decided to limit the Internet Archive’s access to published articles, minimizing the chance […]

NiemanLab: News publishers limit Internet Archive access due to AI scraping concerns. “When The Guardian took a look at who was trying to extract its content, access logs revealed that the Internet Archive was a frequent crawler, said Robert Hahn, head of business affairs and licensing. The publisher decided to limit the Internet Archive’s access to published articles, minimizing the chance […]

NiemanLab: News publishers limit Internet Archive access due to AI scraping concerns. “When The Guardian took a look at who was trying to extract its content, access logs revealed that the Internet Archive was a frequent crawler, said Robert Hahn, head of business affairs and licensing. The publisher decided to limit the Internet Archive’s access to published articles, minimizing the chance […]

NiemanLab: News publishers limit Internet Archive access due to AI scraping concerns. “When The Guardian took a look at who was trying to extract its content, access logs revealed that the Internet Archive was a frequent crawler, said Robert Hahn, head of business affairs and licensing. The publisher decided to limit the Internet Archive’s access to published articles, minimizing the chance […]

NiemanLab: News publishers limit Internet Archive access due to AI scraping concerns. “When The Guardian took a look at who was trying to extract its content, access logs revealed that the Internet Archive was a frequent crawler, said Robert Hahn, head of business affairs and licensing. The publisher decided to limit the Internet Archive’s access to published articles, minimizing the chance […]

How I protect my Forgejo instance from AI web crawlers

https://her.esy.fun/posts/0031-how-i-protect-my-forgejo-instance-from-ai-web-crawlers/index.html

#HackerNews #AIProtection #Forgejo #WebCrawlers #Cybersecurity #TechTips

Picknick an der Datenautobahn

https://niklasbarning.de/2025/12/02/picknick-an-der-datenautobahn/

Picknick an der Datenautobahn

https://niklasbarning.de/2025/12/02/picknick-an-der-datenautobahn/

Picknick an der Datenautobahn

https://niklasbarning.de/2025/12/02/picknick-an-der-datenautobahn/

Picknick an der Datenautobahn

https://niklasbarning.de/2025/12/02/picknick-an-der-datenautobahn/

Search Engine Roundtable: OpenAI Scales Up Crawling & Bots For The Holidays. “OpenAI is reportedly scaling up its crawling infrastructure for the holiday shopping season. The folks at Merj noticed OpenAI adding a lot of new IP ranges for its bots and crawlers.”

https://rbfirehose.com/2025/12/01/search-engine-roundtable-openai-scales-up-crawling-bots-for-the-holidays/

People CEO just called Google a 'bad actor' for allegedly 'stealing content' via its AI crawler. Apparently, you can't block the AI without also blocking the good old web crawler. Convenient, much? What's your take on content ownership when the machines are doing the 'reading'?
https://techcrunch.com/2025/09/12/google-is-a-bad-actor-says-people-ceo-accusing-the-company-of-stealing-content/
#AI #TechNews #Google #ContentOwnership #WebCrawlers

https://thenewleafjournal.com/thinkbot-visits-the-new-leaf-journal/

My site is crawled regularly by a bot called Thinkbot. Thanks to Hacker News, I learned that I am not the only one with this “problem.”

Also #Wikimedia complaints about the constant surge in #LLM #crawling. It is a strain to the free project of #Wikipedia and such. For my MSc thesis, I fetched wikipedia #archives which they regularly snapshot and hand out for free.

Is this resource simply ignored, because it would cause additional processing effort next to the anyway running #webcrawlers? @wikimediafoundation , can you briefly comment on whether or not you have numbers on LLM companies that turn to archives, instead of crawling?

📰🤡 Ah, the digital age's finest: creating fake JPEGs to fool unsuspecting web crawlers! 🎭🐛 This riveting tale involves a Spigot—no, not the leaky kind—that pumps out random #nonsense for #bots to gulp down, while the author marvels at their own concoction like a child watching ants on a sidewalk. 🙄🔍
https://www.ty-penguin.org.uk/~auj/blog/2025/03/25/fake-jpeg/ #fakeJPEGs #webcrawlers #digitalage #Spigot #HackerNews #ngated

📰🤡 Ah, the digital age's finest: creating fake JPEGs to fool unsuspecting web crawlers! 🎭🐛 This riveting tale involves a Spigot—no, not the leaky kind—that pumps out random #nonsense for #bots to gulp down, while the author marvels at their own concoction like a child watching ants on a sidewalk. 🙄🔍
https://www.ty-penguin.org.uk/~auj/blog/2025/03/25/fake-jpeg/ #fakeJPEGs #webcrawlers #digitalage #Spigot #HackerNews #ngated

📰🤡 Ah, the digital age's finest: creating fake JPEGs to fool unsuspecting web crawlers! 🎭🐛 This riveting tale involves a Spigot—no, not the leaky kind—that pumps out random #nonsense for #bots to gulp down, while the author marvels at their own concoction like a child watching ants on a sidewalk. 🙄🔍
https://www.ty-penguin.org.uk/~auj/blog/2025/03/25/fake-jpeg/ #fakeJPEGs #webcrawlers #digitalage #Spigot #HackerNews #ngated

📰🤡 Ah, the digital age's finest: creating fake JPEGs to fool unsuspecting web crawlers! 🎭🐛 This riveting tale involves a Spigot—no, not the leaky kind—that pumps out random #nonsense for #bots to gulp down, while the author marvels at their own concoction like a child watching ants on a sidewalk. 🙄🔍
https://www.ty-penguin.org.uk/~auj/blog/2025/03/25/fake-jpeg/ #fakeJPEGs #webcrawlers #digitalage #Spigot #HackerNews #ngated

Is there a website like https://example.com/, but has multiple URLs and is OK to spider/crawl?

I want to upgrade the examples in spidr's README to websites that wouldn't mind being randomly spidered.
https://github.com/postmodern/spidr#readme

https://example.com isn't a viable option, because it only has one webpage, so not a very good demonstration of web spidering...

#web #webspiders #webcrawlers #exampledotcom #exampledotnet #demo

"To reiterate, whatever one's opinion of these particular AI tools, scraping itself is not the problem. Automated access is a fundamental technique of archivists, computer scientists, and everyday users that we hope is here to stay—as long as it can be done non-destructively. However, we realize that not all implementers will follow our suggestions for bots above, and that our mitigations are both technically advanced and incomplete.

Because we see so many bots operating for the same purpose at the same time, it seems there's an opportunity here to provide these automated data consumers with tailored data providers, removing the need for every AI company to scrape every website, seemingly, every day.

And on the operators' end, we hope to see more web-hosting and framework technology that is built with an awareness of these issues from day one, perhaps building in responses like just-in-time static content generation or dedicated endpoints for crawlers."

https://www.eff.org/deeplinks/2025/06/keeping-web-under-weight-ai-crawlers

#AI #GenerativeAI #WebCrawlers #BigTech #WebScraping #OpenWeb

🚨 Breaking News: Someone discovered inode zero in #POSIX and thought it was worth writing about! 🙄 The author then spirals into a paranoid rant about HTTP UserAgents, because, clearly, the world is ending and it's all the web crawlers' fault. 🌐🔍 The real mystery: how they managed to drag this topic out into a whole article. 😂
https://utcc.utoronto.ca/~cks/space/blog/unix/POSIXAllowsZeroInode #inodezero #HTTPUserAgents #webcrawlers #techhumor #HackerNews #ngated

Nuevo post:

Punto de inflexión tecnológico

#tecnologia #internet #ia #webcrawlers #blog #enmiblog

https://thecheis.com/2025/05/20/punto-de-inflexion-tecnologico/

🤖🚀 Oh no! The web is "broken" because #AI companies are mining data like it's the next #gold rush! 🙄 Turns out you might be running a crawler without even knowing—because apparently #ignorance is the latest tech trend. 🕵️‍♂️💥
https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-2/ #Rush #WebCrawlers #DataMining #HackerNews #ngated

More bots than people. I guess that’s Web 3.0, right?

@calmeilles I remember in the 90s and early 2000s something similar happened with spam e-mail. At one point in time, the bot-to-human ratio reached something like 9:1 or worse. The solution at the time was everyone joining a walled garden like Gmail, or buying expensive spam filtering tools if they had to run e-mail on-premise.

This time, the walled gardens and spam filters (at least how these filters are trained) are themselves the problem.

#tech #internet #ai #scam #Web #WebCrawlers #LLMs

Old man yells at clouds 🤖🌩️: A crusty diatribe against #AI web crawlers that somehow morphs into a shameless plug for server hosting services. Who knew bots had better things to do than read a geriatric infomercial? 🚫📡
https://www.mythic-beasts.com/blog/2025/04/01/abusive-ai-web-crawlers-get-off-my-lawn/ #webcrawlers #infomercial #serverhosting #HackerNews #HackerNews #ngated

Old man yells at clouds 🤖🌩️: A crusty diatribe against #AI web crawlers that somehow morphs into a shameless plug for server hosting services. Who knew bots had better things to do than read a geriatric infomercial? 🚫📡
https://www.mythic-beasts.com/blog/2025/04/01/abusive-ai-web-crawlers-get-off-my-lawn/ #webcrawlers #infomercial #serverhosting #HackerNews #HackerNews #ngated

Old man yells at clouds 🤖🌩️: A crusty diatribe against #AI web crawlers that somehow morphs into a shameless plug for server hosting services. Who knew bots had better things to do than read a geriatric infomercial? 🚫📡
https://www.mythic-beasts.com/blog/2025/04/01/abusive-ai-web-crawlers-get-off-my-lawn/ #webcrawlers #infomercial #serverhosting #HackerNews #HackerNews #ngated

Old man yells at clouds 🤖🌩️: A crusty diatribe against #AI web crawlers that somehow morphs into a shameless plug for server hosting services. Who knew bots had better things to do than read a geriatric infomercial? 🚫📡
https://www.mythic-beasts.com/blog/2025/04/01/abusive-ai-web-crawlers-get-off-my-lawn/ #webcrawlers #infomercial #serverhosting #HackerNews #HackerNews #ngated

Abusive AI Web Crawlers: Get Off My Lawn

https://www.mythic-beasts.com/blog/2025/04/01/abusive-ai-web-crawlers-get-off-my-lawn/

#HackerNews #AbusiveAI #WebCrawlers #GetOffMyLawn #TechEthics #InternetPrivacy #AIRegulation

Abusive AI Web Crawlers: Get Off My Lawn

https://www.mythic-beasts.com/blog/2025/04/01/abusive-ai-web-crawlers-get-off-my-lawn/

#HackerNews #AbusiveAI #WebCrawlers #GetOffMyLawn #TechEthics #InternetPrivacy #AIRegulation

Abusive AI Web Crawlers: Get Off My Lawn

https://www.mythic-beasts.com/blog/2025/04/01/abusive-ai-web-crawlers-get-off-my-lawn/

#HackerNews #AbusiveAI #WebCrawlers #GetOffMyLawn #TechEthics #InternetPrivacy #AIRegulation

Abusive AI Web Crawlers: Get Off My Lawn

https://www.mythic-beasts.com/blog/2025/04/01/abusive-ai-web-crawlers-get-off-my-lawn/

#HackerNews #AbusiveAI #WebCrawlers #GetOffMyLawn #TechEthics #InternetPrivacy #AIRegulation

Google outlines pathway for robots.txt protocol to evolve: How the 30-year-old web crawler control standard could adopt new functionalities while maintaining its simplicity. https://ppc.land/google-outlines-pathway-for-robots-txt-protocol-to-evolve/ #Google #RobotsTxt #WebCrawlers #SEO #DigitalMarketing

ICYMI: Google updates crawler verification processes with daily IP range refreshes: Enhanced security measures help website owners identify legitimate Google web crawlers and protect against potential imposters. https://ppc.land/google-updates-crawler-verification-processes-with-daily-ip-range-refreshes/ #GoogleUpdates #WebCrawlers #SEO #CyberSecurity #DigitalMarketing

Perishable Press: Ultimate Block List to Stop AI Bots. “The focus of this post is aimed at website owners who want to stop AI bots from crawling their web pages, as much as possible. To help people with this, I’ve been collecting data and researching AI bots for many months now, and have put together a ‘Mega Block List’ to help stop AI bots from devouring your content.”

https://rbfirehose.com/2025/02/12/perishable-press-ultimate-block-list-to-stop-ai-bots/

Hackaday: Trap Naughty Web Crawlers In Digestive Juices With Nepenthes. “More commonly known as ‘pitcher plants’, nepenthes is a genus of carnivorous plants that use a fluid-filled cup to trap insects and small critters unfortunate enough to slip & slide down into it. In the case of this Lua-based project the idea is roughly the same. Configured as a trap behind a web server (e.g. […]

https://rbfirehose.com/2025/01/24/hackaday-trap-naughty-web-crawlers-in-digestive-juices-with-nepenthes/

Hackaday: Trap Naughty Web Crawlers In Digestive Juices With Nepenthes. “More commonly known as ‘pitcher plants’, nepenthes is a genus of carnivorous plants that use a fluid-filled cup to trap insects and small critters unfortunate enough to slip & slide down into it. In the case of this Lua-based project the idea is roughly the same. Configured as a trap behind a web server (e.g. […]

https://rbfirehose.com/2025/01/24/hackaday-trap-naughty-web-crawlers-in-digestive-juices-with-nepenthes/