#webcrawling — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #webcrawling, aggregated by home.social.
-
PatronView blocks Amazon's AI crawler after 117,000 daily page reads: Anthropic's crawler hit a 35,000 to 1 crawl ratio, and CAPTCHA solve rates measured just 0.24%. The findings show why small operators are locking down servers. https://ppc.land/patronview-blocks-amazons-ai-crawler-after-117-000-daily-page-reads/ #AI #WebCrawling #CyberSecurity #TechNews #DataPrivacy
-
PatronView blocks Amazon's AI crawler after 117,000 daily page reads: Anthropic's crawler hit a 35,000 to 1 crawl ratio, and CAPTCHA solve rates measured just 0.24%. The findings show why small operators are locking down servers. https://ppc.land/patronview-blocks-amazons-ai-crawler-after-117-000-daily-page-reads/ #AI #WebCrawling #CyberSecurity #TechNews #DataPrivacy
-
RE: https://mastodon.social/@koen_hufkens/116979524452174787
This is absolute insanity.
We could have decentralized index and no reason to crawl websites like madman.
We have search engines that crawl the web regularly, including Google, and they can behave!But ofc people in silicon Valley building shit mentality prefer the hammer style. Brute forcing everything.
See OpenAI and anthropic bellow. That is just evil, it should be considered a DDOS and punishable by law.
What is the solution?
We can't avoid what they are doing right now, the PandoraBox is open, but what can we do?I wonder if we could get traffic shapes to tell us if this is coming from harnesses or directly from different LLM vendors such as OpenAI, Anthropic, ZAI, etc.
This website seems to be facing the later, but traffic from harnesses are still egregious already.If the traffic come from vendors we should sue them for DDOS, if it's from harnesses, I think many of them are open source we could contribute so they start using a distributed index or existing search engine results.
Any other ideas?
#search #distributedsystems #openai #anthropic
#searchengine #webcrawler #webcrawling -
RE: https://mastodon.social/@koen_hufkens/116979524452174787
This is absolute insanity.
We could have decentralized index and no reason to crawl websites like madman.
We have search engines that crawl the web regularly, including Google, and they can behave!But ofc people in silicon Valley building shit mentality prefer the hammer style. Brute forcing everything.
See OpenAI and anthropic bellow. That is just evil, it should be considered a DDOS and punishable by law.
What is the solution?
We can't avoid what they are doing right now, the PandoraBox is open, but what can we do?I wonder if we could get traffic shapes to tell us if this is coming from harnesses or directly from different LLM vendors such as OpenAI, Anthropic, ZAI, etc.
This website seems to be facing the later, but traffic from harnesses are still egregious already.If the traffic come from vendors we should sue them for DDOS, if it's from harnesses, I think many of them are open source we could contribute so they start using a distributed index or existing search engine results.
Any other ideas?
#search #distributedsystems #openai #anthropic
#searchengine #webcrawler #webcrawling -
ICYMI: Google's crawler math turns against it as the open web pushes back: Cloudflare shows AI bots crawling sites 50,000 times per visitor as Munich strips Google's cover for Overviews and Google dismisses a rival crawler standard. https://ppc.land/googles-crawler-math-turns-against-it-as-the-open-web-pushes-back/ #Google #WebCrawling #AI #Cloudflare #OpenWeb
-
ICYMI: Google's crawler math turns against it as the open web pushes back: Cloudflare shows AI bots crawling sites 50,000 times per visitor as Munich strips Google's cover for Overviews and Google dismisses a rival crawler standard. https://ppc.land/googles-crawler-math-turns-against-it-as-the-open-web-pushes-back/ #Google #WebCrawling #AI #Cloudflare #OpenWeb
-
Google's crawler math turns against it as the open web pushes back: Cloudflare shows AI bots crawling sites 50,000 times per visitor as Munich strips Google's cover for Overviews and Google dismisses a rival crawler standard. https://ppc.land/googles-crawler-math-turns-against-it-as-the-open-web-pushes-back/ #Google #AI #WebCrawling #Cloudflare #SEO
-
Google's crawler math turns against it as the open web pushes back: Cloudflare shows AI bots crawling sites 50,000 times per visitor as Munich strips Google's cover for Overviews and Google dismisses a rival crawler standard. https://ppc.land/googles-crawler-math-turns-against-it-as-the-open-web-pushes-back/ #Google #AI #WebCrawling #Cloudflare #SEO
-
Engadget: Cloudflare will filter out web crawlers that serve AI companies . “Cloudflare has announced plans to automatically block mixed-use web crawlers that index websites for search engines and act as AI agents and trainers at the same time. The company previously offered its customers the optional ability to prevent crawlers from scraping their sites for AI chatbots, but now Cloudflare’s […]
https://rbfirehose.com/2026/07/06/engadget-cloudflare-will-filter-out-web-crawlers-that-serve-ai-companies/ -
Engadget: Cloudflare will filter out web crawlers that serve AI companies . “Cloudflare has announced plans to automatically block mixed-use web crawlers that index websites for search engines and act as AI agents and trainers at the same time. The company previously offered its customers the optional ability to prevent crawlers from scraping their sites for AI chatbots, but now Cloudflare’s […]
https://rbfirehose.com/2026/07/06/engadget-cloudflare-will-filter-out-web-crawlers-that-serve-ai-companies/ -
RT @Hobo_Web: Achtung... Ein Blick auf die frisch aktualisierten Daten der Google Search Console zeigt, dass Google erneut bestimmte Seiten/URLs aus dem Web entfernt hat. Dies könnte die Verzögerung der letzten Wochen erklären und ein Indikator für solche zukünftigen Google-Aktivitäten sein. In diesem Beispiel ist die Bereinigung im Bereich der nicht indizierten Seiten sichtbar. Beim letzten Mal wurden dabei auch indizierte Inhalte zu Waren/Produkten betroffen.
mehr auf Arint.info
#GoogleSearchConsole #GoogleUpdate #SEO #Suchmaschinenoptimierung #WebCrawling #Webmaster #arint_info
-
FYI: Cloudflare stops charging AI per crawl and starts paying per answer: Cloudflare ties AI payments to citations as half of bot crawls fetch unchanged pages, and a randomized study finds AI Overviews cut publisher clicks 39.8%. https://ppc.land/cloudflare-stops-charging-ai-per-crawl-and-starts-paying-per-answer/ #Cloudflare #AI #ArtificialIntelligence #WebCrawling #DigitalMarketing
-
FYI: Cloudflare stops charging AI per crawl and starts paying per answer: Cloudflare ties AI payments to citations as half of bot crawls fetch unchanged pages, and a randomized study finds AI Overviews cut publisher clicks 39.8%. https://ppc.land/cloudflare-stops-charging-ai-per-crawl-and-starts-paying-per-answer/ #Cloudflare #AI #ArtificialIntelligence #WebCrawling #DigitalMarketing
-
ICYMI: Cloudflare stops charging AI per crawl and starts paying per answer: Cloudflare ties AI payments to citations as half of bot crawls fetch unchanged pages, and a randomized study finds AI Overviews cut publisher clicks 39.8%. https://ppc.land/cloudflare-stops-charging-ai-per-crawl-and-starts-paying-per-answer/ #Cloudflare #AI #DigitalMarketing #WebCrawling #SEO
-
ICYMI: Cloudflare stops charging AI per crawl and starts paying per answer: Cloudflare ties AI payments to citations as half of bot crawls fetch unchanged pages, and a randomized study finds AI Overviews cut publisher clicks 39.8%. https://ppc.land/cloudflare-stops-charging-ai-per-crawl-and-starts-paying-per-answer/ #Cloudflare #AI #DigitalMarketing #WebCrawling #SEO
-
ICYMI: Cloudflare ties AI payouts to citations as 50% of crawls waste: Ceramic.ai and You.com will now pay publishers per query result, not per page fetch, since over half of good-bot crawls only re-fetch pages that never changed. https://ppc.land/cloudflare-ties-ai-payouts-to-citations-as-50-of-crawls-waste/ #Cloudflare #AI #DigitalMarketing #SEO #WebCrawling
-
ICYMI: Cloudflare ties AI payouts to citations as 50% of crawls waste: Ceramic.ai and You.com will now pay publishers per query result, not per page fetch, since over half of good-bot crawls only re-fetch pages that never changed. https://ppc.land/cloudflare-ties-ai-payouts-to-citations-as-50-of-crawls-waste/ #Cloudflare #AI #DigitalMarketing #SEO #WebCrawling
-
Cloudflare ties AI payouts to citations as 50% of crawls waste: Ceramic.ai and You.com will now pay publishers per query result, not per page fetch, since over half of good-bot crawls only re-fetch pages that never changed. https://ppc.land/cloudflare-ties-ai-payouts-to-citations-as-50-of-crawls-waste/ #Cloudflare #AI #DigitalMarketing #SEO #WebCrawling
-
Cloudflare ties AI payouts to citations as 50% of crawls waste: Ceramic.ai and You.com will now pay publishers per query result, not per page fetch, since over half of good-bot crawls only re-fetch pages that never changed. https://ppc.land/cloudflare-ties-ai-payouts-to-citations-as-50-of-crawls-waste/ #Cloudflare #AI #DigitalMarketing #SEO #WebCrawling
-
FYI: The user agent strings every SEO and site owner needs right now: A technical reference for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Bingbot - the exact user agent strings shaping how AI crawls the web in 2026. https://ppc.land/the-user-agent-strings-every-seo-and-site-owner-needs-right-now/ #SEO #UserAgent #AI #WebCrawling #DigitalMarketing
-
FYI: The user agent strings every SEO and site owner needs right now: A technical reference for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Bingbot - the exact user agent strings shaping how AI crawls the web in 2026. https://ppc.land/the-user-agent-strings-every-seo-and-site-owner-needs-right-now/ #SEO #UserAgent #AI #WebCrawling #DigitalMarketing
-
ICYMI: The user agent strings every SEO and site owner needs right now: A technical reference for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Bingbot - the exact user agent strings shaping how AI crawls the web in 2026. https://ppc.land/the-user-agent-strings-every-seo-and-site-owner-needs-right-now/ #SEO #WebCrawling #UserAgent #DigitalMarketing #AICrawlers
-
ICYMI: The user agent strings every SEO and site owner needs right now: A technical reference for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Bingbot - the exact user agent strings shaping how AI crawls the web in 2026. https://ppc.land/the-user-agent-strings-every-seo-and-site-owner-needs-right-now/ #SEO #WebCrawling #UserAgent #DigitalMarketing #AICrawlers
-
"The worst of these proposed standards would give websites far greater ability to automatically block legitimate, lawful scraping and crawling. For example, the AI Preferences working group is working on proposals to give publishers a way to express “preference signals” against crawling web data for AI-related purposes, including to train models, generate outputs, and help users search the web. These preference signals would be expressed through robots.txt and could potentially become legally binding in some jurisdictions.
Another working group, called Web Bot Auth, is pursuing efforts to protect sites from overly-aggressive bots that strain website resources—a positive goal that could meaningfully improve the internet in the AI era. But Web Bot Auth is simultaneously pursuing a much more dangerous path as well: standards changes that would enable sites to cryptographically identify bots so that they can more easily block anyone they wish—not just “bad” actors, but competitors, dissidents, or anyone who hasn’t paid for the right to access sites using automated tools. If sites restrict crawling to a preapproved list of cryptographically authenticated bots, they could require licensing payments from those wishing to crawl their sites. This would close off the open web to researchers, archivists, and startups without the ability to pay for automated access.
Websites may have legitimate reasons to worry about AI’s impacts on their traffic and advertising revenue, but those reasons must be weighed against the benefits of the open web."
https://www.eff.org/deeplinks/2026/06/free-and-open-web-under-attack-ietf
-
"The worst of these proposed standards would give websites far greater ability to automatically block legitimate, lawful scraping and crawling. For example, the AI Preferences working group is working on proposals to give publishers a way to express “preference signals” against crawling web data for AI-related purposes, including to train models, generate outputs, and help users search the web. These preference signals would be expressed through robots.txt and could potentially become legally binding in some jurisdictions.
Another working group, called Web Bot Auth, is pursuing efforts to protect sites from overly-aggressive bots that strain website resources—a positive goal that could meaningfully improve the internet in the AI era. But Web Bot Auth is simultaneously pursuing a much more dangerous path as well: standards changes that would enable sites to cryptographically identify bots so that they can more easily block anyone they wish—not just “bad” actors, but competitors, dissidents, or anyone who hasn’t paid for the right to access sites using automated tools. If sites restrict crawling to a preapproved list of cryptographically authenticated bots, they could require licensing payments from those wishing to crawl their sites. This would close off the open web to researchers, archivists, and startups without the ability to pay for automated access.
Websites may have legitimate reasons to worry about AI’s impacts on their traffic and advertising revenue, but those reasons must be weighed against the benefits of the open web."
https://www.eff.org/deeplinks/2026/06/free-and-open-web-under-attack-ietf
-
FYI: OpenAI tripled its web crawl after GPT-5 - but ChatGPT users may be declining: New log file analysis of 7 billion OpenAI bot events reveals a 3.5x surge in OAI-SearchBot activity after GPT-5, while ChatGPT user-driven events dropped 28%. https://ppc.land/openai-tripled-its-web-crawl-after-gpt-5-but-chatgpt-users-may-be-declining/ #OpenAI #GPT5 #ChatGPT #AItrends #webcrawling
-
FYI: Google-Agent joins the crawler list as AI browsing gets an official identity: Google on March 20 added Google-Agent to its user-triggered fetchers list, formalizing a new user agent for AI systems like Project Mariner that navigate the web on behalf of users. https://ppc.land/google-agent-joins-the-crawler-list-as-ai-browsing-gets-an-official-identity/ #GoogleAgent #AIBrowsing #UserAgent #WebCrawling #ProjectMariner
-
FYI: Googlebot is not a program - Google engineers finally explain what it really is: Google engineers reveal Googlebot is a misnomer for a central SaaS crawling platform serving dozens of products, with a 15 MB default file size limit and geo-crawling constraints. https://ppc.land/googlebot-is-not-a-program-google-engineers-finally-explain-what-it-really-is/ #Googlebot #SEO #WebCrawling #DigitalMarketing #SaaS
-
FYI: Googlebot is not a program - Google engineers finally explain what it really is: Google engineers reveal Googlebot is a misnomer for a central SaaS crawling platform serving dozens of products, with a 15 MB default file size limit and geo-crawling constraints. https://ppc.land/googlebot-is-not-a-program-google-engineers-finally-explain-what-it-really-is/ #Googlebot #SEO #WebCrawling #DigitalMarketing #SaaS
-
Googlebot is not a program - Google engineers finally explain what it really is: Google engineers reveal Googlebot is a misnomer for a central SaaS crawling platform serving dozens of products, with a 15 MB default file size limit and geo-crawling constraints. https://ppc.land/googlebot-is-not-a-program-google-engineers-finally-explain-what-it-really-is/ #Googlebot #SEO #WebCrawling #SaaS #DigitalMarketing
-
Googlebot is not a program - Google engineers finally explain what it really is: Google engineers reveal Googlebot is a misnomer for a central SaaS crawling platform serving dozens of products, with a 15 MB default file size limit and geo-crawling constraints. https://ppc.land/googlebot-is-not-a-program-google-engineers-finally-explain-what-it-really-is/ #Googlebot #SEO #WebCrawling #SaaS #DigitalMarketing
-
FYI: Google's secret crawl logic, finally explained in one page: Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing. https://ppc.land/googles-secret-crawl-logic-finally-explained-in-one-page/ #Google #SEO #WebCrawling #Googlebot #DigitalMarketing
-
FYI: Google's secret crawl logic, finally explained in one page: Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing. https://ppc.land/googles-secret-crawl-logic-finally-explained-in-one-page/ #Google #SEO #WebCrawling #Googlebot #DigitalMarketing
-
ICYMI: Google's secret crawl logic, finally explained in one page: Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing. https://ppc.land/googles-secret-crawl-logic-finally-explained-in-one-page/ #Google #SEO #WebCrawling #Googlebot #DigitalMarketing
-
ICYMI: Google's secret crawl logic, finally explained in one page: Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing. https://ppc.land/googles-secret-crawl-logic-finally-explained-in-one-page/ #Google #SEO #WebCrawling #Googlebot #DigitalMarketing
-
Google's secret crawl logic, finally explained in one page: Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing. https://ppc.land/googles-secret-crawl-logic-finally-explained-in-one-page/ #Google #SEO #WebCrawling #Googlebot #DigitalMarketing
-
Google's secret crawl logic, finally explained in one page: Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing. https://ppc.land/googles-secret-crawl-logic-finally-explained-in-one-page/ #Google #SEO #WebCrawling #Googlebot #DigitalMarketing
-
Smart TVs are now running Bright SDK to silently crawl the web for AI training, using residential proxies to bypass Google policy. The move sparks a compliance backlash and raises privacy concerns for home devices. How will regulators respond, and what does this mean for open‑source AI? Dive into the details. #SmartTV #BrightSDK #WebCrawling #DeviceCompliance
🔗 https://aidailypost.com/news/smart-tvs-using-bright-sdk-crawl-web-ai-amid-compliance-backlash
-
Smart TVs are now running Bright SDK to silently crawl the web for AI training, using residential proxies to bypass Google policy. The move sparks a compliance backlash and raises privacy concerns for home devices. How will regulators respond, and what does this mean for open‑source AI? Dive into the details. #SmartTV #BrightSDK #WebCrawling #DeviceCompliance
🔗 https://aidailypost.com/news/smart-tvs-using-bright-sdk-crawl-web-ai-amid-compliance-backlash
-
Ah, #wxpath, because using #XPath was just too easy before 🙄. Now with extra layers of #complexity, just in case you weren't already confused enough by web crawling! 🕸️😵💫
https://github.com/rodricios/wxpath #webcrawling #technews #developerhumor #HackerNews #ngated -
Ah, #wxpath, because using #XPath was just too easy before 🙄. Now with extra layers of #complexity, just in case you weren't already confused enough by web crawling! 🕸️😵💫
https://github.com/rodricios/wxpath #webcrawling #technews #developerhumor #HackerNews #ngated -
wxpath – Declarative web crawling in XPath
https://github.com/rodricios/wxpath
#HackerNews #wxpath #webcrawling #XPath #technology #open-source #GitHub
-
wxpath – Declarative web crawling in XPath
https://github.com/rodricios/wxpath
#HackerNews #wxpath #webcrawling #XPath #technology #open-source #GitHub
-
Google Built Its Empire Scraping The Web. Now It’s Suing To Stop Others From Scraping Google
-
Google Built Its Empire Scraping The Web. Now It’s Suing To Stop Others From Scraping Google
-
Cloudflare's 2025 data reveals Google's structural advantage in AI training: Googlebot crawled 11.6% of web pages vs OpenAI's 3.6%. Publishers face an impossible choice - they can't block Google's AI crawling without losing search visibility entirely, since the same bot handles both functions. #AI #WebCrawling #DigitalRights
-
Cloudflare's 2025 data reveals Google's structural advantage in AI training: Googlebot crawled 11.6% of web pages vs OpenAI's 3.6%. Publishers face an impossible choice - they can't block Google's AI crawling without losing search visibility entirely, since the same bot handles both functions. #AI #WebCrawling #DigitalRights
-
Search Engine Roundtable: OpenAI Scales Up Crawling & Bots For The Holidays. “OpenAI is reportedly scaling up its crawling infrastructure for the holiday shopping season. The folks at Merj noticed OpenAI adding a lot of new IP ranges for its bots and crawlers.”
-
Search Engine Roundtable: OpenAI Scales Up Crawling & Bots For The Holidays. “OpenAI is reportedly scaling up its crawling infrastructure for the holiday shopping season. The folks at Merj noticed OpenAI adding a lot of new IP ranges for its bots and crawlers.”
-
Released scrapy-contrib-bigexporter 1.0.0 (https://codeberg.org/ZuInnoTe/scrapy-contrib-bigexporters) - additional export formats for the webscraping framework Scrapy.
Migrated parquet export from fastparquet to pyarrow as fastparquet is deprecated (https://docs.dask.org/en/stable/changelog.html#fastparquet-engine-deprecated)
Migrated orc export from pyorc to pyarrow to reduce the number of dependencies
#scrapy #crawling #python #parquet #orc #pyarrow #webcrawling #scraping
-
Released scrapy-contrib-bigexporter 1.0.0 (https://codeberg.org/ZuInnoTe/scrapy-contrib-bigexporters) - additional export formats for the webscraping framework Scrapy.
Migrated parquet export from fastparquet to pyarrow as fastparquet is deprecated (https://docs.dask.org/en/stable/changelog.html#fastparquet-engine-deprecated)
Migrated orc export from pyorc to pyarrow to reduce the number of dependencies
#scrapy #crawling #python #parquet #orc #pyarrow #webcrawling #scraping
-
Remember when 'robots.txt' was supposed to solve all our crawling problems? Online media brands are trying a new protocol to deter 'unwanted' AI crawlers. Because clearly, we need more digital fences. What's your bet on how long it takes for a savvy AI to find a workaround?
-
Search Engine Land: Google fixes reduced crawling issue impacting some websites. “Google has confirmed it fixed an issue with its crawlers impacting ‘some sites.’ The issue was ‘reduced / fluctuating crawling’ from Google’s end with Googlebot. It is now resolved and Google said the crawling should pick back up in the near future.”