#aicrawlers — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #aicrawlers, aggregated by home.social.
-
AI crawlers now burden git.kernel.org with 6M daily requests, burning more CPU rendering commits for scrapers than on all legitimate access combined.
#AICrawlers #LinuxKernel #Anubis #WebScraping #OpenSource
https://securityonline.info/ai-crawlers-git-kernel/?utm_source=mastodon&utm_medium=jetpack_social
-
🤖 Infrastructura Kernel.org este bombardată de milioane de cereri zilnice ale bot-ilor de Inteligență Artificială!
Publicația Linuxiac relatează o problemă majoră de infrastructură cu care se confruntă administratorii Kernel.org (platforma oficială de găzduire și dezvoltare a nucleului Linux, administrată de Konstantin Ryabitsev): un val masiv de AI crawlers și scrapperi automatizați consumă resurse critice ale serverelor.
✨ Punctele cheie ale situației generate de bot-ii de IA:
💥 Consum masiv de resurse procesor (CPU):
• Serverele git.kernel.org primesc în jur de 6 milioane de cereri zilnice doar pentru paginile individuale ale commit-urilor Git.
• Din cele 90 de nuclee CPU distribuite pe 5 locații geografice, aproximativ 14-16 nuclee sunt folosite constant doar pentru a genera dinamic pagini HTML pentru bot-ii care adună date pentru antrenarea modelelor de limbaj (LLM).📉 Traficul legitim a devenit o minoritate:
• Tranzacțiile legitime (inclusiv descărcările și comanda git clone efectuate de dezvoltatori umani) reprezintă doar 2% din totalul cererilor, restul de 98% fiind generat de bot-i automatizați.⚙️ Ineficiența scraperilor și fentarea restricțiilor:
• În loc să cloneze depozitul Git o singură dată și să lucreze local, mulți crawleri interoghează individual fiecare URL de commit.
• Deși s-au încercat blocări pe bază de IP/ASN și un sistem de verificare Proof-of-Work (Anubis), bot-ii au adaptat rapid atacul trecând prin rețele proxy de IP-uri rezidențiale și de mobil (fentând blocările) și alocând mai multă putere de calcul pentru a trece de verificări.🛡️ Măsurile drastice luate în considerare:
• Echipa de infrastructură ia în calcul reducerea drastică a numărului de URL-uri indexabile/generabile dinamic și restricționarea acțiunilor costisitoare din punct de vedere CPU pentru utilizatorii anonimi, pentru a asigura stabilitatea platformei.📌 Concluzie:
Această situație evidențiază un efect secundar major al cursei pentru antrenarea modelelor de IA: exploatarea excesivă a infrastructurii publice gratuite Open Source, punând o presiune financiară și tehnică uriașă pe proiectele comunitare! 🚀
#KernelOrg #Linux #AICrawlers #Git #SysAdmin #OpenSource #Linuxiac #TechNews #DevOps
-
Git repositories are made to be cloned, not scraped commit page by commit page.
At scale, HTML crawling wastes CPU on rendered pages, creates higher costs for open source hosts, and can lead to proof-of-work checks or rate limits that affect real users first.
Clone once. Fetch updates. Use Git as intended.
-
FYI: Cloudflare blocks opaque AI crawlers from sites that disallow training: Bot Preference Sync rewrites robots.txt from dashboard settings on every plan, free to Enterprise. Ad-funded domains now default to disallowing AI training. https://ppc.land/cloudflare-blocks-opaque-ai-crawlers-from-sites-that-disallow-training/ #Cloudflare #AICrawlers #RobotsTxt #WebDevelopment #AIEthics
-
🔍 Wow, who knew small businesses wanted to be 'visible' only to later be snubbed by AI's selective memory? 😲 Apparently, only a minuscule percentage even care to block the AI crawlers, while the rest are blissfully unaware they're not even invited to the AI party! 🎉
https://website-auditor.io/ai-visibility-index #smallbusiness #AIvisibility #selectiveAI #awareness #AIcrawlers #HackerNews #ngated -
Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers
https://website-auditor.io/ai-visibility-index
Comments: https://news.ycombinator.com/item?id=49143630
#HackerNews #AIcrawlers #WebVisibility #DataInsights #CitingTrends #DigitalFootprint
-
FYI: IAB Australia forces every crawler into one of four verdicts: Just 2.6% of AI crawler traffic serves live queries versus 52% for training, IAB Australia finds, ahead of Cloudflare's default block starting in September. https://ppc.land/iab-australia-forces-every-crawler-into-one-of-four-verdicts/ #IAB #Australia #AICrawlers #WebTraffic #DigitalMarketing
-
IAB Australia forces every crawler into one of four verdicts: Just 2.6% of AI crawler traffic serves live queries versus 52% for training, IAB Australia finds, ahead of Cloudflare's default block starting in September. https://ppc.land/iab-australia-forces-every-crawler-into-one-of-four-verdicts/ #IABAustralia #AICrawlers #DigitalMarketing #Cloudflare #AdTech
-
Welp, now they came for my personal site.
I already reluctantly added Cloudflare to botwiki.org. But I *really* don't want to have to do that to my site.
This kind of blows. Thanks AI bros.
#indieweb #PersonalWebsite #devops #bots #spam #AICrawlers #enshittification
-
FYI: AI crawlers hit sites 50,000 times per human visit, Cloudflare data shows: AI crawlers reach up to 50,000 visits per referred reader, Cloudflare data shows, as a Munich court strips Google of its liability shield over AI Overviews. https://ppc.land/ai-crawlers-hit-sites-50-000-times-per-human-visit-cloudflare-data-shows/ #AICrawlers #Cloudflare #DigitalMarketing #WebTraffic #SEO
-
FYI: Cloudflare exposes AI crawlers hitting sites 50000 times per visitor: Bot Management customers can now filter traffic by Training, Search or Agent use case and compare operators side by side. Will the data shift licensing talks? https://ppc.land/cloudflare-exposes-ai-crawlers-hitting-sites-50000-times-per-visitor/ #Cloudflare #AICrawlers #BotManagement #WebTraffic #DataAnalytics
-
ICYMI: Cloudflare exposes AI crawlers hitting sites 50000 times per visitor: Bot Management customers can now filter traffic by Training, Search or Agent use case and compare operators side by side. Will the data shift licensing talks? https://ppc.land/cloudflare-exposes-ai-crawlers-hitting-sites-50000-times-per-visitor/ #Cloudflare #AICrawlers #BotManagement #Cybersecurity #TrafficFilter
-
AI crawlers hit sites 50,000 times per human visit, Cloudflare data shows: AI crawlers reach up to 50,000 visits per referred reader, Cloudflare data shows, as a Munich court strips Google of its liability shield over AI Overviews. https://ppc.land/ai-crawlers-hit-sites-50-000-times-per-human-visit-cloudflare-data-shows/ #AICrawlers #Cloudflare #DataAnalytics #WebTraffic #SEO
-
Cloudflare exposes AI crawlers hitting sites 50000 times per visitor: Bot Management customers can now filter traffic by Training, Search or Agent use case and compare operators side by side. Will the data shift licensing talks? https://ppc.land/cloudflare-exposes-ai-crawlers-hitting-sites-50000-times-per-visitor/ #Cloudflare #AICrawlers #BotManagement #DigitalMarketing #WebTraffic
-
#Cloudflare will block #AIcrawlers from ad-supported pages starting 15 September, unless site owners opt in. This move aims to address the issue of #AI companies using #webcontent for #training without compensating #publishers, potentially harming the open web. Cloudflare is also introducing a “#PayPerUse” system to compensate publishers when their content influences AI answers. https://thenextweb.com/news/cloudflare-block-ai-crawlers-pay-publishers?eicker.news #tech #media #news
-
ICYMI: The user agent strings every SEO and site owner needs right now: A technical reference for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Bingbot - the exact user agent strings shaping how AI crawls the web in 2026. https://ppc.land/the-user-agent-strings-every-seo-and-site-owner-needs-right-now/ #SEO #WebCrawling #UserAgent #DigitalMarketing #AICrawlers
-
It looks like whoever is behind this has finally come for one of my sites. Manually blocking datacenter IPs has worked for a while, but it seems like the traffic is now coming either from residential IPs, or the IPs are just getting spoofed.
Real conundrum. I didn't want to have to set up Cloudflare for the site, and I have to wonder how good they are at catching this anyway, but it might just come to that.
Anyone else has been dealing with this?
-
FYI: Kinsta adds free bot protection to all WordPress plans: Kinsta on June 9 launched Bot Protection for all plans, giving WordPress owners control over AI crawlers and automated traffic inside MyKinsta at no added cost. https://ppc.land/kinsta-adds-free-bot-protection-to-all-wordpress-plans/ #Kinsta #WordPress #BotProtection #AICrawlers #TechNews
-
@kaffeeringe das hatte ich auch schon, ebenfalls der Hinweis, per htaccess einfach gewisse/die meisten IP Netze zu sperren.
Was ich aber nicht verstehe: wenn so ein KI Bot eine Website "abgrast" sollte das doch mit einem Aufruf pro Seite erledigt sein. Wozu müssen die das 10.000e Male machen? Oder ist diese Technik des Einlesens einfach genauso schlecht und unausgereift, wie die Ausgaben?
#KIBot #KICrawler #AIBot #AICrawler #AICrawlers #AI -
Attaque crawlers IA en cours sur le serveur gayfr.online : 98% des challenges #Anubis sont rejetés (donc sont des bots), plus de 800 000 requêtes sur la semaine.
Du coup aucun impact sur les performances ni la disponibilité. Il reste juste un bot vietnamien malin qui passe sous les radars et interroge le service RSS avec plusieurs secondes ou minutes d'intervalle. Pas gênant mais je lui réglerai son compte.
#gayfr #IA #AI #ArtificialIntelligence #IntelligenceArtificielle #AICrawlers #AIBots
-
🇬🇧 Protect Mastodon against AI bots with Anubis
I wrote a complete guide for Fediverse admins on how to protect against AI bots: https://blogs.gayfr.social/barbapulpe/protect-mastodon-against-ai-bots-with-anubis
To protect our users and our servers against those rogue robots.
Please boost!
#IA #AI #AICrawlers #AIBots #Anubis #Mastodon #MastoAdmin #MastodonAdmin #ArtificialIntelligence #IntelligenceArtificielle
-
🇫🇷 Protéger Mastodon contre les bots IA avec Anubis
J'ai écrit un guide complet pour les administrateurs du fédiverse sur comment se protéger contre les bots IA : https://blogs.gayfr.social/barbapulpe/proteger-mastodon-contre-les-bots-ia-avec-anubis
Pour protéger nos utilisateurs et nos serveurs contre ces robots voyous.
N'hésitez pas à faire tourner !
#IA #AI #AICrawlers #AIBots #Anubis #Mastodon #MastoAdmin #MastodonAdmin #ArtificialIntelligence #IntelligenceArtificielle
-
C'est assez rigolo, les bots IA se calment le week-end, on passe de >90% en semaine à 70% "seulement" en fin de semaine...
-
🇬🇧 If you like numbers, our dashboard lets you view all our sites’ statistics in real time.
https://status.gayfr.social
https://status.gayfr.onlineI’ve added new metrics related to Anubis, which protects us against AI bots. So at the bottom, you’ll see the number of bots blocked on sight (red), the number of visits that need to verify the “I’m not a robot” page (orange), the percentage that succeed (meaning they aren’t bots, in yellow), and finally the number of accepted human visits (green).
What’s interesting is that the percentage of bots varies throughout the week but remains very high (between 65% and 96% depending on the time). That’s huge! They attack in swarms, just like a pain in the ass…
That justifies the efforts made to protect against them.
#gayfr #Anubis #IA #AI #ArtificialIntelligence #Bots #AIBots #AICrawlers