#claudebot — Public Fediverse posts

Разобрал логи сервера: какие ИИ-боты реально ходят по сайту и что они запрашивают

Про ИИ-ботов пишут в основном пересказами документации вендоров: этот собирает данные для обучения, тот ходит за содержанием по запросу пользователя. Я решил посмотреть, что происходит на самом деле — прежде всего по журналам доступа сервера. Ниже представляю вам разбор: кого видно в логах, что они запрашивают, чем отличаются два типа ботов у одного вендора и почему файл llms.txt в этой картине почти не встречается.

https://habr.com/ru/articles/1072256/

#ИИботы #GPTBot #ClaudeBot #PerplexityBot #логи_сервера #robotstxt #llmstxt #UserAgent #парсеры #анализ_трафика

Welp

This is an interesting commentary about something, from #Bluesky #Github:

"[bluesky-social/social-app] Slim down #CLAUDE . md (PR # 11102)"

"Why

CLAUDE.md is loaded into every agent session, so its size has a direct token cost.

The most valuable lines are the ones that contradict an agent's defaults (footguns, negative constraints, project-specific conventions) — long code examples for components the agent will read the source of anyway are mostly fat, and they drift out of date.

This cuts the file from 928 to 580 lines with no loss of behavior-changing guidance.

What changed ..."

And then, ironically, we see this next:

claude[bot] left a comment (bluesky-social/social-app # 11102)

Claude Code is working… claude-thinking.gif (view on web)

"I'll analyze this and get back to you."

View job run

#Claudebot

Always happy to help...

#AI #CodeDev #Github

J'ai limité à 1 req/min certains bots en renvoyant un code 429 #ClaudeBot

Ca fait quand même 3 hits/s #ClaudeBot

Comment bloquer les crawlers IA qui pillent votre site sans vous demander la permission ?

https://fed.brid.gy/r/https://korben.info/bloquer-crawlers-ia-robots-txt-htaccess-nginx.html

#Anthropic really tried to hit #Autocade today … but we had pre-blocked the buggers. #ClaudeBot

Ah, the digital pyramids have spoken: You really don't need 🐕‍🦺 Anubis 🐕‍🦺, unless, of course, you enjoy turning your visitors into cryptographic slaves. 🙃 Apparently, ClaudeBot's relentless pursuit of your sweet, sweet data isn't enough to warrant anything more than a raised eyebrow. Maybe just get a bigger 'robots.txt' and pray to the cyber gods! 🙏💻
https://fxgn.dev/blog/anubis/ #digitalpyramids #cryptographicdata #ClaudeBot #robots.txt #cybersafety #HackerNews #ngated

I was wondering why my social media instance is acting so slow. Turns out my CPU is pegged to 100% due to #Amazon (EC2?) and #Meta/#Facebook IPs absolutely hammering my poor @[email protected] instance, hitting addresses that belong to a copy I have of the QMK firmware.

Some user agents:

* ClaudeBot/1.0; [email protected] (about 90%?)
* facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

Whatever happened to being corteous when crawling??? How is this acceptable? My server is absolutely dying!

#claude #ClaudeBot #MetaAI

That's yet another stupid AI scrapper banned from my sites for driving usage through the roof...

#claudebot #ai #anthropic

Ich arbeite nun schon recht lange für einen kleinen Internet-Provider.
Aber so einen aggressiven Bot, wie den #claudebot von Anthropic, habe ich lange nicht mehr erlebt.
Beim abgrasen des Auftrittes eines Kunden, hat der Bot es geschafft einen (älteren) Server mit 4 CPUs komplett auszulasten. 😠

Kurz das Log-File gescannt und über 1000 verschiedene IPs gefunden, die der #claudebot verwendet hat.

Ein paar Stichproben haben ergeben, das alle IPs zu "Hackers Paradise" (oder AWS) gehören. (1/x)

Has anyone had any luck throttling #ClaudeBot using #rack_attack? It effectively took #FromThePage down last night, so I'm testing this out without much success:

Anthropic’s “ClaudeBot/1.0” is agressively scraping @blendernation to the point where they're causing load issues. Time to add a little #nginx magic and show these freeloaders the door.

#claudebot #llm #damn

Ce matin... deux BOT de scrapping pour alimenter des modèles d'IA/#LLM ont abusé du forum d'@osm_fr

C'est pas la première fois et ça devient vraiment une plaie, surtout quand #ClaudeBot requête les URL de notre ancien #phpBB, remplacé il y a plusieurs années par #discourse

Malgrès plus de 130 000 erreurs 404 rien que ce matin, il continuait à un rythme effréné...

Autre bot albert-bot... de albertai.com (rien avoir avec l'Albert cocorico), bloqué lui aussi.

@paco I've just been hit by "ClaudeBot" from anthropic dot com

from IPs:

3.17.150.89
18.191.88.249
18.118.31.247
18.221.208.183

I wrote this small nginx server rule which will block the bot (and can work for others), especially if the bots are less polite with regards to robots.txt

if ($http_user_agent ~* "ClaudeBot") {
  return 444;
}

if ($http_user_agent ~* "[email protected]") {
  return 444;
}

#FuckAI #ClaudeBot #nginx

After getting alerted tonight because #claudebot from Anthropic was scanning a host so aggressively that all 20 cores where saturated I generated a list of IPs (all/mostly AWS) they used for you to block them too.

https://gist.github.com/leahoswald/935f90ba09b3484d15ea6d20d0f2f99a

The bot is used to fuel their AI model so nobody really needs that and after some research they also seem to ignore robots.txt. By by 👋 🤷‍♀️ #MastoOps

What fresh #AI hell is that?

grep claudebot *.log | grep -c robots.txt
3792

Why the heck is that crap requesting the robots.txt over and over again?

#Claude #ClaudeBot #ClaudeAI #robotstxt

My wife's website is currently under load from a big group of ip's in Amazon Data Services. User agents is shown as #ClaudeBot Does anybody know who is behind and what they want the texts from a Danish garden lover for?