home.social

#crawlers — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #crawlers, aggregated by home.social.

  1. Yesterday I started blocking all non-European IP addresses on our Gitlab instance because of the huge flood of AI scrapers. Sad that we have to do these kind of things

    #gitlab #crawlers #bots #AI

  2. Yesterday I started blocking all non-European IP addresses on our Gitlab instance because of the huge flood of AI scrapers. Sad that we have to do these kind of things

  3. September 15, 2026, #Cloudflare will set updated defaults for new domains: #bots classified as #Training or #Agent will be #blocked on pages that display ads, and Search will remain allowed. #Crawlers that combine Search&Training will also be blocked. #AI

    developers.cloudflare.com/bots

  4. RE: social.edu.nl/@wlaatje/1169708

    "The web is full of independent archives, hobby databases, local news sites, forums, reference works. Decades of accumulated human effort, running on old code, maintained by small teams or single individuals, quietly holding up far more of our shared knowledge than anyone acknowledges."

    And proponents of so-called 'AI' are speedrunning their destruction.

    #noAI #scrapers #scraping #crawlers #AI #genAI

  5. ICYMI: Microsoft Clarity now flags robots.txt violations inside Bot Analytics: Microsoft Clarity now surfaces robots.txt violations in Bot Analytics, showing publishers which AI crawlers break access rules and what content they target. ppc.land/microsoft-clarity-now #MicrosoftClarity #BotAnalytics #SEO #WebAnalytics #Crawlers

  6. 🚀 My new #DDoS book "DDoS: Understanding Real-Life Attacks and Mitigation Strategies" is now also available as an eBook! 🎉

    Check it out here: ddos-book.com/

    I’ve packed in everything I’ve learned from defending major German government sites against groups like Anonymous, Killnet, and NoName057(16).

    It covers mitigations against #AI #crawlers and many other defenses for all network layers.

    If you find it useful, I’d love it if you could boost and share to help more people defend themselves. ❤️

    Thank you! 🙏

    #DDoSProtection #NetworkSecurity #DDoS #RealWorldDefense #InfoSec #CyberSecurity #eBook #book