#crawlers — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #crawlers, aggregated by home.social.
-
I tried to extract what's not personal: https://got.thinkberg.com/?action=summary&path=gotwebd-guard.git
It may be useful as a pattern for web crawlers in general.
-
I tried to extract what's not personal: https://got.thinkberg.com/?action=summary&path=gotwebd-guard.git
It may be useful as a pattern for web crawlers in general.
-
I tried to extract what's not personal: https://got.thinkberg.com/?action=summary&path=gotwebd-guard.git
It may be useful as a pattern for web crawlers in general.
-
I tried to extract what's not personal: https://got.thinkberg.com/?action=summary&path=gotwebd-guard.git
It may be useful as a pattern for web crawlers in general.
-
I tried to extract what's not personal: https://got.thinkberg.com/?action=summary&path=gotwebd-guard.git
It may be useful as a pattern for web crawlers in general.
-
The #crawlers are pretty annoying, especially when looking at irrelevant stuff digging deeper than necessary. Fortunately, in #OpenBSD using the #fail2ban pattern can be applied as well. My #GoT web server got hammered and first I just banned all the found crawler names. However, now, every action they do on gotwebd is remembered with an effort number and if that adds up to 100, the IP is banned. Additonally, connection storms are also banned if they follow certain patterns. #pf, #perl and I am done using only on-board tools.
Why? I didn't want to install #anubis. Not because I don't like it, it is just because I like to do the minimum.
-
The #crawlers are pretty annoying, especially when looking at irrelevant stuff digging deeper than necessary. Fortunately, in #OpenBSD using the #fail2ban pattern can be applied as well. My #GoT web server got hammered and first I just banned all the found crawler names. However, now, every action they do on gotwebd is remembered with an effort number and if that adds up to 100, the IP is banned. Additonally, connection storms are also banned if they follow certain patterns. #pf, #perl and I am done using only on-board tools.
Why? I didn't want to install #anubis. Not because I don't like it, it is just because I like to do the minimum.
-
The #crawlers are pretty annoying, especially when looking at irrelevant stuff digging deeper than necessary. Fortunately, in #OpenBSD using the #fail2ban pattern can be applied as well. My #GoT web server got hammered and first I just banned all the found crawler names. However, now, every action they do on gotwebd is remembered with an effort number and if that adds up to 100, the IP is banned. Additonally, connection storms are also banned if they follow certain patterns. #pf, #perl and I am done using only on-board tools.
Why? I didn't want to install #anubis. Not because I don't like it, it is just because I like to do the minimum.
-
The #crawlers are pretty annoying, especially when looking at irrelevant stuff digging deeper than necessary. Fortunately, in #OpenBSD using the #fail2ban pattern can be applied as well. My #GoT web server got hammered and first I just banned all the found crawler names. However, now, every action they do on gotwebd is remembered with an effort number and if that adds up to 100, the IP is banned. Additonally, connection storms are also banned if they follow certain patterns. #pf, #perl and I am done using only on-board tools.
Why? I didn't want to install #anubis. Not because I don't like it, it is just because I like to do the minimum.
-
The #crawlers are pretty annoying, especially when looking at irrelevant stuff digging deeper than necessary. Fortunately, in #OpenBSD using the fail2ban pattern can be applied as well. My #GoT web server got hammered and first I just banned all the found crawler names. However, now, every action they do on gotwebd is remembered with an effort number and if that adds up to 100, the IP is banned. Additonally, connection storms are also banned if they follow certain patterns. #pf, #perl and I am done using only on-board tools.
Why? I didn't want to install #anubis. Not because I don't like it, it is just because I like to do the minimum.
-
ICYMI: Microsoft Clarity now flags robots.txt violations inside Bot Analytics: Microsoft Clarity now surfaces robots.txt violations in Bot Analytics, showing publishers which AI crawlers break access rules and what content they target. https://ppc.land/microsoft-clarity-now-flags-robots-txt-violations-inside-bot-analytics/ #MicrosoftClarity #BotAnalytics #SEO #WebAnalytics #Crawlers
-
The initial problem is the aggressiveness of #LLM web #crawlers that don't respect "robots.txt". The first idea that comes to mind is IP #blocking . However, web crawlers have circumvented this restriction by using individual IPs via specialized #botnets .
Another solution is therefore to exhaust the resources of the harvesters. With a #zipbomb , we attempt to #exhaust their #RAM .
-
🚀 My new #DDoS book "DDoS: Understanding Real-Life Attacks and Mitigation Strategies" is now also available as an eBook! 🎉
Check it out here: https://ddos-book.com/
I’ve packed in everything I’ve learned from defending major German government sites against groups like Anonymous, Killnet, and NoName057(16).
It covers mitigations against #AI #crawlers and many other defenses for all network layers.
If you find it useful, I’d love it if you could boost and share to help more people defend themselves. ❤️
Thank you! 🙏
#DDoSProtection #NetworkSecurity #DDoS #RealWorldDefense #InfoSec #CyberSecurity #eBook #book
-
@Pastafari has asked for spider pictures. 💚 We therefore link to our blog article, which is exclusively about eight-legged #crawlers in the Costa Rican #rainforest. With lots of pictures. 🤓 For more #spider content. Have fun! 🕷️
#spiders #spinnen #arachnologie #arachnology #creepycrawlies
http://nerds-in-der-wildnis.de/achtbeiner-aus-der-gruenen-hoelle/