#crawlers — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #crawlers, aggregated by home.social.
-
I tried to extract what's not personal: https://got.thinkberg.com/?action=summary&path=gotwebd-guard.git
It may be useful as a pattern for web crawlers in general.
-
The #crawlers are pretty annoying, especially when looking at irrelevant stuff digging deeper than necessary. Fortunately, in #OpenBSD using the #fail2ban pattern can be applied as well. My #GoT web server got hammered and first I just banned all the found crawler names. However, now, every action they do on gotwebd is remembered with an effort number and if that adds up to 100, the IP is banned. Additonally, connection storms are also banned if they follow certain patterns. #pf, #perl and I am done using only on-board tools.
Why? I didn't want to install #anubis. Not because I don't like it, it is just because I like to do the minimum.
-
The #crawlers are pretty annoying, especially when looking at irrelevant stuff digging deeper than necessary. Fortunately, in #OpenBSD using the #fail2ban pattern can be applied as well. My #GoT web server got hammered and first I just banned all the found crawler names. However, now, every action they do on gotwebd is remembered with an effort number and if that adds up to 100, the IP is banned. Additonally, connection storms are also banned if they follow certain patterns. #pf, #perl and I am done using only on-board tools.
Why? I didn't want to install #anubis. Not because I don't like it, it is just because I like to do the minimum.
-
The #crawlers are pretty annoying, especially when looking at irrelevant stuff digging deeper than necessary. Fortunately, in #OpenBSD using the #fail2ban pattern can be applied as well. My #GoT web server got hammered and first I just banned all the found crawler names. However, now, every action they do on gotwebd is remembered with an effort number and if that adds up to 100, the IP is banned. Additonally, connection storms are also banned if they follow certain patterns. #pf, #perl and I am done using only on-board tools.
Why? I didn't want to install #anubis. Not because I don't like it, it is just because I like to do the minimum.
-
The #crawlers are pretty annoying, especially when looking at irrelevant stuff digging deeper than necessary. Fortunately, in #OpenBSD using the #fail2ban pattern can be applied as well. My #GoT web server got hammered and first I just banned all the found crawler names. However, now, every action they do on gotwebd is remembered with an effort number and if that adds up to 100, the IP is banned. Additonally, connection storms are also banned if they follow certain patterns. #pf, #perl and I am done using only on-board tools.
Why? I didn't want to install #anubis. Not because I don't like it, it is just because I like to do the minimum.
-
The #crawlers are pretty annoying, especially when looking at irrelevant stuff digging deeper than necessary. Fortunately, in #OpenBSD using the fail2ban pattern can be applied as well. My #GoT web server got hammered and first I just banned all the found crawler names. However, now, every action they do on gotwebd is remembered with an effort number and if that adds up to 100, the IP is banned. Additonally, connection storms are also banned if they follow certain patterns. #pf, #perl and I am done using only on-board tools.
Why? I didn't want to install #anubis. Not because I don't like it, it is just because I like to do the minimum.
-
#Business #Launches
Cloudflare Radar’s crawl-to-refer ratio · Monitor AI crawls vs. real website traffic sent https://ilo.im/1652pb_____
#SEO #AnswerEngines #AI #Crawlers #Referrals #Content #Website #Development #WebDev #Frontend -
How crawlers impact the operations of the Wikimedia projects
https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/
via #nonostantement