home.social

#crawlers — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #crawlers, aggregated by home.social.

  1. A valid bomb

    The initial problem is the aggressiveness of web that don't respect "robots.txt". The first idea that comes to mind is IP . However, web crawlers have circumvented this restriction by using individual IPs via specialized .

    Another solution is therefore to exhaust the resources of the harvesters. With a , we attempt to their .

    💭 ache.one/notes/html_zip_bomb

  2. @Pastafari has asked for spider pictures. 💚 We therefore link to our blog article, which is exclusively about eight-legged #crawlers in the Costa Rican #rainforest. With lots of pictures. 🤓 For more #spider content. Have fun! 🕷️
    #spiders #spinnen #arachnologie #arachnology #creepycrawlies
    nerds-in-der-wildnis.de/achtbe