home.social

#crawlers — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #crawlers, aggregated by home.social.

  1. Frehi @[email protected] ·

    Yesterday I started blocking all non-European IP addresses on our Gitlab instance because of the huge flood of AI scrapers. Sad that we have to do these kind of things

    #gitlab #crawlers #bots #AI

    #gitlab #crawlers #bots #ai
  2. Frehi @frehi ·

    Yesterday I started blocking all non-European IP addresses on our Gitlab instance because of the huge flood of AI scrapers. Sad that we have to do these kind of things

    #gitlab #crawlers #bots #ai
  3. Wilfredo 💬 @wilfredo ·

    A valid bomb

    The initial problem is the aggressiveness of web that don't respect "robots.txt". The first idea that comes to mind is IP . However, web crawlers have circumvented this restriction by using individual IPs via specialized .

    Another solution is therefore to exhaust the resources of the harvesters. With a , we attempt to their .

    💭 ache.one/notes/html_zip_bomb

    #html #zip #llm #crawlers #blocking #botnets