#crawlerbacklash — Public Fediverse posts

#Robotstxt #CrawlerBacklash Trickle-down effects: "people start blocking all crawlers, and some crawlers are very important, for search indexing, internet archiving, some are used for academic research, and so the bad behaviours of all these #AIcompanies, and the backlash to it, is kind of fundamentally changing how the Internet works, how it is remembered and indexed..."
https://pca.st/yto6v3il?t=11m34s