home.social

#redditbots — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #redditbots, aggregated by home.social.

  1. Reddit has updated its robots.txt to block all web crawlers

    Bing, DuckDuckGo, Yandex, Yahoo, and other search engines have not indexed any pages from Reddit in the past 24 hours.

    Google, on the other hand, still keeps indexing pages.

    stackdiary.com/reddit-has-upda

    #Reddit #WebCrawlers #DataPrivacy #SEO #DataProtection #WebScraping #InternetArchive #OpenAI #PrivacyPolicy #DataControl #SearchEngine #DataMining #RedditBots #PublicContent #TechNews #OnlinePrivacy #DataSecurity