#ccbot — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #ccbot, aggregated by home.social.
-
Robots.txt, OpenAI’s GPTBot, Common Crawl’s CCBot: How to block AI crawlers from gathering text and images from your website: https://katharinabrunner.de/2023/08/robots-txt-openais-gptbot-common-crawls-ccbot-how-to-block-ai-crawlers-from-gathering-text-and-images-from-your-website/
#ai #openAI #crawler #commoncrawl #ccbot #GPTBot #robotstxt #wordpress