#webindexing — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #webindexing, aggregated by home.social.
-
Search Engine Journal: Indexed Claude Chats Show Why Disallow Is Not Noindex. “Anthropic told TechCrunch that share links will appear in search results only if people post them in accessible locations for crawlers, and that links sent through private messages won’t be indexed…. Google spokesperson Ned Adriance stated that search engines do not determine which pages are made public, and that […]
https://rbfirehose.com/2026/07/28/search-engine-journal-indexed-claude-chats-show-why-disallow-is-not-noindex/ -
ICYMI: Cloudflare gives OpenAI network signals covering 20% of the web: The pilot shares Cloudflare content-freshness and traffic data with OpenAI to test faster web indexing. No pricing or timeline was disclosed for wider rollout. https://ppc.land/cloudflare-gives-openai-network-signals-covering-20-of-the-web/ #Cloudflare #OpenAI #WebIndexing #ContentFreshness #DigitalMarketing
-
Cloudflare gives OpenAI network signals covering 20% of the web: The pilot shares Cloudflare content-freshness and traffic data with OpenAI to test faster web indexing. No pricing or timeline was disclosed for wider rollout. https://ppc.land/cloudflare-gives-openai-network-signals-covering-20-of-the-web/ #Cloudflare #OpenAI #WebIndexing #DigitalMarketing #SEO
-
Engadget: Cloudflare will filter out web crawlers that serve AI companies . “Cloudflare has announced plans to automatically block mixed-use web crawlers that index websites for search engines and act as AI agents and trainers at the same time. The company previously offered its customers the optional ability to prevent crawlers from scraping their sites for AI chatbots, but now Cloudflare’s […]
https://rbfirehose.com/2026/07/06/engadget-cloudflare-will-filter-out-web-crawlers-that-serve-ai-companies/ -
Search Engine Journal: Google Publishes New Robots.txt Explainer. “Google published a new Robots.txt refresher explaining how Robots.txt enables publishers and SEOs to control search engine crawlers and other bots (that obey Robots.txt). The documentation includes examples of blocking specific pages (like shopping carts), restricting certain bots, and managing crawling behavior with simple […]
https://rbfirehose.com/2025/03/13/search-engine-journal-google-publishes-new-robots-txt-explainer/
-
Search Engine Journal: SEO Data Suggests Google Indexing Rates Are Improving. “New research of over 16 million webpages shows that Google indexing rates have improved but that many pages in the dataset were not indexed and over 20% of the pages were eventually deindexed. The findings may be representative of trends and challenges that are specific to sites that are concerned about SEO and […]