ICYMI: Cloudflare drops AI opt-outs from 118 sites' robots.txt, HasData finds: Of 592 sites declaring a GPTBot ban, 234 still served it a live page, while publishers turn away AI bots at 56.4%. Has enforcement moved from files to CDNs? https://ppc.land/cloudflare-drops-ai-opt-outs-from-118-sites-robots-txt-hasdata-finds/ #Cloudflare #AI #digitalmarketing #robots_txt #GPTBot
#robots_txt — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #robots_txt, aggregated by home.social.
-
Oh wow, #OpenAI is #scraping #CT #logs like a kid in a candy store 🍬. Apparently, they're on a mission to hunt down... robots.txt files? 🤖🗂️ Because who doesn't love a treasure trove of 404 errors and TLS certificates? 💾🔍
https://benjojo.co.uk/u/benjojo/h/Gxy2qrCkn1Y327Y6D3 #robots_txt #404_errors #TLS_certificates #tech_news #HackerNews #ngated -
Thinking about your robots.txt file? It might seem counterintuitive, but disallowing RSS feeds and certain pagination paths can be a smart SEO move.
This technique helps search engines focus crawl budgets on your most important pages to avoid potential duplicate content.
This post on WebHeads United looks at the technical reasons behind this strategy and whether it's right for your site.
Read the SEO deep dive: https://webheadsunited.com/why-disallow-rss-feeds-and-pagination/
-
Please notice that I've upvoted that comment. I want upvotes for my comment, too.
-
I’ve made a little something, so I thought I'd share.
Gort is a robots.txt parser and evaluator. It implements RFC 9309.
More details in the ReadMe: https://github.com/pointlessone/gort
-
My local government just launched a site redesign, changing CMSes and permalink structures.
They didn't set up redirects for old URLs.
Half the site is still blocked in robots.txt.
I'm professionally flabbergasted.
-
#Development #Initiatives
Google to explore alternatives to robots.txt · Generative AI would require new machine-readable methods https://ilo.im/13z0t1_____
#AI #GenerativeAI #ChatBots #BotAccess #Website #WebDevelopment #WebDev #Community #Discussion #Protocol #Robots_txt
