#robotsdottxt — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #robotsdottxt, aggregated by home.social.
-
@compsci_discussions So what DOES one actually put in the robots.txt file? #webcrawler #robotsdottxt