home.social

#rfc9309 — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #rfc9309, aggregated by home.social.

fetched live
  1. Inautilo @[email protected] ·

    #Development #Obituaries
    Farewell to robots.txt (1994-2025) · “You were too good for this world.” ilo.im/167q2b

    _____
    #SearchEngine #InternetArchive #AI #Content #Website #RobotsTxt #RFC9309 #WebDev #Backend

    #development #obituaries #searchengine #internetarchive #ai #content
  2. Inautilo @[email protected] ·

    #Development #Obituaries
    Farewell to robots.txt (1994-2025) · “You were too good for this world.” ilo.im/167q2b

    _____
    #SearchEngine #InternetArchive #AI #Content #Website #RobotsTxt #RFC9309 #WebDev #Backend

    #development #obituaries #searchengine #internetarchive #ai #content
  3. PointlessOne :loading: @[email protected] ·

    I’ve made a little something, so I thought I'd share.

    Gort is a robots.txt parser and evaluator. It implements RFC 9309.

    More details in the ReadMe: github.com/pointlessone/gort

    #Ruby #rubygem #release #robotstxt #robots_txt #rfc9309

    #ruby #rubygem #release #robotstxt #robots_txt #rfc9309
  4. Ross A. Baker @[email protected] ·

    Setting up /robots.txt, not because it helps, but because being crabby in compliance with an RFC is satisfying.

    Who has some unsavory ones besides ChatGPT and Twitterbot?

    rossabaker.com/configs/website

    #RFC9309 #RobotsTxt

    #rfc9309 #robotstxt
  5. Ross A. Baker @[email protected] ·

    Setting up /robots.txt, not because it helps, but because being crabby in compliance with an RFC is satisfying.

    Who has some unsavory ones besides ChatGPT and Twitterbot?

    rossabaker.com/configs/website

    #RFC9309 #RobotsTxt

    #rfc9309 #robotstxt