#exampledotnet — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #exampledotnet, aggregated by home.social.
-
Is there a website like https://example.com/, but has multiple URLs and is OK to spider/crawl?
I want to upgrade the examples in spidr's README to websites that wouldn't mind being randomly spidered.
https://github.com/postmodern/spidr#readmehttps://example.com isn't a viable option, because it only has one webpage, so not a very good demonstration of web spidering...
#web #webspiders #webcrawlers #exampledotcom #exampledotnet #demo