#webspiders — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #webspiders, aggregated by home.social.
-
Is there a website like https://example.com/, but has multiple URLs and is OK to spider/crawl?
I want to upgrade the examples in spidr's README to websites that wouldn't mind being randomly spidered.
https://github.com/postmodern/spidr#readmehttps://example.com isn't a viable option, because it only has one webpage, so not a very good demonstration of web spidering...
#web #webspiders #webcrawlers #exampledotcom #exampledotnet #demo
-
** not all homes look like you imagine ** 🕷️🕸️🪨🕷️
#photography #rocks #nature #blackandwhitephotography #cobwebs #spiders #natureisart #darktable #monochrome #texture #webspiders #photographer #photooftheday