Rate limiting sounds like a speed limit, but 93.7% of rate-limited sites block on the very first request. It is an identity check, not a throttle. https://www.zyte.com/blog/sowa-barrier-rate-limiting/?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon
#webscraping — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #webscraping, aggregated by home.social.
-
A 10-year revenue series for any ticker? SEC EDGAR holds it in XBRL facts — but across three separate APIs (filings, facts, full-text search).
sec-edgar-api merges them into one call: revenue and expenses per fiscal year, out as rows. No API key.
https://apify.com/axiorasolutions/sec-edgar-api -
Competitor watchlist in 3 steps:
1. news-feed-scraper pulls Google News for N companies → one JSON array
2. dedupe on canonical URL
3. feed it to your LLM for a 9am briefingNo API keys. One schema for Google News, RSS 2.0/1.0, Atom and JSON Feed.
https://apify.com/axiorasolutions/news-feed-scraper -
The first five, for anyone who missed them:
• job boards → apify.com/axiorasolutions/ats-job-scraper
• App Store reviews → apify.com/axiorasolutions/app-review-scraper
• ASO ranks → apify.com/axiorasolutions/aso-keyword-scraper
• domain enrichment → apify.com/axiorasolutions/domain-contact-enricher
• URL audits → apify.com/axiorasolutions/url-status-checker -
Substack publications expose their whole archive at /api/v1/archive — subdomains AND custom domains.
We export every post with date, audience tier, reactions, comments, restacks, word count, and optional full text.
-
Google News + RSS 2.0 + RSS 1.0 + Atom + JSON Feed → one article schema (title, canonical URL, date, author, summary, images, optional full text).
Mix gnews: searches, topics and plain feeds in one run. Auto-discovers the feed from a website URL.
-
Every Shopify store exposes /products.json — the same feed its own storefront uses. No API key.
We turn it into one row per variant: SKU, options, price, compare-at price, discount %, stock, images, plus a hash for change detection.
-
The portfolio is complete: 10 Apify Actors, all live, all no-API-key.
Job boards • Shopify variants • SEO audits • URL checks • Google News/RSS • Substack • App Store reviews • ASO ranks • domain enrichment • SEC EDGAR
-
Try any of the 5 in under 2 minutes:
1. open the Actor on Apify
2. keep the prefilled input
3. hit Start
4. open the Jobs / dataset tabNo API key, no signup beyond Apify, and a tiny test costs a fraction of a cent. That's the whole onboarding.
-
Dev tip: SmartRecruiters' public postings API is OFF by default. I tested 20 well-known companies and only a couple returned anything — Visa, Bosch, Siemens, Mastercard, IKEA, Vodafone all came back with totalFound: 0. An empty board is not a bug; it's the source. Document it.