#web-scraping — Public Fediverse posts

How Film Industry Data Website The-Numbers.com got Mauled by Bots

https://fed.brid.gy/r/https://hackaday.com/2026/07/25/how-film-industry-data-website-the-numbers-com-got-mauled-by-bots/

How Film Industry Data Website The-Numbers.com got Mauled by Bots

https://web.brid.gy/r/https://hackaday.com/2026/07/25/how-film-industry-data-website-the-numbers-com-got-mauled-by-bots/

Stephen Follows: What just happened to TheNumbers.com should worry us all. “Its hand-researched data is the highest quality, tracking box office grosses, budgets, home video and streaming across more than 78,000 films and 236,000 people. It gets north of eight million visitors a year, and is treated as THE definitive authority by journalists, academics, filmmakers, prediction markets, and even […]

Stephen Follows: What just happened to TheNumbers.com should worry us all. “Its hand-researched data is the highest quality, tracking box office grosses, budgets, home video and streaming across more than 78,000 films and 236,000 people. It gets north of eight million visitors a year, and is treated as THE definitive authority by journalists, academics, filmmakers, prediction markets, and even […]

Reuters: News Corp countersues Brave for allegedly ‘scraping’ articles for AI . “News Corp, facing a lawsuit by search engine Brave Software, has filed a countersuit accusing it ‌of “flagrant theft” in distributing and selling versions of articles from the Wall Street Journal and New York Post to AI companies.”

Reuters: News Corp countersues Brave for allegedly ‘scraping’ articles for AI . “News Corp, facing a lawsuit by search engine Brave Software, has filed a countersuit accusing it ‌of “flagrant theft” in distributing and selling versions of articles from the Wall Street Journal and New York Post to AI companies.”

Turns out you can't just ask an LLM for CSS selectors and ship them. In our scraping system, first-attempt selectors returned nothing 30 to 40% of the time. The trick that made it work: check JSON-LD first, then run every generated selector through a validation loop against the real DOM before trusting it. https://go.upgradejs.com/qru #LLM #WebScraping #AI

Turns out you can't just ask an LLM for CSS selectors and ship them. In our scraping system, first-attempt selectors returned nothing 30 to 40% of the time. The trick that made it work: check JSON-LD first, then run every generated selector through a validation loop against the real DOM before trusting it. https://go.upgradejs.com/qru #LLM #WebScraping #AI

MediaPost: Judge Dismisses Google Complaint Against SerpApi Over Scraping. “A federal judge has dismissed Google’s complaint against the Texas-based company SerpApi, which allegedly circumvented attempts to prevent it from scraping search results. The ruling, issued Monday by U.S. District Court Judge Yvonne Gonzalez Rogers, allows Google to amend its complaint and bring it again.”

MediaPost: Judge Dismisses Google Complaint Against SerpApi Over Scraping. “A federal judge has dismissed Google’s complaint against the Texas-based company SerpApi, which allegedly circumvented attempts to prevent it from scraping search results. The ruling, issued Monday by U.S. District Court Judge Yvonne Gonzalez Rogers, allows Google to amend its complaint and bring it again.”

🕷️ Anakin-Inc/anakin

Converts websites to clean markdown or JSON with fallback scraping handlers and proxy auto-selection

⭐ Stars: 687
📅 Last Update: Jul 20, 2026

https://github.com/Anakin-Inc/anakin

#selfhosted #homelab #selfhost #selfhosting #opensource #webscraping #api

Mixfont: Decoy Font. “Decoy font is a font that prints a decoy for every letter, making it more difficult for AI to read what you type. The font works by using separate spatial frequencies to communicate two different letters in the same space.”

Mixfont: Decoy Font. “Decoy font is a font that prints a decoy for every letter, making it more difficult for AI to read what you type. The font works by using separate spatial frequencies to communicate two different letters in the same space.”

Launch HN: Context.dev (YC S26) – API to get structured data from any website

https://www.context.dev

Comments: https://news.ycombinator.com/item?id=48847562

#HackerNews #LaunchHN #ContextDev #API #WebScraping #YC #S26

Launch HN: Context.dev (YC S26) – API to get structured data from any website

https://www.context.dev

Comments: https://news.ycombinator.com/item?id=48847562

#HackerNews #LaunchHN #ContextDev #API #WebScraping #YC #S26

Engadget: Cloudflare will filter out web crawlers that serve AI companies . “Cloudflare has announced plans to automatically block mixed-use web crawlers that index websites for search engines and act as AI agents and trainers at the same time. The company previously offered its customers the optional ability to prevent crawlers from scraping their sites for AI chatbots, but now Cloudflare’s […]

Engadget: Cloudflare will filter out web crawlers that serve AI companies . “Cloudflare has announced plans to automatically block mixed-use web crawlers that index websites for search engines and act as AI agents and trainers at the same time. The company previously offered its customers the optional ability to prevent crawlers from scraping their sites for AI chatbots, but now Cloudflare’s […]

RT @NousResearch: Der Hermes-Agent liest die Webinhalte nun bis zu 60-mal schneller und zu 49-mal niedrigeren Kosten. Scraping-Backends übergeben saubere Inhalte direkt an den Agenten, ohne redundante Verarbeitungsschritte; große Seiten werden lokal gespeichert und bei Bedarf seitenweise abgerufen, sodass Sie die gleiche Qualität zu einem Bruchteil der Zeit und Kosten erhalten. Video

mehr auf Arint.info

#Effizienz #HermesAgent #Kostensenkung #Video #WebScraping #arint_info

https://x.com/NousResearch/status/2071974594961977727#m

I tested every way to scrape Amazon in 2026 — plain requests, Selenium, Playwright, free proxies, paid proxies. https://hackernoon.com/i-tried-every-way-to-scrape-amazon-in-2026-here-is-what-actually-works #webscraping

I tested every way to scrape Amazon in 2026 — plain requests, Selenium, Playwright, free proxies, paid proxies. https://hackernoon.com/i-tried-every-way-to-scrape-amazon-in-2026-here-is-what-actually-works #webscraping

RT @NousResearch: Der Hermes-Agent liest das Web nun bis zu 60-mal schneller und 49-mal günstiger. Scraping-Backends übergeben saubere Inhalte direkt an den Agenten ohne redundante Verarbeitungsschritte; große Seiten werden lokal gespeichert und bei Bedarf aufgerufen, sodass Sie die gleiche Qualität zu einem Bruchteil der Zeit und Kosten erhalten. Video

mehr auf Arint.info

#HermesAgent #Kosteneffizienz #Performance #Technologie #WebScraping #arint_info

https://x.com/NousResearch/status/2071974594961977727#m

Why configure your AI Model Harness? #webscraping #podcast #ai #programming https://www.youtube.com/watch?v=xdVcK-XfxLo?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

Why configure your AI Model Harness? #webscraping #podcast #ai #programming https://www.youtube.com/watch?v=xdVcK-XfxLo?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

Keep your context window clean using these #webscraping #ai #podcast https://www.youtube.com/watch?v=wNbLzW1huZM?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

Keep your context window clean using these #webscraping #ai #podcast https://www.youtube.com/watch?v=wNbLzW1huZM?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

New Jersey Globe: Nearly 400 local newspapers sue OpenAI, Microsoft over alleged copyright theft. “The massive coalition of local newspaper publishers filed a federal lawsuit today against OpenAI and Microsoft, alleging the technology companies systematically copied copyrighted reporting from nearly 400 local newspapers to train and develop commercial artificial intelligence products, including […]

New Jersey Globe: Nearly 400 local newspapers sue OpenAI, Microsoft over alleged copyright theft. “The massive coalition of local newspaper publishers filed a federal lawsuit today against OpenAI and Microsoft, alleging the technology companies systematically copied copyrighted reporting from nearly 400 local newspapers to train and develop commercial artificial intelligence products, including […]

Search Engine Journal: US Publishers Demand Common Crawl Stop Scraping Their Content. “Digital Content Next, a trade body representing US digital publishers, has sent a cease and desist letter to the Common Crawl Foundation. The letter demands Common Crawl stop collecting publisher content and remove material already in its datasets.”

Search Engine Journal: US Publishers Demand Common Crawl Stop Scraping Their Content. “Digital Content Next, a trade body representing US digital publishers, has sent a cease and desist letter to the Common Crawl Foundation. The letter demands Common Crawl stop collecting publisher content and remove material already in its datasets.”

Web scraping gets blocked by weak headers, broken sessions, poor IP reputation, fast requests, and careless proxy rotation. https://hackernoon.com/why-scrapers-fail-headers-sessions-ip-reputation-and-request-patterns #webscraping

Web scraping gets blocked by weak headers, broken sessions, poor IP reputation, fast requests, and careless proxy rotation. https://hackernoon.com/why-scrapers-fail-headers-sessions-ip-reputation-and-request-patterns #webscraping

No-one likes an out-of-touch AI assistant. Fortunately, rapid refreshing can keep AI models aware of the very latest public information. https://www.zyte.com/blog/enhancing-ai-model-performance-with-fresh-web-data?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

🎉 Behold, the future of browser automation, where writing code is for peasants! 🤖 Intuned claims to solve all your web scraping woes with magic #AI that doesn't need you, unless it breaks. But don't worry, you have a whole smorgasbord of buzzwords like "Playwright" and "RPAAI" to keep you entertained while you wait. 🚀✨
https://intunedhq.com #browserautomation #webscraping #Playwright #RPAAI #HackerNews #ngated

🎉 Behold, the future of browser automation, where writing code is for peasants! 🤖 Intuned claims to solve all your web scraping woes with magic #AI that doesn't need you, unless it breaks. But don't worry, you have a whole smorgasbord of buzzwords like "Playwright" and "RPAAI" to keep you entertained while you wait. 🚀✨
https://intunedhq.com #browserautomation #webscraping #Playwright #RPAAI #HackerNews #ngated

Released hydrascrape — a free, self-healing scraper for sites behind Imperva/Distil bot walls.

patched-Chrome (patchright) clears the JS challenge; a fleet of Tor exits beats per-IP limits and auto-rotates the bad ones. Resumable, with a live dashboard. Proven on a ~58k-page catalog from one laptop, $0.

Full "what I tried and why" writeup included. MIT.

🔗 github.com/philipposk/hydrascrape
#OpenSource #WebScraping #Tor #Python #SelfHosting

When you can scrape the web by API, a world of possibility opens up. Yes, you can extract live web data using iOS Shortcuts. https://www.zyte.com/blog/web-scraping-on-an-iphone?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

Developers are embracing agentic coding tools - but data engineers need tools with specialist scraping skills. https://www.zyte.com/blog/introducing-zyte-web-data-for-claude-code?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

Я сошёл с ума и сдаю свой браузер ИИ-агентам

Я совсем поехал кукухой — начал сдавать в аренду свой браузер за деньги. Началось всё с того, что мои ИИ-агенты не смогли нормально зарегаться из-за капчей и прочего, чужие расширения меня не устраивали — они плохо интегрировались в мой флоу и были завязаны на провайдера, что полный отстой. В итоге я интегрировал это в свой пет-проект, и в итоге сделал так, что браузер в аренду может взять любой желающий. Заодно сделал SDK, CLI и доки. Вот моя история погружения в пучину безумия. Погрузиться в пучину.

https://habr.com/ru/articles/1041768/

#aiагенты #browserautomation #chromeрасширения #mcp #петпроект #mcpserver #llm #webscraping #антидетект #криптоплатежи