#web-scraping — Public Fediverse posts

Launch HN: Context.dev (YC S26) – API to get structured data from any website

https://www.context.dev

Comments: https://news.ycombinator.com/item?id=48847562

#HackerNews #LaunchHN #ContextDev #API #WebScraping #YC #S26

Launch HN: Context.dev (YC S26) – API to get structured data from any website

https://www.context.dev

Comments: https://news.ycombinator.com/item?id=48847562

#HackerNews #LaunchHN #ContextDev #API #WebScraping #YC #S26

Engadget: Cloudflare will filter out web crawlers that serve AI companies . “Cloudflare has announced plans to automatically block mixed-use web crawlers that index websites for search engines and act as AI agents and trainers at the same time. The company previously offered its customers the optional ability to prevent crawlers from scraping their sites for AI chatbots, but now Cloudflare’s […]

Engadget: Cloudflare will filter out web crawlers that serve AI companies . “Cloudflare has announced plans to automatically block mixed-use web crawlers that index websites for search engines and act as AI agents and trainers at the same time. The company previously offered its customers the optional ability to prevent crawlers from scraping their sites for AI chatbots, but now Cloudflare’s […]

RT @NousResearch: Der Hermes-Agent liest die Webinhalte nun bis zu 60-mal schneller und zu 49-mal niedrigeren Kosten. Scraping-Backends übergeben saubere Inhalte direkt an den Agenten, ohne redundante Verarbeitungsschritte; große Seiten werden lokal gespeichert und bei Bedarf seitenweise abgerufen, sodass Sie die gleiche Qualität zu einem Bruchteil der Zeit und Kosten erhalten. Video

mehr auf Arint.info

#Effizienz #HermesAgent #Kostensenkung #Video #WebScraping #arint_info

https://x.com/NousResearch/status/2071974594961977727#m

I tested every way to scrape Amazon in 2026 — plain requests, Selenium, Playwright, free proxies, paid proxies. https://hackernoon.com/i-tried-every-way-to-scrape-amazon-in-2026-here-is-what-actually-works #webscraping

I tested every way to scrape Amazon in 2026 — plain requests, Selenium, Playwright, free proxies, paid proxies. https://hackernoon.com/i-tried-every-way-to-scrape-amazon-in-2026-here-is-what-actually-works #webscraping

RT @NousResearch: Der Hermes-Agent liest das Web nun bis zu 60-mal schneller und 49-mal günstiger. Scraping-Backends übergeben saubere Inhalte direkt an den Agenten ohne redundante Verarbeitungsschritte; große Seiten werden lokal gespeichert und bei Bedarf aufgerufen, sodass Sie die gleiche Qualität zu einem Bruchteil der Zeit und Kosten erhalten. Video

mehr auf Arint.info

#HermesAgent #Kosteneffizienz #Performance #Technologie #WebScraping #arint_info

https://x.com/NousResearch/status/2071974594961977727#m

Why configure your AI Model Harness? #webscraping #podcast #ai #programming https://www.youtube.com/watch?v=xdVcK-XfxLo?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

Why configure your AI Model Harness? #webscraping #podcast #ai #programming https://www.youtube.com/watch?v=xdVcK-XfxLo?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

Keep your context window clean using these #webscraping #ai #podcast https://www.youtube.com/watch?v=wNbLzW1huZM?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

Keep your context window clean using these #webscraping #ai #podcast https://www.youtube.com/watch?v=wNbLzW1huZM?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

New Jersey Globe: Nearly 400 local newspapers sue OpenAI, Microsoft over alleged copyright theft. “The massive coalition of local newspaper publishers filed a federal lawsuit today against OpenAI and Microsoft, alleging the technology companies systematically copied copyrighted reporting from nearly 400 local newspapers to train and develop commercial artificial intelligence products, including […]

New Jersey Globe: Nearly 400 local newspapers sue OpenAI, Microsoft over alleged copyright theft. “The massive coalition of local newspaper publishers filed a federal lawsuit today against OpenAI and Microsoft, alleging the technology companies systematically copied copyrighted reporting from nearly 400 local newspapers to train and develop commercial artificial intelligence products, including […]

Search Engine Journal: US Publishers Demand Common Crawl Stop Scraping Their Content. “Digital Content Next, a trade body representing US digital publishers, has sent a cease and desist letter to the Common Crawl Foundation. The letter demands Common Crawl stop collecting publisher content and remove material already in its datasets.”

Search Engine Journal: US Publishers Demand Common Crawl Stop Scraping Their Content. “Digital Content Next, a trade body representing US digital publishers, has sent a cease and desist letter to the Common Crawl Foundation. The letter demands Common Crawl stop collecting publisher content and remove material already in its datasets.”

Web scraping gets blocked by weak headers, broken sessions, poor IP reputation, fast requests, and careless proxy rotation. https://hackernoon.com/why-scrapers-fail-headers-sessions-ip-reputation-and-request-patterns #webscraping

Web scraping gets blocked by weak headers, broken sessions, poor IP reputation, fast requests, and careless proxy rotation. https://hackernoon.com/why-scrapers-fail-headers-sessions-ip-reputation-and-request-patterns #webscraping

No-one likes an out-of-touch AI assistant. Fortunately, rapid refreshing can keep AI models aware of the very latest public information. https://www.zyte.com/blog/enhancing-ai-model-performance-with-fresh-web-data?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

🎉 Behold, the future of browser automation, where writing code is for peasants! 🤖 Intuned claims to solve all your web scraping woes with magic #AI that doesn't need you, unless it breaks. But don't worry, you have a whole smorgasbord of buzzwords like "Playwright" and "RPAAI" to keep you entertained while you wait. 🚀✨
https://intunedhq.com #browserautomation #webscraping #Playwright #RPAAI #HackerNews #ngated

🎉 Behold, the future of browser automation, where writing code is for peasants! 🤖 Intuned claims to solve all your web scraping woes with magic #AI that doesn't need you, unless it breaks. But don't worry, you have a whole smorgasbord of buzzwords like "Playwright" and "RPAAI" to keep you entertained while you wait. 🚀✨
https://intunedhq.com #browserautomation #webscraping #Playwright #RPAAI #HackerNews #ngated

Released hydrascrape — a free, self-healing scraper for sites behind Imperva/Distil bot walls.

patched-Chrome (patchright) clears the JS challenge; a fleet of Tor exits beats per-IP limits and auto-rotates the bad ones. Resumable, with a live dashboard. Proven on a ~58k-page catalog from one laptop, $0.

Full "what I tried and why" writeup included. MIT.

🔗 github.com/philipposk/hydrascrape
#OpenSource #WebScraping #Tor #Python #SelfHosting

When you can scrape the web by API, a world of possibility opens up. Yes, you can extract live web data using iOS Shortcuts. https://www.zyte.com/blog/web-scraping-on-an-iphone?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

Developers are embracing agentic coding tools - but data engineers need tools with specialist scraping skills. https://www.zyte.com/blog/introducing-zyte-web-data-for-claude-code?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

Я сошёл с ума и сдаю свой браузер ИИ-агентам

Я совсем поехал кукухой — начал сдавать в аренду свой браузер за деньги. Началось всё с того, что мои ИИ-агенты не смогли нормально зарегаться из-за капчей и прочего, чужие расширения меня не устраивали — они плохо интегрировались в мой флоу и были завязаны на провайдера, что полный отстой. В итоге я интегрировал это в свой пет-проект, и в итоге сделал так, что браузер в аренду может взять любой желающий. Заодно сделал SDK, CLI и доки. Вот моя история погружения в пучину безумия. Погрузиться в пучину.

https://habr.com/ru/articles/1041768/

#aiагенты #browserautomation #chromeрасширения #mcp #петпроект #mcpserver #llm #webscraping #антидетект #криптоплатежи

Your VPS is ready, but now you need to work through the same sequence you have run a dozen times before: apt update, apt install python3-pip, pip install scrapy, playwright install chromium, the Chromium dependency list that never installs cleanly on the first try, Redis, possibly Postgres, whatever else this particular project needs. https://www.zyte.com/blog/automate-deployment-of-your-web-scraper-on-vps-with-ubuntu-24-04-cloud-init?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

Multi-agent orchestration is having its moment. The diagrams are everywhere now. Boxes for planners, boxes for hands, boxes for daemons, arrows to a shared brain, a human floating at the top. They keep getting prettier. The part where the web pushes back is still the part nobody draws. https://www.zyte.com/blog/multi-agent-orchestration-in-a-large-scale-web-scraping-project?utm_campaign=blog-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

@christianschwaegerl @amnesty_digital_de

wer sagts der politik ? bzw. kontrolliert die Verfahren ?

der #Papst_Leo_XIV hatte ja diesbezüglich schon einen Aufschlag gemacht.

@Rainer_Rehak
hat nen artikel in #EnergieZukunft ( https://www.energiezukunft.eu/wirtschaft/ki-und-klimaschutz-eine-bestandsaufnahme )

zum thema .

---

@okfde
@AwetTesfaiesus
ist das Thema in der Politik präsent und auch inhaltlich angekommen / technisch korrekt von Entscheidern rezipiert ?

besonders dieses #webscraping zieht auch in aderen kontexten u.a. wohl auch alle daten für #Cybercrime-Angriffe und verteilt sie an geeignete Stelle ?!?

insb. für bildung und wiss-komm bzw. auch recht&ordnung .. relevant
#edu -> @bpb
@Orkan_der_rechtspflege

#KI
#AI #UnlawfulWebscraping

#AmnestyInternational

@christianschwaegerl @amnesty_digital_de

wer sagts der politik ? bzw. kontrolliert die Verfahren ?

der #Papst_Leo_XIV hatte ja diesbezüglich schon einen Aufschlag gemacht.

@Rainer_Rehak
hat nen artikel in #EnergieZukunft ( https://www.energiezukunft.eu/wirtschaft/ki-und-klimaschutz-eine-bestandsaufnahme )

zum thema .

---

@okfde
@AwetTesfaiesus
ist das Thema in der Politik präsent und auch inhaltlich angekommen / technisch korrekt von Entscheidern rezipiert ?

besonders dieses #webscraping zieht auch in aderen kontexten u.a. wohl auch alle daten für #Cybercrime-Angriffe und verteilt sie an geeignete Stelle ?!?

insb. für bildung und wiss-komm bzw. auch recht&ordnung .. relevant
#edu -> @bpb
@Orkan_der_rechtspflege

#KI
#AI #UnlawfulWebscraping

#AmnestyInternational

🔍 #Firecrawl /monitor: keep your #AI #agent in sync with the web.
Point it at a URL, describe changes in plain English, get a webhook or email when something meaningful changes ☝️ #webscraping #devtools

https://docs.firecrawl.dev/features/monitoring

🔍 #Firecrawl /monitor: keep your #AI #agent in sync with the web.
Point it at a URL, describe changes in plain English, get a webhook or email when something meaningful changes ☝️ #webscraping #devtools

https://docs.firecrawl.dev/features/monitoring

Learn how rotating proxies work, when to use them for web scraping, and why IP rotation alone is not enough for reliable data access. https://www.zyte.com/learn/how-do-rotating-proxies-work?utm_campaign=learn-posts&utm_activity=ORS&utm_medium=social&utm_source=mastodon

#webscraping #webdata #data #web

Discover how hotel and flight pricing intelligence is transforming real-time travel market decisions. ✈️

With AI-powered web data collection and analytics, travel businesses can track competitor pricing, monitor demand trends, and optimize revenue strategies more effectively. Real-time travel data helps companies stay competitive, improve forecasting, and respond quickly to changing market conditions.

👉 Read more: https://bit.ly/4vfqZd8

#TravelTech #WebScraping #DataAnalytics #AI

@RainbowFrog moi j'utilise l'extension webscraper pour ce genre de truc https://webscraper.io/ (dans Firefox ou Chrome)
#webscraping #scraping
@belett @Ash_Crow

Web Scraping vs In-House Data Collection, Which delivers better ROI?

While in-house data collection offers control, web scraping wins on speed, scalability, and cost-efficiency.

✅ Faster market insights
✅ Lower operational costs
✅ Real-time competitive intelligence
✅ Scalable data acquisition

Learn More: https://www.webscreenscraping.com/web-scraping-vs-in-house-data-collection/

#WebScraping #DataCollection #BusinessIntelligence #DataStrategy #AI #BigData

🎉 Look, another web scraper! 🎉 Because we *definitely* needed one more tool to fetch JSON from #Wikipedia faster than a cheetah on Red Bull. 🐆💨 No doubt, this will revolutionize the already groundbreaking field of #scraping celebrity birthdates. 🙄✨
https://scrapewithruno.com/ #webscraping #JSONtools #celebritybirthdates #technology #HackerNews #ngated