#webscraping — Public Fediverse posts

Fedi, I need your input! Since the web is dead and googling "which user agent to use for responsible web scraping" mostly returns AI-generated garbage promoting how to spoof the user agent for non-responsible web scraping: What are your best practices? Any guide you would recommend? #FediHelp #WebScraping #DigitalHumanities

Heurísticas para Web Scraping co…

Los Modelos de Lenguaje de Gran Escala (LLMs) son sistemas de inteligencia artificial diseñados para entender y generar texto. Su aplicación en web scraping radica en la capacidad de analizar y extraer datos de páginas web complejas.

https://norvik.tech/news/analisis-llms-para-web-scraping

#Technology #WebScraping #Llms #Heuristicas #DesarrolloWeb #NorvikTech #DesarrolloSoftware #TechInnovation

El lado del mal - El Captcha Cognitivo de TikTok al estilo de "The Secret of the Monkey Island" que resuelve la Inteligencia Artificial https://www.elladodelmal.com/2026/05/el-captcha-cognitivo-de-tiktok-al.html #Captcha #IA #AI #InteligenciaArtificial #Cognitive #Gemini #MonkeyIsland #Hacking #Pentesting #Webscraping #TikTok

ICYMI: News publishers target Common Crawl, the AI training data backdoor: News/Media Alliance sent a formal letter to Common Crawl demanding it stop unauthorized scraping and block AI companies from using news content for training. https://ppc.land/news-publishers-target-common-crawl-the-ai-training-data-backdoor/ #AI #NewsMedia #CommonCrawl #DataPrivacy #WebScraping

The Register: Stale gov.uk pages are feeding AI overviews old data and Brits are believing it. “AI overviews from the likes of Google are serving up false summaries of UK government information by drawing on stale GOV.UK pages, according to content designers at the Department for Business and Trade (DBT). The problem, senior content designer Giorgio Di Tunno and content operations lead Neil […]

Join us on April 29, 4:30-5:30pm Eastern for a DRP Volunteer-led workshop on #webscraping! This will be a #handson opportunity to expand your technical skills to save data (or, as our volunteers put it, get your very own rocket booster!). Learn more: www.datarescueproject.org/web-scraping...

Scrapers, Pipelines, and AI, O...

RT @TheAhmadOsman: PRO-TIPP Mein Agent Web-Stack - SearXNG: Entdeckung potenzieller Quellen - Firecrawl: Scraping und Crawling bekannter URLs - Camofox: Browser-Fallback für JS/Interaktion Suchen - Extrahieren - Interagieren P.S. Gib dies deinem bevorzugten Agenten und sage ihm, er soll diese Tools zur Nutzung mit lokalen Modellen einrichten. Ahmad (@TheAhmadOsman) Nutzt du lokale LLMs? Stelle sicher, dass du die Websuche für sie einrichtest. Sag deinem bevorzugten Agenten, er soll SearNg für dich einrichten. Gib das deinen lokalen LLMs (sag einem Agenten, dass er das ebenfalls einrichten soll). Beobachte, wie sie viel intelligenter und effizienter werden. Bitte sehr — https://nitter.net/TheAhmadOsman/status/2043884414774505741#m

mehr auf Arint.info

#AIAgents #LLM #LocalAI #TechStack #WebScraping #arint_info

https://x.com/TheAhmadOsman/status/2044142893242204550#m

Initial questions about this #retraction risk calculator:

What about negative phrases in social media posts that are NOT actually about the linked article?

What about negative posts about a paper that don’t actually link to the paper? (Screenshots, “link in comment” posts, etc.)

#NLP #sentimentAnalysis #webScraping #bibliometrics #Altmetric #stats

Quo Vadis, Crawlers? Progress and what’s next on safeguarding our infrastructure https://diff.wikimedia.org/2026/03/26/quo-vadis-crawlers-progress-and-whats-next-on-safeguarding-our-infrastructure/ #AI, #AIDataCrawlers, #Crawlers, #Infrastructure, #Knowledge, #KnowledgeAsAService, #Scraping, #ScrapingBots, #WebScraping, #WikimediaFoundation, #WikimediaProjects

Oh joy, another #GitHub repository rehashing the same #overhyped #AI tricks we've seen a thousand times before. 🚀👏 Now with 100% more #TypeScript to make sure your web scraping dreams are both verbose and complicated! 🏆🎉
https://github.com/lightfeed/extractor #WebScraping #HackerNews #ngated

🤔 Ah, the noble pursuit of creating an #SDK that scrambles HTML like an egg to "outsmart" scrapers. Because nothing screams #innovation like making your website look like it went through a blender. 😅 Bravo for single-handedly advancing the art of content protection to new heights of absurdity! 🙌
https://www.obscrd.dev/ #ContentProtection #WebScraping #Absurdity #HackerNews #ngated

TechSpot: Smart TV apps are quietly scraping web data for AI training. “Companies specializing in scraping or otherwise harvesting publicly available content to train AI models are becoming increasingly common. In particular, some firms are targeting smart TV applications and similar platforms, attempting to leverage users’ internet connectivity in exchange for low-cost incentives such as […]

Apropos of content heists…

DIY anti-scraping movement, why bother blocking when you can’t win? Poison instead. https://alexschroeder.ch/view/2026-02-20-garbage

#webscraping #datapoisoning #aitraining #ai

The Register: SerpApi says Google is the pot calling the kettle black when it comes to scraping. “SerpApi, a Texas-based web scraping company, has asked a California court to dismiss Google’s claim that that it bypassed digital locks to gather copyrighted content in Google Search results.”

In commercial real estate, speed matters. On-demand property data extraction gives firms real-time access to listings, pricing & availability enabling faster, smarter decisions.

Read More: https://www.webscreenscraping.com/on-demand-property-data-extraction-for-commercial-real-estate.php

#CommercialRealEstate #PropTech #WebScraping #RealEstateData #MarketIntelligence #DataDriven #CRE

Maxun v0.0.32 ra mắt với tính năng ghi âm thời gian thực, hỗ trợ đồng bộ trạng thái website thực tế, thao tác live như gõ, nhấn, cuộn, điều hướng. Hỗ trợ tích hợp SDK: LlamaIndex, Google Sheets, Airtable, LangChain, OpenAI và nhiều hơn nữa. Chế độ AI tự động tìm và trích xuất dữ liệu mà không cần URL. Mã nguồn mở, tự lưu trữ. #Maxun #WebScraping #OpenSource #SelfHosted #AI #LlamaIndex #LangChain #NoCode #DataExtraction #CôngCụMãNguồnMở #TríchXuấtDữLiệu #AI #TựHost

https://www.reddit.com/r/selfh

Search Engine Land: Does llms.txt matter? We tracked 10 sites to find out. “We wanted data, not debates. So we tracked llms.txt adoption across 10 sites in finance, B2B SaaS, ecommerce, insurance, and pet care — 90 days before implementation and 90 days after.”

Search Engine Roundtable: Google Sued SerpApi Over Scraping Search Results. “On Friday, Google announced it had filed a lawsuit (PDF) against SerpApi for scraping the Google search results. Google alleges that SerpApi is running an ‘unlawful’ operation that bypasses Google’s security measures to scrape search results at an astonishing scale.”

Google sues SerpApi over scraping

https://web.brid.gy/r/https://nerds.xyz/2025/12/google-sues-serpapi/

Search Engine Roundtable: OpenAI Scales Up Crawling & Bots For The Holidays. “OpenAI is reportedly scaling up its crawling infrastructure for the holiday shopping season. The folks at Merj noticed OpenAI adding a lot of new IP ranges for its bots and crawlers.”

https://rbfirehose.com/2025/12/01/search-engine-roundtable-openai-scales-up-crawling-bots-for-the-holidays/

Mashable: Common Crawl accused of feeding paywalled content to AI companies. “In a detailed investigation for The Atlantic, reporter Alex Reisner reveals that several major AI companies have quietly partnered with the Common Crawl Foundation — a nonprofit that scrapes the web to build a massive public archive of the internet for research purposes.”

https://rbfirehose.com/2025/11/09/mashable-common-crawl-accused-of-feeding-paywalled-content-to-ai-companies/

Craigslist isn’t just a marketplace — it’s a goldmine of data.
By scraping Craigslist listings, businesses unlock real-time market intelligence:

🏠 Real estate pros track pricing trends
💼 Recruiters analyze job demand
🚗 Auto dealers monitor competitive pricing

Learn more: https://www.webscreenscraping.com/how-craigslist-data-scraping-transforms-industries.php

Smarter insights. Sharper strategies. All powered by data. 💡

#WebScraping #CraigslistData #MarketIntelligence #DataAnalytics #PropTech #RecruitmentTrends #AutoIndustry #BusinessGrowth #AIInsights

Wow, someone finally cracked the code on turning URLs into RSS feeds 🔮, using the arcane magic of CSS selectors 🧙‍♂️. Apparently, it's "quickly" done if you have a PhD in web scraping and a deep spiritual connection with the spirit of XML 😂. Now you too can relive the 2000s dream of reading your news the way your grandparents never did 🌐.
https://feedmaker.fly.dev #URLtoRSS #CSSselectors #WebScraping #XMLmagic #2000sNostalgia #HackerNews #ngated

"In our piece exploring whether the AI revolution is leaving APIs behind, we wrote about some of the factors limiting the extent to which AI tools like chatbots can interface with APIs.

Some of these include:

- Limited or no access to APIs for developers
- APIs are sometimes overcomplicated, bloated, or difficult to call
- Legacy APIs (WS/RPC) lack thorough or up-to-date documentation
- APIs sometimes only cover a fraction of the functions available via the UI

It’s worth noting that many of these points impact human API consumers just as much as they do agentic ones. If you’ve ever been in the position of trying to use an API and it falling short of your expectations, you’ll know just how frustrating it can be.

While it’s possible that some of those users will get in touch to ask you to add certain endpoints or clarify things, plenty more won’t. Some developers are more likely to take the view that it’s easier to ask for forgiveness later than permission now, and find some other way to extract the data they’re looking for. In many cases, web scraping offers just such a solution.

Web scraping APIs are a natural evolution of manual scraping techniques, such as using Python to scrape websites. Used for everything from scraping search engine results, like SERP APIs, to product prices and sentiment analysis, there are various services out there that make web scraping very straightforward. And they’re big business."

https://nordicapis.com/are-web-scraping-tools-overtaking-official-apis/

#APIs #WebScraping #SoftwareDevelopment #Programming #APIDocumentation #APIDesign #Python

Google's hiring an Anti-Scraping Engineering Analyst to battle SERP scrapers! This could mean cleaner data for SEO pros but headaches for third-party tools.

https://hostvix.com/google-is-hiring-an-analyst-to-fight-search-scrapers/

#Google #AntiScraping #SEO #SearchEngine #SERP #DataAnalysis #TechHiring #MachineLearning #ThreatIntelligence #FraudDetection #WebScraping #DigitalSecurity #AI #SearchTools #TechNews #EngineeringJobs #GoogleSearch #AbuseDetection #DataScience #CyberSecurity

Making the most out of a small LLM

Yesterday i finally built my own #AI #server. I had a spare #Nvidia RTX 2070 with 8GB of #VRAM laying around and wanted to do this for a long time.

The problem is that most #LLMs need a lot of VRAM and i don't want to buy another #GPU just to host my own AI. Then i came across #gemma3 and #qwen3. Both of these are amazing #quantized models with stunning reasoning given that they need so less resources.

I chose huihui_ai/qwen3-abliterated:14b since it supports #deepthinking, #toolcalling and is pretty unrestricted. After some testing i noticed that the 8b model performs even better than the 14b variant with drastically better performance. I can't make out any quality loss there to be honest. The 14b model sneaked in chinese characters into the response very often. The 8b model on the other hand doesn't.

Now i've got a very fast model with amazing reasoning (even in German) and tool calling support. The only thing left to improve is knowledge. #Firecrawl is a great tool for #webscraping and as soon as i implemented websearching, the setup was complete. At least i thought it was.

I want to make the most out of this LLM and therefore my next step is to implement a basic #webserver that exposes the same #API #endpoints as #ollama so that everywhere ollama is supported, i can point it to my python script instead. This way it feels like the model is way more capable than it actually is. I can use these advanced features everywhere without being bound to it's actual knowledge.

To improve this setup even more i will likely switch to a #mixture_of_experts architecture soon. This project is a lot of fun and i can't wait to integrate it into my homelab.

#homelab #selfhosting #privacy #ai #llm #largelanguagemodels #coding #developement

Perplexity ignoriert robots.txt: Kontroverse um Daten-Scraping für KI-Training
Das Training großer Sprachmodelle beruht auf einer Vielzahl von Webdaten. Die Einhaltu
https://www.apfeltalk.de/magazin/news/perplexity-ignoriert-robots-txt-kontroverse-um-daten-scraping-fuer-ki-training/
#News #Apple #Applebot #Cloudflare #Cybersecurity #Datenanalyse #Datensicherheit #EthikInDerKI #KITraining #KnstlicheIntelligenz #OpenWeb #Perplexity #robotstxt #Sprachmodell #WebScraping #WebseitenBetreiber

🔍 / #software / #automation

#UiVision Open-Source RPA Software with Computer Vision, OCR, Anthropic Computer Use/LLM. Selenium IDE import/export.

🐱🔗 https://laravista.altervista.org/CatLink/links/299

#catlink #SoftwareAutomation #RPA #OCR #LLM #A9T9 #WebScraping #BrowserExtension #imacros #SeleniumIDE #BrowserAutomation #WebAutomation #DesktopAutomation #DataDrivenTests #Anthropic #AnthropicClaude #ComputerUse #tutorial

PyDoll – Async Python scraping engine with native CAPTCHA bypass

https://github.com/autoscrape-labs/pydoll

#HackerNews #PyDoll #Async #Python #scraping #CAPTCHA #bypass #WebScraping #PythonTools

Need to grab specific info from a webpage regularly? 🤔 Browser Actions can help! Create a Shortcut to: Open URL ➡️ Wait for data element ➡️ Run JavaScript to extract text ➡️ Pass it back to Shortcuts!

If you need help with that, just follow the Forum link on the site!

https://actions.work/browser-actions?ref=mastodon-b10

#macOS #Shortcuts #WebScraping #DataExtraction #BrowserAutomation

3/

For more on scraping (as in web-scraping) see here:
https://mastodon.social/@reiver/114353728684249608

CC: @404mediaco

#Scraper #Scraping #WebScraper #WebScraping

2/

Scraping (as in Web Scraping) is the act of extracting data from HTML web-pages where the data is NOT machine-legible.

If the data, even in an HTML web-page, is in a machine-legible format, then it is NOT scraping.

...

And, getting data in JSON (key-value pairs) is definitely NOT scraping — as JSON's purpose is to communicate data in a machine-legible manner.

CC: @404mediaco

#Scraper #Scraping #WebScraper #WebScraping

1/

If these researchers used a typical HTTP-based API that returns JSON, then —

What these researchers did is NOT scraping.

CC: @404mediaco

RE: https://www.404media.co/researchers-scrape-2-billion-discord-messages-and-publish-them-online/

#Scraper #Scraping #WebScraper #WebScraping

TorrentFreak: Alleged Anna’s Archive Operator Dropped from U.S. ‘Scraping’ Lawsuit. “American nonprofit OCLC sued Anna’s Archive last year for alleged hacking and unauthorized publishing of its WorldCat database. The sole named defendant in the case, an archivist from the Seattle area, denied any involvement with the site. After the court referred several scraping-related questions to […]

https://rbfirehose.com/2025/04/18/torrentfreak-alleged-annas-archive-operator-dropped-from-u-s-scraping-lawsuit/

For Immediate Release, April 1, 2025: University of Michigan Press will publish all of the content on Meta platforms as a series of printed books.
https://www.linkedin.com/posts/charles-watkinson-7553a257_amphibians-and-reptiles-of-the-great-lakes-activity-7312775744932179968-sLSu

#MetaPlatforms #Instagram #Facebook #ThreadsApp #Copyright #BookPublishing #TextMining #TextCorpora #WebScraping #AIethics

AI Crawlers Overwhelm Open-Source Projects, Forcing Developers to Block Entire Countries

#AI #Web #Robotstxt #AIScraping #OpenSource #Cybersecurity #DataScraping #Scraping #WebScraping

https://winbuzzer.com/2025/03/26/ai-crawlers-overwhelm-open-source-projects-forcing-developers-to-block-entire-countries-xcxwbn/

Ars Technica: Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries. “Software developer Xe Iaso reached a breaking point earlier this year when aggressive AI crawler traffic from Amazon overwhelmed their Git repository service, repeatedly causing instability and downtime. Despite configuring standard defensive measures—adjusting robots.txt, blocking known […]

https://rbfirehose.com/2025/03/26/ars-technica-open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/

Reuters: News Corp sued by Brave Software, a Google search engine rival. “News Corp has been sued by Google search engine rival Brave Software, which seeks to forestall a lawsuit by Rupert Murdoch’s company for when readers are directed to copyrighted articles from the Wall Street Journal and New York Post.”

https://rbfirehose.com/2025/03/15/reuters-news-corp-sued-by-brave-software-a-google-search-engine-rival/

#ThingsITried #LeadAutomation #WebScraping #PythonAutomation #LLMvsRegex #OpenAI #WebDataExtraction #MarketingTech #LeadGeneration #AutomationTools #PythonDev

https://thingsitried.com/2025/03/08/leads-automation-using-old-school-blog-freshness-estimate/

Hey does anyone know if there's still a working zip bomb style exploit that can be deployed on a static site/JS (or as a asset/resource)? Specifically to target web scrapers and AI bullshit? The second any server goes online now it's immediately bombarded by stupid numbers of requests.

#hacking #aislop #crawlers #webscraping #webcrawler #robots #zipbomb #zipbombing #exploit #robotstxt #server #scraper

And now @leonardr has "released the biggest update to Beautiful Soup in many years." Upgrade to 4.13.1 to enjoy better warnings, type hints, generated API docs, and more.

https://wandering.shop/@leonardr/113935996322582259

#Python #civictech #opensource #FLOSS #BeautifulSoup #civicdata #datascience #scraping #webscraping #datahoarder

And now @leonardr has "released the biggest update to Beautiful Soup in many years." Upgrade to 4.13.1 to enjoy better warnings, type hints, generated API docs, and more.

https://wandering.shop/@leonardr/113935996322582259

#Python #civictech #opensource #FLOSS #BeautifulSoup #civicdata #datascience #scraping #webscraping #datahoarder

And now @leonardr has "released the biggest update to Beautiful Soup in many years." Upgrade to 4.13.1 to enjoy better warnings, type hints, generated API docs, and more.

https://wandering.shop/@leonardr/113935996322582259

#Python #civictech #opensource #FLOSS #BeautifulSoup #civicdata #datascience #scraping #webscraping #datahoarder

And now @leonardr has "released the biggest update to Beautiful Soup in many years." Upgrade to 4.13.1 to enjoy better warnings, type hints, generated API docs, and more.

https://wandering.shop/@leonardr/113935996322582259

#Python #civictech #opensource #FLOSS #BeautifulSoup #civicdata #datascience #scraping #webscraping #datahoarder

And now @leonardr has "released the biggest update to Beautiful Soup in many years." Upgrade to 4.13.1 to enjoy better warnings, type hints, generated API docs, and more.

https://wandering.shop/@leonardr/113935996322582259

#Python #civictech #opensource #FLOSS #BeautifulSoup #civicdata #datascience #scraping #webscraping #datahoarder