#scraper — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #scraper, aggregated by home.social.
-
Spoiler: I did not, in fact, "claim", approve, or authorize anything.
Does anyone know what's the best way to make these scammers stop impersonating me?
-
Spoiler: I did not, in fact, "claim", approve, or authorize anything.
Does anyone know what's the best way to make these scammers stop impersonating me?
-
Gentoo bugzilla closed due AI bot scraper overload
https://social.treehouse.systems/@mgorny/117058483039362779
Comments: https://news.ycombinator.com/item?id=49221864
#HackerNews #Gentoo #Bugzilla #AI #bot #scraper #overload #tech #news #software #development
-
Gentoo bugzilla closed due AI bot scraper overload
https://social.treehouse.systems/@mgorny/117058483039362779
Comments: https://news.ycombinator.com/item?id=49221864
#HackerNews #Gentoo #Bugzilla #AI #bot #scraper #overload #tech #news #software #development
-
🚨 BREAKING: #LWN is still dealing with the dastardly #scraper #menace, and they've come up with a groundbreaking solution: *drumroll*... ask for money! 💸 Apparently, reading about the universe of #Linux now requires a #subscription because, clearly, no one else could possibly report on #free #software. 🙄
https://lwn.net/SubscriberLink/1080822/990a8a5e2d379085/ #News #HackerNews #ngated -
🚨 BREAKING: #LWN is still dealing with the dastardly #scraper #menace, and they've come up with a groundbreaking solution: *drumroll*... ask for money! 💸 Apparently, reading about the universe of #Linux now requires a #subscription because, clearly, no one else could possibly report on #free #software. 🙄
https://lwn.net/SubscriberLink/1080822/990a8a5e2d379085/ #News #HackerNews #ngated -
An Update on the scraper situation
https://lwn.net/SubscriberLink/1080822/990a8a5e2d379085/
Comments: https://news.ycombinator.com/item?id=48864252
#HackerNews #scraper #situation #update #tech #news #web #scraping #developer #community
-
An Update on the scraper situation
https://lwn.net/SubscriberLink/1080822/990a8a5e2d379085/
Comments: https://news.ycombinator.com/item?id=48864252
#HackerNews #scraper #situation #update #tech #news #web #scraping #developer #community
-
What are people doing to defend against scraper waves that totally over do it nowdays?
Anubis? AS blocking? fail2ban? Banning expensive endpoints behind logins? nginx rate limits? What else?I would very much appreciate if people could share blogs, configs, ideas, repos, anything. :blobcathearthug:
-
What are people doing to defend against scraper waves that totally over do it nowdays?
Anubis? AS blocking? fail2ban? Banning expensive endpoints behind logins? nginx rate limits? What else?I would very much appreciate if people could share blogs, configs, ideas, repos, anything. :blobcathearthug:
-
https://codeberg.org/audricd/WCOScraper
otro scraper, para quien le sea util. se agradece feedback
#cartons #scraper #dev -
https://codeberg.org/audricd/WCOScraper
otro scraper, para quien le sea util. se agradece feedback
#cartons #scraper #dev -
Eine großartige Scheibe die die #Scraper Boys da geschaffen haben, ich hatte das Glück letztes Wochenende bei der Release Show die Limited Edition zu ergattern 🤘🤘🤘 jetzt muss ich zusehen das ich die Patches noch bis Samstag auf die Kutte kriege 🤘🤘🤘 #scrapermetal
-
SCRAPER (Alemanya) presenta nou àlbum: "Infinite Journey" #Scraper #ThrashMetal #Maig2026 #Alemanya #NouÀlbum #Metall #Metal #MúsicaMetal #MetalMusic
-
Posts nur für den Feed
Ich las vorhin davon, dass Sourcefeed einem ein Feed-Only-Publishing anbietet. Irgendwie klingt das erstmal wild, aber wenn man darüber nachdenkt, ist es wie ein Podcast für Blogs. Du machst quasi Blogbeiträge nur für Leute, die dich im Feedreader abonniert haben.
Mir gefällt dieser Gedanke. Auf der „Why“-Seite gibt es noch ein paar gute Gründe, wie beispielsweise die Tatsache, dass die typischen AI-Scraper die Seiten nicht als Trainingsmaterial verwenden, weil sie Feeds ignorieren.
Na, mal sehen. Ich würde jetzt kein Geld für so ein Feature ausgeben, deshalb habe ich das einfach mal hier im Blog verbaut. Diese Posts werden vorerst noch ganz normal ins Fediverse gepusht, aber mal schauen, wie lange noch.
Deshalb: Feed abonnieren.
#shortpost #blogpost #blog #Feed #Sourcefeed #FeedOnly #AI #Scraper
-
Posts nur für den Feed
Ich las vorhin davon, dass Sourcefeed einem ein Feed-Only-Publishing anbietet. Irgendwie klingt das erstmal wild, aber wenn man darüber nachdenkt, ist es wie ein Podcast für Blogs. Du machst quasi Blogbeiträge nur für Leute, die dich im Feedreader abonniert haben.
Mir gefällt dieser Gedanke. Auf der „Why“-Seite gibt es noch ein paar gute Gründe, wie beispielsweise die Tatsache, dass die typischen AI-Scraper die Seiten nicht als Trainingsmaterial verwenden, weil sie Feeds ignorieren.
Na, mal sehen. Ich würde jetzt kein Geld für so ein Feature ausgeben, deshalb habe ich das einfach mal hier im Blog verbaut. Diese Posts werden vorerst noch ganz normal ins Fediverse gepusht, aber mal schauen, wie lange noch.
Deshalb: Feed abonnieren.
#shortpost #blogpost #blog #Feed #Sourcefeed #FeedOnly #AI #Scraper
-
Some updates on my website maintenance woes:
Starting last July, I built a new wiki for my translations of German folk tales. And soon after I started doing so, it started to experience frequent, hours-long outages. I started to research possible causes, but eventually concluded that the primary cause were so many requests from anonymous #scraper bot networks deserpate for new scraps of data to feed into their #LLM models that the wiki simply couldn't cope. Even when I increased my hosting plan _twice_ last September, this only served to make the outages less common - not to stop them.
In March, I drastically reduced the amount of work I did on the wiki, as it was functionally complete - I had added more than 700 folk tales to it by that stage. Sure, there are always further tales to add - I didn't stop translating those tales, after all. But now I am adding 10-20 tales per month, not 100+.
And funnily enough, I haven't noticed any major outages for this past month - or even minor ones. I guess the scraper bot networks noticed that I don't have that much new data to steal, and largely moved on to new prey they can harass.
So, what can we conclude from this?
If you are maintaining a website that produces lots of new content on a regular basis, you _will_ get hammered by these scrapers. robots.txt will do nothing - these use anonymous, ever-changing IP addresses. Maybe you can thwart them with #Cloudfare or similar technologies which I haven't tried out (I am a rank beginner when it comes to website administration, to be frank).
Otherwise you will either have to slow down the publication of new content, pay lots of money for an oversized hosting plan, or live with periodic outages until the #AIBubble bursts, and there is no longer a trillion dollar business case for scraping every website a thousand times a month.
-
Some updates on my website maintenance woes:
Starting last July, I built a new wiki for my translations of German folk tales. And soon after I started doing so, it started to experience frequent, hours-long outages. I started to research possible causes, but eventually concluded that the primary cause were so many requests from anonymous #scraper bot networks deserpate for new scraps of data to feed into their #LLM models that the wiki simply couldn't cope. Even when I increased my hosting plan _twice_ last September, this only served to make the outages less common - not to stop them.
In March, I drastically reduced the amount of work I did on the wiki, as it was functionally complete - I had added more than 700 folk tales to it by that stage. Sure, there are always further tales to add - I didn't stop translating those tales, after all. But now I am adding 10-20 tales per month, not 100+.
And funnily enough, I haven't noticed any major outages for this past month - or even minor ones. I guess the scraper bot networks noticed that I don't have that much new data to steal, and largely moved on to new prey they can harass.
So, what can we conclude from this?
If you are maintaining a website that produces lots of new content on a regular basis, you _will_ get hammered by these scrapers. robots.txt will do nothing - these use anonymous, ever-changing IP addresses. Maybe you can thwart them with #Cloudfare or similar technologies which I haven't tried out (I am a rank beginner when it comes to website administration, to be frank).
Otherwise you will either have to slow down the publication of new content, pay lots of money for an oversized hosting plan, or live with periodic outages until the #AIBubble bursts, and there is no longer a trillion dollar business case for scraping every website a thousand times a month.
-
Backing up Spotify - Anna’s Blog
"We backed up Spotify (metadata and music files). It’s distributed in bulk torrents (~300TB). It’s the world’s first “preservation archive” for music which is fully open (meaning it can easily be mirrored by anyone with enough disk space), with 86 million music files, representing around 99.6% of listens."
-
Backing up Spotify - Anna’s Blog
"We backed up Spotify (metadata and music files). It’s distributed in bulk torrents (~300TB). It’s the world’s first “preservation archive” for music which is fully open (meaning it can easily be mirrored by anyone with enough disk space), with 86 million music files, representing around 99.6% of listens."
-
#Cloudflare 09:00: I will protect you from all these scrapers smoking your cpu and webserver bandwidth.
Takes a lunch 🥪
Cloudflare 13:00: Hey AI-model hot boys… I have a 1 click #scraper for you to fill up that model!
https://developers.cloudflare.com/changelog/post/2026-03-10-br-crawl-endpoint/
-
#Cloudflare 09:00: I will protect you from all these scrapers smoking your cpu and webserver bandwidth.
Takes a lunch 🥪
Cloudflare 13:00: Hey AI-model hot boys… I have a 1 click #scraper for you to fill up that model!
https://developers.cloudflare.com/changelog/post/2026-03-10-br-crawl-endpoint/
-
New rule: Every time I notice an overnight outage with my website, a new scraper gets added to my robots.txt file.
Welcome to the list, "Amzn-SearchBot".
-
New rule: Every time I notice an overnight outage with my website, a new scraper gets added to my robots.txt file.
Welcome to the list, "Amzn-SearchBot".
-
weiß jemand ob es einen Weg gibt aus #Microsoft #Teams die #Schichten zu extrahieren?
Würde die Daten gerne in den Familien #Kalender mit aufnehmen.
-
Robots.txt Generator - Retro Terminal Edition - Mehr als 200 Bots in der kostenfreien Version. Pures HTML, Javascript und ein bisschen CSS. Keine Third Parties, kein Framework, kein CDN, keine Cookies, kein Tracking, keine Werbung, kein BigTech-Gedönse, keine KI, sehr datenschutzfreundlich. Simple und effektiv im Retro-Style. Demnächst online.
#teufelswerk #HTML #javascript #app #entwicklung #code #retro #css #robotstxt #generator #stopbots #bots #crawler #scraper #keineKI #cookieless #datenschutz
-
Robots.txt Generator - Retro Terminal Edition - Mehr als 200 Bots in der kostenfreien Version. Pures HTML, Javascript und ein bisschen CSS. Keine Third Parties, kein Framework, kein CDN, keine Cookies, kein Tracking, keine Werbung, kein BigTech-Gedönse, keine KI, sehr datenschutzfreundlich. Simple und effektiv im Retro-Style. Demnächst online.
#teufelswerk #HTML #javascript #app #entwicklung #code #retro #css #robotstxt #generator #stopbots #bots #crawler #scraper #keineKI #cookieless #datenschutz
-
LWN is currently under the heaviest scraper attack seen yet
https://social.kernel.org/notice/B2JlhcxNTfI8oDVoyO
#HackerNews #LWN #Scraper #Attack #Heavy #Traffic #Cybersecurity #OpenSource #News
-
LWN is currently under the heaviest scraper attack seen yet
https://social.kernel.org/notice/B2JlhcxNTfI8oDVoyO
#HackerNews #LWN #Scraper #Attack #Heavy #Traffic #Cybersecurity #OpenSource #News
-
🏔️ Tôi đã xây dựng thư viện Node.js "dao Thụy Sĩ" cho tác vụ web:
➤ Tải video không logo từ TikTok/YouTube/Instagram/Facebook
➤ Tìm kiếm nhạc, chụp trang web & nâng cấp ảnh AI (công cụ Remini)
➤ Hỗ trợ cả ESM (import) và CJS (require)
➤ TypeScript + Xoay User-Agent phá chặn
➤ Chỉ 20KB, cài đặt bằng:
npm install @heavstaltech/api✨ Code mẫu:
import { downloader } from '@heavstaltech/api';
const tiktokVideo = await downloader.tiktok("URL"); -
-
-
Ah, the eternal struggle between the noble #blogger and those dastardly HTML-scraping fiends 🤖🦹♂️! Terence Eden just can't wrap his head around why anyone would choose brute force over the #elegance of an API—because heaven forbid anyone use a different method to gather useless #data from his #digital #diary 🎭.
https://shkspr.mobi/blog/2025/12/stop-crawling-my-html-you-dickheads-use-the-api/ #vs #scraper #API #gathering #HackerNews #ngated -
Ah, the eternal struggle between the noble #blogger and those dastardly HTML-scraping fiends 🤖🦹♂️! Terence Eden just can't wrap his head around why anyone would choose brute force over the #elegance of an API—because heaven forbid anyone use a different method to gather useless #data from his #digital #diary 🎭.
https://shkspr.mobi/blog/2025/12/stop-crawling-my-html-you-dickheads-use-the-api/ #vs #scraper #API #gathering #HackerNews #ngated -
staying stealthy and reliable so your agents always get what they need without friction
"Staying stealthy"!? Hey #Mozilla, stealthy against whom? The companies trying to track users across the open web? That's the type of tech your charter calls for right!? No?
It can click, scroll, search, and submit just like a human, navigating complex flows with real-time feedback and adaptive behavior. Get full control of
the web, without the complexity.You're building tech for the parasites to scrape the open web "without friction"? You're mimicing the very users your foundation claims to put first against extractive systems. Do you have no sense of shame or at least irony!? We're looking to you to be the last bastion that can defend the web at scale and you're out here building a Trojan horse in the cover of darkness.
It's not that I have just lost respect I've lost hope, I am not sorry for this very confrontational and scorched earth post. I'm sad and seething!
-
staying stealthy and reliable so your agents always get what they need without friction
"Staying stealthy"!? Hey #Mozilla, stealthy against whom? The companies trying to track users across the open web? That's the type of tech your charter calls for right!? No?
It can click, scroll, search, and submit just like a human, navigating complex flows with real-time feedback and adaptive behavior. Get full control of
the web, without the complexity.You're building tech for the parasites to scrape the open web "without friction"? You're mimicing the very users your foundation claims to put first against extractive systems. Do you have no sense of shame or at least irony!? We're looking to you to be the last bastion that can defend the web at scale and you're out here building a Trojan horse in the cover of darkness.
It's not that I have just lost respect I've lost hope, I am not sorry for this very confrontational and scorched earth post. I'm sad and seething!
-
MWoffliner 1.17 has been released!
https://www.npmjs.com/package/mwoffliner/v/1.17.0This is a minor new version of our flagship #MediaWiki #scraper but the changelog is still huge with really a lot of bug fixes!
https://github.com/openzim/mwoffliner/releases/tag/v1.17.0 -
Bad morning,
I just woke up to find out that openAI scraped my entire website, ignoring robots.txt
