#mastoddos — Public Fediverse posts

Let's test my theory about the bots harvesting links from here on Mastodon.

These links have not been seen anywhere so far, they're unique to this post.

https://another.evilgeniusrobot.uk/test-on-mastodon-only

https://an.evilgeniusrobot.uk/test-on-mastodon-exclusive

I suspect these will start to show up in the logs at some point. I exclude the #MastoDDoS effect from the stats, that doesn't count.

#Thinkbot has been pretty fast on the uptake but it could be that those posts got boosted widely, I dunno.

[Hmm.. "clever hand sparrow", indeed :) ]

#botnet

So about that #MastoDDoS issue. I just made a test post.

With my 730 followers that created a pretty steady stream of about 5 requests per second during half a minute, with some stragglers popping up every now and then afterwards.

I have a static Hugo blog so this is "nothing" - but if I would have ten times the number of followers I guess there would be a lot more than 150 instances fetching during that time. Let's go with 1000, so 30 requests per second instead.

At which point would this become an issue? 30 requests per second is nothing in the world of SaaS, but the Mastodon DDoS effect seems to have a noticeable impact on smaller sites as well as creators that while they might have a huge following of blog readers those would all be humans clicking at human speeds.

10Mbit/s is still a common connection speed globally, also for very low cost VPS. That's around 1 Megabyte per second in actual transfer speed, so divided by 30 that means that the preview Mastodon requests can be no larger than 33kb in size before we start running into problems. As soon as it's larger than 33kb any buffers in the system will start to back up - and this includes the IP-stack in the OS as well! Here's a good article on that subject - if you're using the defaults that might well be why a site becomes unreachable: https://www.cyberciti.biz/faq/linux-tcp-tuning/

On a VPS I assume the networking equipment is capable, but if you're self hosting out of your home you need to look into whether your switches, routers and firewall are up to the task as well.

This is *not* a "Mastodon problem" - but Mastodon can help alleviate it. Even at 100Mbit/s connection speed a single image can be larger than 330kb so I think the math holds.

@renchap investigated the options a while back. I'm thinking Solution 2. Let me, the person including the link, "own" all the content of my post - including the preview.

https://gist.github.com/renchap/3ae0df45b7b4534f98a8055d91d52186

ping @MarkPrince and @ele whom I know have seen this too.

So just as an experiment, I tried reposting our Threads federated post about our Ultimate Coffee Gear Wish List here via this account.

And yup, it took our website down for a few minutes. It's still down with all the card requests as I type this.

So no more of that, sadly.

#fediverse #mastoddos

Ok I didn't realise just posting something while having a link in your profile could provoke the #mastoddos, because various servers that haven't seen you for a while check for rel="me" stuff. Luckily I have a tiny and entirely static site but that's a bit silly

I have to stop posting active links to our content on Mastodon.

Every time I do so now, it brings down our website for up to 5 minutes.

We've tried pretty much every claimed fix, including third party caching (which in turn breaks other elements of our website's dynamic display abilities), code changes and such on our back end code, and more stuff I don't understand at all (but have spent money paying our WP developer to implement). None of it has worked.

The #fediverse powers that be need to fix this growing problem of the #MastoDDos effect on websites. The more followers and more servers your followers are from, the more impact this has on literally bringing a website to its knees with all the DB calls.

For instance, this morning, I posted the lovely article our creative writer Ethan wrote, which ended up only getting 2 boosts and one "favourite" here, but it brought down our website for 4 minutes and 12 seconds.

That's not sustainable.

I've never been sure whether the reason I've never experienced the #MastoDdos effect is that nothing on my site has ever been shared widely enough or because most of it is static files and I have a static cache for the parts that aren't.

https://boostie.social/@courtney/113034655455201552

Случайно узнал о забавном явлении #MastoDDoS (вот тут есть подробнее с другими пострадавшими).

Если кратко, то проблема такая:
- популярный блогер вешает ссылку на статью на своём сайте в Феди
- его пост распространяется на серверы подписчиков и все эти серверы идут по ссылке и стягивают превьюху
- сайт под нагрузкой ложится, ведь туда за короткое время прилетела пара десятков тысяч одинаковых запросов, причём лежать может довольно продолжительное время

Даже не представлял, что такое возможно, ведь запросы небольшие, разовые и одинаковые, да и сколько этих узлов в Феди наберётся активных, тысяч двадцать? Но поскольку у всех динамические движки, скрипты и картинки в изобилии, то этого хватает, даже Клаудфларь не справляется, поэтому просят решить проблему со стороны Мастодона (правда, непонятно как).

Вспомнился Синдром Кесслера, но для веба.
#ПрекрасноеНастоящее
RE: flipboard.social/users/coffeeg…

@hrefna @coffeegeek The devil is in the defaults… I had the same problem with my generic WordPress site until I enabled Automattic’s WP Super Cache plug-in. Perhaps the best way WordPress could support the social web is for it to improve its caching defaults for bursty traffic in general.

#MastoDDOS

So some bad news regarding our account here on Mastodon and the #Fediverse

I'm going to have to suspend, at least for now, posting any links to content on our website on this platform. The #MastoDdos effect is only getting worse, and it now brings our website down for a solid 5 to 7 minutes everytime I post a URL. We've tried to fix this on our end, without any real luck. Caching, Cloudfare, etc etc. None of it works.

What I'll do instead is post links to our Threads account when new content is posted on our website. A bit of a weird fix, but until the infrastructure in the fediverse evolves to deal with this growing problem, I can't have our website go down that often. It has further unintended side effects (like harming our SEO ranking).

Of course, I'll still be participating here daily on coffee subjects!

I wonder if Goog is blocking card delivery to mastodon because of the mastoddos attacks whenever cards are called by the fediverse.

Just another reason for the powers that be in the fediverse structure to get their shite together, re cards and urls. ;)

My own website goes down for about 2-4 minutes after I post any URL from CoffeeGeek. #mastoddos

I think I have to stop posting links to CoffeeGeek here on Mastodon until this mastoddos thing is solved by the powers that be.

#mastoddos

Request for info from folks familiar with #CDN, #Cloudfare, #DDos and the #mastoddos phenomenon.

Is there specific settings we should be looking at, in setting up our CDN specific to the heavy load mastodon throws our way when I post a url here?

@vincib @RGrunblatt Ah mais c'est de l'auto #MastoDDoS ça ^^'

Just adjusted caching settings on my site. Hopefully this will fix the issue of Mastodon bringing down my site (RAM at 100%) a post gets linked to.

#WordPress #MastoDDOS

Sometimes I think my website exists just to keep Mastodon servers happy. I guess they must get lonely - perhaps they're reaching out for company.

[technical note: this is what happens when you have a 'verified' link in your profile, and someone boosts or faves one of your posts. It seems each server doesn't visit more than once per day which is a small mercy. But it's still a LOT of requests. NB this link is *not* in the post that was faved or boosted. @renchap #mastoddos ]

Apropos of nothing - this is what happens to my homepage website *every time* someone boosts one of my posts, simply because I have it linked into my profile.. I will do another #MastoDDos blog post on this when I get around to it, as the traffic to my site is something else - bursts of 300-odd requests in a few seconds :)

I mean my site can handle it now but I'm guessing not everyone's can. And this will only get worse as more server instances are added - I don't even have many followers!

My original post on #MastoDDoS is still here: https://mastoddos.evilgeniusrobot.uk/

OK so I've hopefully made my home page a bit more performant now, let's see if it can stand up to the next #MastoDDoS just from me posting / replying to someone.

I think I've uncovered a second #mastoddos effect. I recently added a little website link to my Mastodon profile, and now every time I post, or reply, it gets a stack of GET requests for the home page.

Earlier today I got a reply from @Gargron which obviously will have gone to a huge number of Mastodon servers - my poor little web box couldn't cope - it looks like it's had something close to 2000 requests within a few seconds.

The URL wasn't referenced in the post, only in my profile.

Coup de chaud sur mon site Web qui a été inaccessible en temps raisonnable (moins de 16s) durant une ou deux minutes.

La cause ? Entre 16:08:38 et 16:08:39, un #MastoDDoS vers ma galerie photos à base de 155 GET (et 6 gentils HEAD). Petite machine pas aimer beaucoup de requêtes vers un machin qui utilise du PHP

Avec l'augmentation de la taille du Fedivers, le problème bien connu peut empirer 🤔

#MastoMeta #Mastodon #MastoDDoS
Tim Harford posted a link to smbc-comics.com in https://econtwitter.net/@TimHarford/109637566951730350.
I happened to be watching my feed scroll by when he posted it, so I clicked on the link only 37 seconds after he posted it.
After several minutes the page never loaded. I had to stop the page load and refresh to get it to load.
Mastodon now has enough instances and users that the DDoS effect of posting links is no longer theoretical. It's real NOW. A solution is needed.

Experiment #2: The first time I posted this link was in a reply to another post. Let's see if creating a new post also triggers a thundering herd of Mastobots (I am assuming it won't, as all the servers should have already cached the OG card, but..)

https://mastoddos.evilgeniusrobot.uk/

#mastoddos

@davidgerard Stats for yesterday's experiment: https://mastoddos.evilgeniusrobot.uk/stats/

#mastoddos #stampede

@davidgerard So this site should serve up a different OG card to each Mastodon instance.

I also wrote up a little post about the "MastoDDoS Effect" while I was at it. Feel free to boost - it'll help the experiment and I'll publish a graph of hits later on. My tiny server might slow down a bit, but it'll soon recover. In theory.

https://mastoddos.evilgeniusrobot.uk/

#mastoddos #stampede

@Shaft tu disais quoi à propos de nettoyer les logs? https://www.shaftinc.fr/?q=yo
:troll: #MastoDDoS