#mastodonau — Public Fediverse posts

As I sent out a personalised invite to the #AusNOG crowd to join MAU, I thought you lot may be interested in how #MastodonAU (and #MastodonNZ) connects to the interwebs!

MAU and MNZ (and related things) are hosted mostly in a datacenter up here in #Gladstone (Central #Queensland, #Australia).

I have redundant paths to Brisbane - one 9G connection to #NextDC B1, and a 10G Wave to B2.

I peer with #EdgeIX in both B1 and B2, #IXAU in B1, and Aussie Broadband and Telstra up here.

All this networking and redundancy and stuff is why it is REALLY REALLY ANNOYING that something dumb like 'I forgot to enable #VRRP on the Mastodon edge router' causes everything to fall apart.

The backend is #VyOS and #Mikrotik routers/switches, with #Proxmox for the virtualization, #Ubuntu for the OS's and #ZFS for the storage.

I'm also an #IPv6 guy, so everything is IPv6 enabled. In fact, I love it so much, I own the website. https://ipv6.com.au/

Oh, the reason all this infrastructure exists is because of my real job - https://aussievoip.com.au/ - So if you want #VoIP, or #SIP phones, you know a guy.

These spam accounts are getting VERY Random, just to avoid filters. Here's the latest one (already suspended)

#Mastodon #MastodonAU

More random nerding, I just replaced my last HP G7 blade with a 'v zero' CPU. This is the new one, a G9 with a v4 CPU, but only 128gb of RAM - RAM is crazy expensive these days. However, if you notice that #MastodonAU and #MastodonNZ are unusually fast and responsive, you now know why!

Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster!

The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM.

Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves.

I am doing my best to avoid any more visits from the #fuckup-fairy

Now to finish #MastodonNZ

Sorry about that long-ish outage, but we were YET AGAIN getting smashed by AI bots. I have given up fighting them, and I'm just scaling up until #mastodonAU can handle the load. This time the #PostgreSQL database was running at a load of 100+ because it was disk IO bound. I did *try* to live migrate it, but it was just insanely slow so I made the call to shut it down (hence the 'Something went wrong' errors) and move it offline. It's moved. Now it's running on a pure SSD-only backing store, which hopefully will handle everything the #AI scrapers throw at us.

I'll be doing the same thing with #MastodonNZ too, shortly (but hopefully without the outage, as that's a much more lightweight instance)

I also really need to fix the status page that comes up when mau is down.

I've been doing some nerding today, replacing and upgrading routers. Here's some before-and-after CPU utilization graphs (higher is worse). I wasn't at capacity, but it was getting close which is what brought on the upgrade.

For the nerds, it's routing about 400k PPS at 5Gb-ish. For the REAL nerds, who I am certain will comment, YES I KNOW THAT'S NOT THAT MUCH!

#MastodonAU #MastodonNZ #routing #ipv4

Finally, as a birthday present to myself (it's 4 mins past midnight here), I have upgraded all the storage bits and pieces to the latest firmware versions. I don't think it'll fix anything, and I'm expecting to go crawling through my box of spare parts tomorrow to find replacement bits.

Goodnight #MastodonAU and #MastodonNZ - The silver lining is that if everything falls apart overnight, we do know what the problem is!

For anyone still following along with the #MastodonAU storage server crash, I did find an upgraded version that never made it onto a SPP ISO - Vers 7.2. So now I'm moving everything (mastnz and mastau, to act as a load test) back off that storage server, and I'll upgrade it and see what happens. It's currently running 7.0.

So for the NEXT pile of funtimes tonight at #MastodonAU and #MastodonNZ, as I was going through tidying things up, more strangeness happened. One of the storage servers (not the one that had the replaced motherboard on Wednesday) decided that it didn't want to be a storage server, and stopped talking to its storage controller.

Luckily for me, I had already moved almost everything off it because I didn't trust it after it was playing silly buggers during the outage.

Hopefully that's the end of the blips for tonight. #MastodonAU has been upgraded to 4.6, with some new features that people may want.

https://blog.joinmastodon.org/2026/06/mastodon-4.6/

I believe I have found the cause for the 503s, and it's going to be annoying to fix.

For the nerds, Caddy is running as the front end proxy, sending requests to 4 back end servers. Caddy checks the responses to make sure the back end server is up and responding rapidly (eg, if it sees a bunch of '503 service unavailable' errors, it will take that back end out of rotation).

The problem is that a couple of popular servers are down, and when things are linked to them, the back end server responds 'Can't connect to xyz, 503 service unavailable'.

As a bunch of clients retry, this snowballs into all the backend servers being marked as offline, because the proxy thinks THEY are generating the 503.

I've put a workaround in place, but I'm not terribly happy with it (increase the number of consecutive errors before a backend is marked as dead), and will be looking harder later.

#MastodonAU #MastodonNZ

Good news, everyone! I turned off the blade, and turned it back on, and the bad drive was so bad it vanished! This means it won't be locking up the server every 30 mins or so. #MastodonAU win!

The bad news is that SOMETHING is still causing 503s and it doesn't appear to be the usual AI scrapers going wild.

I also managed to "slightly" break #MastodonNZ when I moved it from the failing blade and... kinda forgot to turn it back on again when it had moved. Sorry about that MNZ!

Well that's annoying. I randomly had a bunch of 503s (for about 30 seconds) from #mastodonAU but there was no reason! I just spent 30 mins or so trawling through everything it possibly could be.

You know what it could be? A failing boot drive that didn't raise any alarms. It's not that annoying, I already have some replacement drives on the way, but now I have to migrate everything off that blade!

RE: https://mastodon.au/@xrobau/116921386685116675

About 30 mins after I posted that, everything fell apart in a much more "interesting" way, where almost everything shut down and wouldn't start up again. The end result of the carnage was 3 power supplies, 2 compute nodes, and one core storage server dead. Luckily for me I have standardized on HP DL380 and 360 machines which have interchangeable components (motherboards, power supplies, etc) so it ended up being a slog of loading up the car with spare servers, and replacing components to get it all working again.

What a day. #MastodonAU #MastodonNZ

For those that noticed the #MastodonAU / #MastodonNZ outage, a quick post mortem: Power failed. UPS ran for a few mins, then generators kicked in. When the generators kicked in, two out of three servers decided that their UPS-fed power supplies should be sacrificed to the gods of 240v. No idea why. Those two out of three servers lost power, and then didn't fail across properly, and I had to do a bunch of manual fixing. Power supplies have been replaced, and we'll be switching OFF generator in a few hours. Let's see what happens.

New router is working well. This is a switchport in NextDC B2, connected into a #VyOS VM router running in #Proxmox.

This has been something I've been working on for quite a few months now! #MastodonAU #mastodonNZ #nerd #mikrotik

For those that were dreading the second-sunday-of-the-month-slowdown, it's not going to happen this month, as it's already happened and you didn't notice! #mastodonAU #mastodonNZ

There's always money in the banana stand for SSDs, right? Thanks for @colinburgess for the image! This has come about because I *really* need to upgrade the storage backends for #MastodonAU and #MastodonNZ which are still all stored on a bunch of spinning rust, so I need to figure out how to make donations worthwhile to the donators!

Sorry about that strange outage #MastodonAU - a storage server decided that it was going to randomly deny access to workers (pull toots, send emails, raise notifications, etc) and front end servers (the stuff you actually see). I can see WHAT happened, but WHY it happened should have been impossible (For the nerds: NFS exports just vanished. I had to delete and recreate them, and then everything started working).

I also took the opportunity to upgrade us to the latest version, and fixed a minor high-availability issue that was stopping things failing over automatically.

Everything is looking good so if there are any more issues, please @ me!

RE: https://mastodon.au/@xrobau/116260636297253647

📯 just in case anyone on #mastodonAU (*#maau) missed this

@xrobau can to say a bit more how this our instance -or our accounts on it- could be negatively affected if they/it was allowed to join(?) #MastodonAU so it becomes #MastodonAUai? (*allowed to do what?)

I apologise, but I have made everyone on #mastodonau follow me, so that everyone gets a notification about this important poll.

We have a new account that is an Autonomous AI. I'm not a fan of AI, but MAU is not a dictatorship. You get to have a say,

Should we allow AI accounts? This is a real, important, decision that you get to make. I am going to honour the results of the poll.

Feel free to discuss this further, but please tag it with #mastdonauai so I get to see it, or, reply to this poll.

If you DON'T want to see more of my posts, please feel free to unfollow me.

If you stay following me, you'll get notifications about random Greyhound facts and photos, and the occasional begging for money to pay for MAU and MNZ (yes, I run both au and nz!). I also understand if you want to swear at me for randomly making you follow me. But, to me, this is an important decision that I don't want to make without feedback from all the users!

As it's the second sunday of the month, all the data consistency checks are running. This means things will be slow today.It's fine, it's expected, and they'll all be finished by tonight. #mastodonAU #mastodonNZ

Ooh, it looks like we're part of the new, super-special, #Mastodon default signup club!

Welcome all new users! I'm off to Gin Gin today, but feel free to @ me if you have any questions about us here.

#Mastodon #MastodonAU #Australia #QLD

Me: WHY DOES THE #MastodonAU FRONT END PROXY KEEP CRASHING. It's saying it can't run more than 512 things at a time. THAT'S NOT RIGHT, the soft limit is set at 4096.

Also me: `LimitNPROC=512` in the systemd service.

Sorry #Mastodon peeps. Fixed now.

Heya #MastodonAU people - I've increased the custom emoji size (eg :laserkiwi: but not normal ones like 😬 - they're much harder) in the web app. What do you lot think? Keep it, or switch it back? It would be PERFECT if I could add a toggle, but that's not a problem for now.

@xrobau (*waves*) (do you rather follow/see tags than notifications? #mastodonAU)

@dansup hello Dan I know you're deep in loops land but please let us know how to get added to your FediDB server listing ☺️ #MastodonAU #MastodonNZ @xrobau

Before (during data consistency check) and after. Sorry #Mastodon #MastodonAU #MastodonNZ

my online experience the last hour 😆 #mastodonAu