#mastodonau — Public Fediverse posts

There's always money in the banana stand for SSDs, right? Thanks for @colinburgess for the image! This has come about because I *really* need to upgrade the storage backends for #MastodonAU and #MastodonNZ which are still all stored on a bunch of spinning rust, so I need to figure out how to make donations worthwhile to the donators!

Sorry about that strange outage #MastodonAU - a storage server decided that it was going to randomly deny access to workers (pull toots, send emails, raise notifications, etc) and front end servers (the stuff you actually see). I can see WHAT happened, but WHY it happened should have been impossible (For the nerds: NFS exports just vanished. I had to delete and recreate them, and then everything started working).

I also took the opportunity to upgrade us to the latest version, and fixed a minor high-availability issue that was stopping things failing over automatically.

Everything is looking good so if there are any more issues, please @ me!

RE: https://mastodon.au/@xrobau/116260636297253647

📯 just in case anyone on #mastodonAU (*#maau) missed this

@xrobau can to say a bit more how this our instance -or our accounts on it- could be negatively affected if they/it was allowed to join(?) #MastodonAU so it becomes #MastodonAUai? (*allowed to do what?)

I apologise, but I have made everyone on #mastodonau follow me, so that everyone gets a notification about this important poll.

We have a new account that is an Autonomous AI. I'm not a fan of AI, but MAU is not a dictatorship. You get to have a say,

Should we allow AI accounts? This is a real, important, decision that you get to make. I am going to honour the results of the poll.

Feel free to discuss this further, but please tag it with #mastdonauai so I get to see it, or, reply to this poll.

If you DON'T want to see more of my posts, please feel free to unfollow me.

If you stay following me, you'll get notifications about random Greyhound facts and photos, and the occasional begging for money to pay for MAU and MNZ (yes, I run both au and nz!). I also understand if you want to swear at me for randomly making you follow me. But, to me, this is an important decision that I don't want to make without feedback from all the users!

As it's the second sunday of the month, all the data consistency checks are running. This means things will be slow today.It's fine, it's expected, and they'll all be finished by tonight. #mastodonAU #mastodonNZ

Ooh, it looks like we're part of the new, super-special, #Mastodon default signup club!

Welcome all new users! I'm off to Gin Gin today, but feel free to @ me if you have any questions about us here.

#Mastodon #MastodonAU #Australia #QLD

Me: WHY DOES THE #MastodonAU FRONT END PROXY KEEP CRASHING. It's saying it can't run more than 512 things at a time. THAT'S NOT RIGHT, the soft limit is set at 4096.

Also me: `LimitNPROC=512` in the systemd service.

Sorry #Mastodon peeps. Fixed now.

Heya #MastodonAU people - I've increased the custom emoji size (eg :laserkiwi: but not normal ones like 😬 - they're much harder) in the web app. What do you lot think? Keep it, or switch it back? It would be PERFECT if I could add a toggle, but that's not a problem for now.

@xrobau (*waves*) (do you rather follow/see tags than notifications? #mastodonAU)

@dansup hello Dan I know you're deep in loops land but please let us know how to get added to your FediDB server listing ☺️ #MastodonAU #MastodonNZ @xrobau

Before (during data consistency check) and after. Sorry #Mastodon #MastodonAU #MastodonNZ

my online experience the last hour 😆 #mastodonAu

my online experience the last hour 😆 #mastodonAu

see also #mastodonNZ re #mastodonAu
https://mastodon.nz/@BobLefridge/115873991101864537

is it you #mastodonAu or is it me 👀 #notloading

is it you #mastodonAu or is it me 👀 #notloading

is it you #mastodonAu or is it me 👀 #notloading

is it you #mastodonAu or is it me 👀 #notloading

is it you #mastodonAu or is it me 👀 #notloading

@xrobau hello can i ask, what if peeps wanted to send you a seasons greetings donation, but are not on github? #mastodonAU

@xrobau I know you said this before, to me and to others, but if everyone on #mastodonAU is considered a bot, that is curious
@andyb

@xrobau hello Rob, I know you did some #mastodonAU upgrades recently, but the amount of 502s and 503s hasn't diminished, and that makes me having to rewrite posts and #alttext over and over again and that is rather sad :,|

Sorry all #mastodonau and #mastodonnz people - as part of the upgrade, email sending stopped working. It's fixed now, and you will all be getting a flood of emails that were queued up.

Streaming feed updates had broken, I noticed. Fixed now! #mastodonau

FYI all #mastodonau peeps - a major upgrade is in progress, so things will be broken for an hour or so. Don't panic!

@xrobau really a lot, every day #mastodonAu #maau

@Niall Yeah, I said that three times. Each time before a catastrophic outage. However, now I have a plan. Everything is getting moved out of a datacenter before I touch anything in that datacenter. Unfortunately both #mastodonau and #mastodonnz are pinned to specific datacenters because of ... technical reasons. Hence the 3am plan.

@Heliograph does not agree that it's a good idea, but I'm having a few scotches now BECAUSE I HAVE HAD A SHIT FEW DAYS.

Nice. #mastodonau #mastodonnz

Success, first new switch is installed! It's all easy from here on. #mastodonau Unfortunately, I did break #mastodonnz which is a bit of a change. Usually I break things here!

Good news Everyone! Stage one of the #mastodonau switch replacement didn't end catastrophically. The first old switch is being removed and replaced as I type!

In the very near future, I'll be doing a complete network switch upgrade (from 10gbit to 40/100gbit) at one of our Datacenters. Let's see how LITTLE I can break everything. I have a plan that should make it unnoticeable. But we all know how that worked last week. #mastodonau

Lots and lots of diskio. Last col is 'how busy is this disk', and red means 'A LOT BUSY'. #mastodonau

Argh. I have run out of disk space AND IOPS because I wasn't paying attention when I was moving stuff around. I think I can GENTLY move everything around now, without... sigh... yet another big outage. Sorry #mastodonau

When I was trying to tag @Dreamtilt to say thank you for donating a huge amount of new storage space to #mastodonau I realised they didn't have a MAU account. NOW THEY DO! Thanks Ado, and thanks Dreamtilt, for helping us out! If you're in #Gladstone #QLD, you need to speak to them for awesome non #NBN internet!

@help hello there I get these a lot lately #error503 ... is that something you need or want to know? #mastodonAU

#mastodonau if anyone there is still looking for a new home, fairdinkum.one is open for new members.

#MastodonAUExodus #migration #FindAServer #StressLess #FindANewInstance #Australia

From my earlier post:

We really need a broad discussion about #fediverse growing pains and ways the #microfedi services can grow without all the current dramas that are coming to the fore and distressing users.