#mastodonau — Public Fediverse posts

More random nerding, I just replaced my last HP G7 blade with a 'v zero' CPU. This is the new one, a G9 with a v4 CPU, but only 128gb of RAM - RAM is crazy expensive these days. However, if you notice that #MastodonAU and #MastodonNZ are unusually fast and responsive, you now know why!

More random nerding, I just replaced my last HP G7 blade with a 'v zero' CPU. This is the new one, a G9 with a v4 CPU, but only 128gb of RAM - RAM is crazy expensive these days. However, if you notice that #MastodonAU and #MastodonNZ are unusually fast and responsive, you now know why!

More random nerding, I just replaced my last HP G7 blade with a 'v zero' CPU. This is the new one, a G9 with a v4 CPU, but only 128gb of RAM - RAM is crazy expensive these days. However, if you notice that #MastodonAU and #MastodonNZ are unusually fast and responsive, you now know why!

More random nerding, I just replaced my last HP G7 blade with a 'v zero' CPU. This is the new one, a G9 with a v4 CPU, but only 128gb of RAM - RAM is crazy expensive these days. However, if you notice that #MastodonAU and #MastodonNZ are unusually fast and responsive, you now know why!

More random nerding, I just replaced my last HP G7 blade with a 'v zero' CPU. This is the new one, a G9 with a v4 CPU, but only 128gb of RAM - RAM is crazy expensive these days. However, if you notice that #MastodonAU and #MastodonNZ are unusually fast and responsive, you now know why!

Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster!

The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM.

Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves.

I am doing my best to avoid any more visits from the #fuckup-fairy

Now to finish #MastodonNZ

Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster!

The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM.

Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves.

I am doing my best to avoid any more visits from the #fuckup-fairy

Now to finish #MastodonNZ

Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster!

The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM.

Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves.

I am doing my best to avoid any more visits from the #fuckup-fairy

Now to finish #MastodonNZ

Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster!

The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM.

Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves.

I am doing my best to avoid any more visits from the #fuckup-fairy

Now to finish #MastodonNZ

Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster!

The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM.

Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves.

I am doing my best to avoid any more visits from the #fuckup-fairy

Now to finish #MastodonNZ

Sorry about that long-ish outage, but we were YET AGAIN getting smashed by AI bots. I have given up fighting them, and I'm just scaling up until #mastodonAU can handle the load. This time the #PostgreSQL database was running at a load of 100+ because it was disk IO bound. I did *try* to live migrate it, but it was just insanely slow so I made the call to shut it down (hence the 'Something went wrong' errors) and move it offline. It's moved. Now it's running on a pure SSD-only backing store, which hopefully will handle everything the #AI scrapers throw at us.

I'll be doing the same thing with #MastodonNZ too, shortly (but hopefully without the outage, as that's a much more lightweight instance)

I also really need to fix the status page that comes up when mau is down.

Sorry about that long-ish outage, but we were YET AGAIN getting smashed by AI bots. I have given up fighting them, and I'm just scaling up until #mastodonAU can handle the load. This time the #PostgreSQL database was running at a load of 100+ because it was disk IO bound. I did *try* to live migrate it, but it was just insanely slow so I made the call to shut it down (hence the 'Something went wrong' errors) and move it offline. It's moved. Now it's running on a pure SSD-only backing store, which hopefully will handle everything the #AI scrapers throw at us.

I'll be doing the same thing with #MastodonNZ too, shortly (but hopefully without the outage, as that's a much more lightweight instance)

I also really need to fix the status page that comes up when mau is down.

Sorry about that long-ish outage, but we were YET AGAIN getting smashed by AI bots. I have given up fighting them, and I'm just scaling up until #mastodonAU can handle the load. This time the #PostgreSQL database was running at a load of 100+ because it was disk IO bound. I did *try* to live migrate it, but it was just insanely slow so I made the call to shut it down (hence the 'Something went wrong' errors) and move it offline. It's moved. Now it's running on a pure SSD-only backing store, which hopefully will handle everything the #AI scrapers throw at us.

I'll be doing the same thing with #MastodonNZ too, shortly (but hopefully without the outage, as that's a much more lightweight instance)

I also really need to fix the status page that comes up when mau is down.

Sorry about that long-ish outage, but we were YET AGAIN getting smashed by AI bots. I have given up fighting them, and I'm just scaling up until #mastodonAU can handle the load. This time the #PostgreSQL database was running at a load of 100+ because it was disk IO bound. I did *try* to live migrate it, but it was just insanely slow so I made the call to shut it down (hence the 'Something went wrong' errors) and move it offline. It's moved. Now it's running on a pure SSD-only backing store, which hopefully will handle everything the #AI scrapers throw at us.

I'll be doing the same thing with #MastodonNZ too, shortly (but hopefully without the outage, as that's a much more lightweight instance)

I also really need to fix the status page that comes up when mau is down.

Sorry about that long-ish outage, but we were YET AGAIN getting smashed by AI bots. I have given up fighting them, and I'm just scaling up until #mastodonAU can handle the load. This time the #PostgreSQL database was running at a load of 100+ because it was disk IO bound. I did *try* to live migrate it, but it was just insanely slow so I made the call to shut it down (hence the 'Something went wrong' errors) and move it offline. It's moved. Now it's running on a pure SSD-only backing store, which hopefully will handle everything the #AI scrapers throw at us.

I'll be doing the same thing with #MastodonNZ too, shortly (but hopefully without the outage, as that's a much more lightweight instance)

I also really need to fix the status page that comes up when mau is down.

I've been doing some nerding today, replacing and upgrading routers. Here's some before-and-after CPU utilization graphs (higher is worse). I wasn't at capacity, but it was getting close which is what brought on the upgrade.

For the nerds, it's routing about 400k PPS at 5Gb-ish. For the REAL nerds, who I am certain will comment, YES I KNOW THAT'S NOT THAT MUCH!

#MastodonAU #MastodonNZ #routing #ipv4

I've been doing some nerding today, replacing and upgrading routers. Here's some before-and-after CPU utilization graphs (higher is worse). I wasn't at capacity, but it was getting close which is what brought on the upgrade.

For the nerds, it's routing about 400k PPS at 5Gb-ish. For the REAL nerds, who I am certain will comment, YES I KNOW THAT'S NOT THAT MUCH!

#MastodonAU #MastodonNZ #routing #ipv4

I've been doing some nerding today, replacing and upgrading routers. Here's some before-and-after CPU utilization graphs (higher is worse). I wasn't at capacity, but it was getting close which is what brought on the upgrade.

For the nerds, it's routing about 400k PPS at 5Gb-ish. For the REAL nerds, who I am certain will comment, YES I KNOW THAT'S NOT THAT MUCH!

#MastodonAU #MastodonNZ #routing #ipv4

I've been doing some nerding today, replacing and upgrading routers. Here's some before-and-after CPU utilization graphs (higher is worse). I wasn't at capacity, but it was getting close which is what brought on the upgrade.

For the nerds, it's routing about 400k PPS at 5Gb-ish. For the REAL nerds, who I am certain will comment, YES I KNOW THAT'S NOT THAT MUCH!

#MastodonAU #MastodonNZ #routing #ipv4

I've been doing some nerding today, replacing and upgrading routers. Here's some before-and-after CPU utilization graphs (higher is worse). I wasn't at capacity, but it was getting close which is what brought on the upgrade.

For the nerds, it's routing about 400k PPS at 5Gb-ish. For the REAL nerds, who I am certain will comment, YES I KNOW THAT'S NOT THAT MUCH!

#MastodonAU #MastodonNZ #routing #ipv4

Finally, as a birthday present to myself (it's 4 mins past midnight here), I have upgraded all the storage bits and pieces to the latest firmware versions. I don't think it'll fix anything, and I'm expecting to go crawling through my box of spare parts tomorrow to find replacement bits.

Goodnight #MastodonAU and #MastodonNZ - The silver lining is that if everything falls apart overnight, we do know what the problem is!

Finally, as a birthday present to myself (it's 4 mins past midnight here), I have upgraded all the storage bits and pieces to the latest firmware versions. I don't think it'll fix anything, and I'm expecting to go crawling through my box of spare parts tomorrow to find replacement bits.

Goodnight #MastodonAU and #MastodonNZ - The silver lining is that if everything falls apart overnight, we do know what the problem is!

Finally, as a birthday present to myself (it's 4 mins past midnight here), I have upgraded all the storage bits and pieces to the latest firmware versions. I don't think it'll fix anything, and I'm expecting to go crawling through my box of spare parts tomorrow to find replacement bits.

Goodnight #MastodonAU and #MastodonNZ - The silver lining is that if everything falls apart overnight, we do know what the problem is!

Finally, as a birthday present to myself (it's 4 mins past midnight here), I have upgraded all the storage bits and pieces to the latest firmware versions. I don't think it'll fix anything, and I'm expecting to go crawling through my box of spare parts tomorrow to find replacement bits.

Goodnight #MastodonAU and #MastodonNZ - The silver lining is that if everything falls apart overnight, we do know what the problem is!

Finally, as a birthday present to myself (it's 4 mins past midnight here), I have upgraded all the storage bits and pieces to the latest firmware versions. I don't think it'll fix anything, and I'm expecting to go crawling through my box of spare parts tomorrow to find replacement bits.

Goodnight #MastodonAU and #MastodonNZ - The silver lining is that if everything falls apart overnight, we do know what the problem is!

For anyone still following along with the #MastodonAU storage server crash, I did find an upgraded version that never made it onto a SPP ISO - Vers 7.2. So now I'm moving everything (mastnz and mastau, to act as a load test) back off that storage server, and I'll upgrade it and see what happens. It's currently running 7.0.

For anyone still following along with the #MastodonAU storage server crash, I did find an upgraded version that never made it onto a SPP ISO - Vers 7.2. So now I'm moving everything (mastnz and mastau, to act as a load test) back off that storage server, and I'll upgrade it and see what happens. It's currently running 7.0.

For anyone still following along with the #MastodonAU storage server crash, I did find an upgraded version that never made it onto a SPP ISO - Vers 7.2. So now I'm moving everything (mastnz and mastau, to act as a load test) back off that storage server, and I'll upgrade it and see what happens. It's currently running 7.0.

For anyone still following along with the #MastodonAU storage server crash, I did find an upgraded version that never made it onto a SPP ISO - Vers 7.2. So now I'm moving everything (mastnz and mastau, to act as a load test) back off that storage server, and I'll upgrade it and see what happens. It's currently running 7.0.

For anyone still following along with the #MastodonAU storage server crash, I did find an upgraded version that never made it onto a SPP ISO - Vers 7.2. So now I'm moving everything (mastnz and mastau, to act as a load test) back off that storage server, and I'll upgrade it and see what happens. It's currently running 7.0.

So for the NEXT pile of funtimes tonight at #MastodonAU and #MastodonNZ, as I was going through tidying things up, more strangeness happened. One of the storage servers (not the one that had the replaced motherboard on Wednesday) decided that it didn't want to be a storage server, and stopped talking to its storage controller.

Luckily for me, I had already moved almost everything off it because I didn't trust it after it was playing silly buggers during the outage.

So for the NEXT pile of funtimes tonight at #MastodonAU and #MastodonNZ, as I was going through tidying things up, more strangeness happened. One of the storage servers (not the one that had the replaced motherboard on Wednesday) decided that it didn't want to be a storage server, and stopped talking to its storage controller.

Luckily for me, I had already moved almost everything off it because I didn't trust it after it was playing silly buggers during the outage.

So for the NEXT pile of funtimes tonight at #MastodonAU and #MastodonNZ, as I was going through tidying things up, more strangeness happened. One of the storage servers (not the one that had the replaced motherboard on Wednesday) decided that it didn't want to be a storage server, and stopped talking to its storage controller.

Luckily for me, I had already moved almost everything off it because I didn't trust it after it was playing silly buggers during the outage.

So for the NEXT pile of funtimes tonight at #MastodonAU and #MastodonNZ, as I was going through tidying things up, more strangeness happened. One of the storage servers (not the one that had the replaced motherboard on Wednesday) decided that it didn't want to be a storage server, and stopped talking to its storage controller.

Luckily for me, I had already moved almost everything off it because I didn't trust it after it was playing silly buggers during the outage.

So for the NEXT pile of funtimes tonight at #MastodonAU and #MastodonNZ, as I was going through tidying things up, more strangeness happened. One of the storage servers (not the one that had the replaced motherboard on Wednesday) decided that it didn't want to be a storage server, and stopped talking to its storage controller.

Luckily for me, I had already moved almost everything off it because I didn't trust it after it was playing silly buggers during the outage.

Hopefully that's the end of the blips for tonight. #MastodonAU has been upgraded to 4.6, with some new features that people may want.

https://blog.joinmastodon.org/2026/06/mastodon-4.6/

Hopefully that's the end of the blips for tonight. #MastodonAU has been upgraded to 4.6, with some new features that people may want.

https://blog.joinmastodon.org/2026/06/mastodon-4.6/

Hopefully that's the end of the blips for tonight. #MastodonAU has been upgraded to 4.6, with some new features that people may want.

https://blog.joinmastodon.org/2026/06/mastodon-4.6/

Hopefully that's the end of the blips for tonight. #MastodonAU has been upgraded to 4.6, with some new features that people may want.

https://blog.joinmastodon.org/2026/06/mastodon-4.6/

Hopefully that's the end of the blips for tonight. #MastodonAU has been upgraded to 4.6, with some new features that people may want.

https://blog.joinmastodon.org/2026/06/mastodon-4.6/

I believe I have found the cause for the 503s, and it's going to be annoying to fix.

For the nerds, Caddy is running as the front end proxy, sending requests to 4 back end servers. Caddy checks the responses to make sure the back end server is up and responding rapidly (eg, if it sees a bunch of '503 service unavailable' errors, it will take that back end out of rotation).

The problem is that a couple of popular servers are down, and when things are linked to them, the back end server responds 'Can't connect to xyz, 503 service unavailable'.

As a bunch of clients retry, this snowballs into all the backend servers being marked as offline, because the proxy thinks THEY are generating the 503.

I've put a workaround in place, but I'm not terribly happy with it (increase the number of consecutive errors before a backend is marked as dead), and will be looking harder later.

#MastodonAU #MastodonNZ

I believe I have found the cause for the 503s, and it's going to be annoying to fix.

For the nerds, Caddy is running as the front end proxy, sending requests to 4 back end servers. Caddy checks the responses to make sure the back end server is up and responding rapidly (eg, if it sees a bunch of '503 service unavailable' errors, it will take that back end out of rotation).

The problem is that a couple of popular servers are down, and when things are linked to them, the back end server responds 'Can't connect to xyz, 503 service unavailable'.

As a bunch of clients retry, this snowballs into all the backend servers being marked as offline, because the proxy thinks THEY are generating the 503.

I've put a workaround in place, but I'm not terribly happy with it (increase the number of consecutive errors before a backend is marked as dead), and will be looking harder later.

#MastodonAU #MastodonNZ

I believe I have found the cause for the 503s, and it's going to be annoying to fix.

For the nerds, Caddy is running as the front end proxy, sending requests to 4 back end servers. Caddy checks the responses to make sure the back end server is up and responding rapidly (eg, if it sees a bunch of '503 service unavailable' errors, it will take that back end out of rotation).

The problem is that a couple of popular servers are down, and when things are linked to them, the back end server responds 'Can't connect to xyz, 503 service unavailable'.

As a bunch of clients retry, this snowballs into all the backend servers being marked as offline, because the proxy thinks THEY are generating the 503.

I've put a workaround in place, but I'm not terribly happy with it (increase the number of consecutive errors before a backend is marked as dead), and will be looking harder later.

#MastodonAU #MastodonNZ

I believe I have found the cause for the 503s, and it's going to be annoying to fix.

For the nerds, Caddy is running as the front end proxy, sending requests to 4 back end servers. Caddy checks the responses to make sure the back end server is up and responding rapidly (eg, if it sees a bunch of '503 service unavailable' errors, it will take that back end out of rotation).

The problem is that a couple of popular servers are down, and when things are linked to them, the back end server responds 'Can't connect to xyz, 503 service unavailable'.

As a bunch of clients retry, this snowballs into all the backend servers being marked as offline, because the proxy thinks THEY are generating the 503.

I've put a workaround in place, but I'm not terribly happy with it (increase the number of consecutive errors before a backend is marked as dead), and will be looking harder later.

#MastodonAU #MastodonNZ

I believe I have found the cause for the 503s, and it's going to be annoying to fix.

For the nerds, Caddy is running as the front end proxy, sending requests to 4 back end servers. Caddy checks the responses to make sure the back end server is up and responding rapidly (eg, if it sees a bunch of '503 service unavailable' errors, it will take that back end out of rotation).

The problem is that a couple of popular servers are down, and when things are linked to them, the back end server responds 'Can't connect to xyz, 503 service unavailable'.

As a bunch of clients retry, this snowballs into all the backend servers being marked as offline, because the proxy thinks THEY are generating the 503.

I've put a workaround in place, but I'm not terribly happy with it (increase the number of consecutive errors before a backend is marked as dead), and will be looking harder later.

#MastodonAU #MastodonNZ

Good news, everyone! I turned off the blade, and turned it back on, and the bad drive was so bad it vanished! This means it won't be locking up the server every 30 mins or so. #MastodonAU win!

The bad news is that SOMETHING is still causing 503s and it doesn't appear to be the usual AI scrapers going wild.

I also managed to "slightly" break #MastodonNZ when I moved it from the failing blade and... kinda forgot to turn it back on again when it had moved. Sorry about that MNZ!

Good news, everyone! I turned off the blade, and turned it back on, and the bad drive was so bad it vanished! This means it won't be locking up the server every 30 mins or so. #MastodonAU win!

The bad news is that SOMETHING is still causing 503s and it doesn't appear to be the usual AI scrapers going wild.

I also managed to "slightly" break #MastodonNZ when I moved it from the failing blade and... kinda forgot to turn it back on again when it had moved. Sorry about that MNZ!

Good news, everyone! I turned off the blade, and turned it back on, and the bad drive was so bad it vanished! This means it won't be locking up the server every 30 mins or so. #MastodonAU win!

The bad news is that SOMETHING is still causing 503s and it doesn't appear to be the usual AI scrapers going wild.

I also managed to "slightly" break #MastodonNZ when I moved it from the failing blade and... kinda forgot to turn it back on again when it had moved. Sorry about that MNZ!

Good news, everyone! I turned off the blade, and turned it back on, and the bad drive was so bad it vanished! This means it won't be locking up the server every 30 mins or so. #MastodonAU win!

The bad news is that SOMETHING is still causing 503s and it doesn't appear to be the usual AI scrapers going wild.

I also managed to "slightly" break #MastodonNZ when I moved it from the failing blade and... kinda forgot to turn it back on again when it had moved. Sorry about that MNZ!

Good news, everyone! I turned off the blade, and turned it back on, and the bad drive was so bad it vanished! This means it won't be locking up the server every 30 mins or so. #MastodonAU win!

The bad news is that SOMETHING is still causing 503s and it doesn't appear to be the usual AI scrapers going wild.

I also managed to "slightly" break #MastodonNZ when I moved it from the failing blade and... kinda forgot to turn it back on again when it had moved. Sorry about that MNZ!

Well that's annoying. I randomly had a bunch of 503s (for about 30 seconds) from #mastodonAU but there was no reason! I just spent 30 mins or so trawling through everything it possibly could be.

You know what it could be? A failing boot drive that didn't raise any alarms. It's not that annoying, I already have some replacement drives on the way, but now I have to migrate everything off that blade!

Well that's annoying. I randomly had a bunch of 503s (for about 30 seconds) from #mastodonAU but there was no reason! I just spent 30 mins or so trawling through everything it possibly could be.

You know what it could be? A failing boot drive that didn't raise any alarms. It's not that annoying, I already have some replacement drives on the way, but now I have to migrate everything off that blade!

Well that's annoying. I randomly had a bunch of 503s (for about 30 seconds) from #mastodonAU but there was no reason! I just spent 30 mins or so trawling through everything it possibly could be.

You know what it could be? A failing boot drive that didn't raise any alarms. It's not that annoying, I already have some replacement drives on the way, but now I have to migrate everything off that blade!

Well that's annoying. I randomly had a bunch of 503s (for about 30 seconds) from #mastodonAU but there was no reason! I just spent 30 mins or so trawling through everything it possibly could be.

You know what it could be? A failing boot drive that didn't raise any alarms. It's not that annoying, I already have some replacement drives on the way, but now I have to migrate everything off that blade!

Well that's annoying. I randomly had a bunch of 503s (for about 30 seconds) from #mastodonAU but there was no reason! I just spent 30 mins or so trawling through everything it possibly could be.

You know what it could be? A failing boot drive that didn't raise any alarms. It's not that annoying, I already have some replacement drives on the way, but now I have to migrate everything off that blade!

RE: https://mastodon.au/@xrobau/116921386685116675

About 30 mins after I posted that, everything fell apart in a much more "interesting" way, where almost everything shut down and wouldn't start up again. The end result of the carnage was 3 power supplies, 2 compute nodes, and one core storage server dead. Luckily for me I have standardized on HP DL380 and 360 machines which have interchangeable components (motherboards, power supplies, etc) so it ended up being a slog of loading up the car with spare servers, and replacing components to get it all working again.

What a day. #MastodonAU #MastodonNZ

RE: https://mastodon.au/@xrobau/116921386685116675

About 30 mins after I posted that, everything fell apart in a much more "interesting" way, where almost everything shut down and wouldn't start up again. The end result of the carnage was 3 power supplies, 2 compute nodes, and one core storage server dead. Luckily for me I have standardized on HP DL380 and 360 machines which have interchangeable components (motherboards, power supplies, etc) so it ended up being a slog of loading up the car with spare servers, and replacing components to get it all working again.

What a day. #MastodonAU #MastodonNZ

RE: https://mastodon.au/@xrobau/116921386685116675

About 30 mins after I posted that, everything fell apart in a much more "interesting" way, where almost everything shut down and wouldn't start up again. The end result of the carnage was 3 power supplies, 2 compute nodes, and one core storage server dead. Luckily for me I have standardized on HP DL380 and 360 machines which have interchangeable components (motherboards, power supplies, etc) so it ended up being a slog of loading up the car with spare servers, and replacing components to get it all working again.

What a day. #MastodonAU #MastodonNZ

RE: https://mastodon.au/@xrobau/116921386685116675

About 30 mins after I posted that, everything fell apart in a much more "interesting" way, where almost everything shut down and wouldn't start up again. The end result of the carnage was 3 power supplies, 2 compute nodes, and one core storage server dead. Luckily for me I have standardized on HP DL380 and 360 machines which have interchangeable components (motherboards, power supplies, etc) so it ended up being a slog of loading up the car with spare servers, and replacing components to get it all working again.

What a day. #MastodonAU #MastodonNZ

RE: https://mastodon.au/@xrobau/116921386685116675

About 30 mins after I posted that, everything fell apart in a much more "interesting" way, where almost everything shut down and wouldn't start up again. The end result of the carnage was 3 power supplies, 2 compute nodes, and one core storage server dead. Luckily for me I have standardized on HP DL380 and 360 machines which have interchangeable components (motherboards, power supplies, etc) so it ended up being a slog of loading up the car with spare servers, and replacing components to get it all working again.

What a day. #MastodonAU #MastodonNZ