#mastodonau — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #mastodonau, aggregated by home.social.
-
RE: https://mastodon.au/@xrobau/117302078925376195
In comparison, #MastodonAU is aware of just under 200m posts. Lots of #tooting and #tooters!
-
RE: https://mastodon.au/@xrobau/117302078925376195
In comparison, #MastodonAU is aware of just under 200m posts. Lots of #tooting and #tooters!
-
RE: https://mastodon.au/@xrobau/117302078925376195
In comparison, #MastodonAU is aware of just under 200m posts. Lots of #tooting and #tooters!
-
RE: https://mastodon.au/@xrobau/117302078925376195
In comparison, #MastodonAU is aware of just under 200m posts. Lots of #tooting and #tooters!
-
RE: https://mastodon.au/@xrobau/117302078925376195
In comparison, #MastodonAU is aware of just under 200m posts. Lots of #tooting and #tooters!
-
Sorry #MastodonNZ but there is a good chance I'm going to have to do a complete database dump and restore, as there's still some bits of corruption left over from the crash the other day. This is probably an hour or so outage. I'm actually not even totally sure that #PostgreSQL will even restart properly when I shut it down. More information will be in this thread (so you can look at it on #MastodonAU directly via the web interface.)
-
Sorry #MastodonNZ but there is a good chance I'm going to have to do a complete database dump and restore, as there's still some bits of corruption left over from the crash the other day. This is probably an hour or so outage. I'm actually not even totally sure that #PostgreSQL will even restart properly when I shut it down. More information will be in this thread (so you can look at it on #MastodonAU directly via the web interface.)
-
Sorry #MastodonNZ but there is a good chance I'm going to have to do a complete database dump and restore, as there's still some bits of corruption left over from the crash the other day. This is probably an hour or so outage. I'm actually not even totally sure that #PostgreSQL will even restart properly when I shut it down. More information will be in this thread (so you can look at it on #MastodonAU directly via the web interface.)
-
Sorry #MastodonNZ but there is a good chance I'm going to have to do a complete database dump and restore, as there's still some bits of corruption left over from the crash the other day. This is probably an hour or so outage. I'm actually not even totally sure that #PostgreSQL will even restart properly when I shut it down. More information will be in this thread (so you can look at it on #MastodonAU directly via the web interface.)
-
Sorry #MastodonNZ but there is a good chance I'm going to have to do a complete database dump and restore, as there's still some bits of corruption left over from the crash the other day. This is probably an hour or so outage. I'm actually not even totally sure that #PostgreSQL will even restart properly when I shut it down. More information will be in this thread (so you can look at it on #MastodonAU directly via the web interface.)
-
Unfun morning, everyone. The storage server that held #MastodonAU and #MastodonNZ s postgres database server dropped TWO hard drives (it could tolerate losing two), and then decided that almost everything written to it was corrupt and needed to be validated. I tried to keep mau and mnz up, but it was just taking too long to try to repair AND move the data off. MAU is back up (obviously!), and mnz is next. If you were unlucky, you saw this page saying (effectively) everything was on fire.
-
Unfun morning, everyone. The storage server that held #MastodonAU and #MastodonNZ s postgres database server dropped TWO hard drives (it could tolerate losing two), and then decided that almost everything written to it was corrupt and needed to be validated. I tried to keep mau and mnz up, but it was just taking too long to try to repair AND move the data off. MAU is back up (obviously!), and mnz is next. If you were unlucky, you saw this page saying (effectively) everything was on fire.
-
Unfun morning, everyone. The storage server that held #MastodonAU and #MastodonNZ s postgres database server dropped TWO hard drives (it could tolerate losing two), and then decided that almost everything written to it was corrupt and needed to be validated. I tried to keep mau and mnz up, but it was just taking too long to try to repair AND move the data off. MAU is back up (obviously!), and mnz is next. If you were unlucky, you saw this page saying (effectively) everything was on fire.
-
Unfun morning, everyone. The storage server that held #MastodonAU and #MastodonNZ s postgres database server dropped TWO hard drives (it could tolerate losing two), and then decided that almost everything written to it was corrupt and needed to be validated. I tried to keep mau and mnz up, but it was just taking too long to try to repair AND move the data off. MAU is back up (obviously!), and mnz is next. If you were unlucky, you saw this page saying (effectively) everything was on fire.
-
Unfun morning, everyone. The storage server that held #MastodonAU and #MastodonNZ s postgres database server dropped TWO hard drives (it could tolerate losing two), and then decided that almost everything written to it was corrupt and needed to be validated. I tried to keep mau and mnz up, but it was just taking too long to try to repair AND move the data off. MAU is back up (obviously!), and mnz is next. If you were unlucky, you saw this page saying (effectively) everything was on fire.
-
Whoops. I forgot I needed a nodejs upgrade. Post/notification streaming is fixed on #MastodonAU and #MastodonNZ so you don't need to keep refreshing the page. Sorry!
-
Whoops. I forgot I needed a nodejs upgrade. Post/notification streaming is fixed on #MastodonAU and #MastodonNZ so you don't need to keep refreshing the page. Sorry!
-
Whoops. I forgot I needed a nodejs upgrade. Post/notification streaming is fixed on #MastodonAU and #MastodonNZ so you don't need to keep refreshing the page. Sorry!
-
Whoops. I forgot I needed a nodejs upgrade. Post/notification streaming is fixed on #MastodonAU and #MastodonNZ so you don't need to keep refreshing the page. Sorry!
-
Whoops. I forgot I needed a nodejs upgrade. Post/notification streaming is fixed on #MastodonAU and #MastodonNZ so you don't need to keep refreshing the page. Sorry!
-
RE: https://mastodon.au/@xrobau/117273777972336409
Upgrades finished for #MastodonAU , now to finish up #MastodonNZ 's upgrades.
-
RE: https://mastodon.au/@xrobau/117273777972336409
Upgrades finished for #MastodonAU , now to finish up #MastodonNZ 's upgrades.
-
RE: https://mastodon.au/@xrobau/117273777972336409
Upgrades finished for #MastodonAU , now to finish up #MastodonNZ 's upgrades.
-
RE: https://mastodon.au/@xrobau/117273777972336409
Upgrades finished for #MastodonAU , now to finish up #MastodonNZ 's upgrades.
-
RE: https://mastodon.au/@xrobau/117273777972336409
Upgrades finished for #MastodonAU , now to finish up #MastodonNZ 's upgrades.
-
Heads up #MastodonAU - there is an upgrade in progress which requires DB changes, so you may be getting errors when trying to change preferences and those sort of things. Shouldn't be too much longer.
-
Heads up #MastodonAU - there is an upgrade in progress which requires DB changes, so you may be getting errors when trying to change preferences and those sort of things. Shouldn't be too much longer.
-
Heads up #MastodonAU - there is an upgrade in progress which requires DB changes, so you may be getting errors when trying to change preferences and those sort of things. Shouldn't be too much longer.
-
Heads up #MastodonAU - there is an upgrade in progress which requires DB changes, so you may be getting errors when trying to change preferences and those sort of things. Shouldn't be too much longer.
-
Heads up #MastodonAU - there is an upgrade in progress which requires DB changes, so you may be getting errors when trying to change preferences and those sort of things. Shouldn't be too much longer.
-
As I sent out a personalised invite to the #AusNOG crowd to join MAU, I thought you lot may be interested in how #MastodonAU (and #MastodonNZ) connects to the interwebs!
MAU and MNZ (and related things) are hosted mostly in a datacenter up here in #Gladstone (Central #Queensland, #Australia).
I have redundant paths to Brisbane - one 9G connection to #NextDC B1, and a 10G Wave to B2.
I peer with #EdgeIX in both B1 and B2, #IXAU in B1, and Aussie Broadband and Telstra up here.
All this networking and redundancy and stuff is why it is REALLY REALLY ANNOYING that something dumb like 'I forgot to enable #VRRP on the Mastodon edge router' causes everything to fall apart.
The backend is #VyOS and #Mikrotik routers/switches, with #Proxmox for the virtualization, #Ubuntu for the OS's and #ZFS for the storage.
I'm also an #IPv6 guy, so everything is IPv6 enabled. In fact, I love it so much, I own the website. https://ipv6.com.au/
Oh, the reason all this infrastructure exists is because of my real job - https://aussievoip.com.au/ - So if you want #VoIP, or #SIP phones, you know a guy.
-
As I sent out a personalised invite to the #AusNOG crowd to join MAU, I thought you lot may be interested in how #MastodonAU (and #MastodonNZ) connects to the interwebs!
MAU and MNZ (and related things) are hosted mostly in a datacenter up here in #Gladstone (Central #Queensland, #Australia).
I have redundant paths to Brisbane - one 9G connection to #NextDC B1, and a 10G Wave to B2.
I peer with #EdgeIX in both B1 and B2, #IXAU in B1, and Aussie Broadband and Telstra up here.
All this networking and redundancy and stuff is why it is REALLY REALLY ANNOYING that something dumb like 'I forgot to enable #VRRP on the Mastodon edge router' causes everything to fall apart.
The backend is #VyOS and #Mikrotik routers/switches, with #Proxmox for the virtualization, #Ubuntu for the OS's and #ZFS for the storage.
I'm also an #IPv6 guy, so everything is IPv6 enabled. In fact, I love it so much, I own the website. https://ipv6.com.au/
Oh, the reason all this infrastructure exists is because of my real job - https://aussievoip.com.au/ - So if you want #VoIP, or #SIP phones, you know a guy.
-
As I sent out a personalised invite to the #AusNOG crowd to join MAU, I thought you lot may be interested in how #MastodonAU (and #MastodonNZ) connects to the interwebs!
MAU and MNZ (and related things) are hosted mostly in a datacenter up here in #Gladstone (Central #Queensland, #Australia).
I have redundant paths to Brisbane - one 9G connection to #NextDC B1, and a 10G Wave to B2.
I peer with #EdgeIX in both B1 and B2, #IXAU in B1, and Aussie Broadband and Telstra up here.
All this networking and redundancy and stuff is why it is REALLY REALLY ANNOYING that something dumb like 'I forgot to enable #VRRP on the Mastodon edge router' causes everything to fall apart.
The backend is #VyOS and #Mikrotik routers/switches, with #Proxmox for the virtualization, #Ubuntu for the OS's and #ZFS for the storage.
I'm also an #IPv6 guy, so everything is IPv6 enabled. In fact, I love it so much, I own the website. https://ipv6.com.au/
Oh, the reason all this infrastructure exists is because of my real job - https://aussievoip.com.au/ - So if you want #VoIP, or #SIP phones, you know a guy.
-
As I sent out a personalised invite to the #AusNOG crowd to join MAU, I thought you lot may be interested in how #MastodonAU (and #MastodonNZ) connects to the interwebs!
MAU and MNZ (and related things) are hosted mostly in a datacenter up here in #Gladstone (Central #Queensland, #Australia).
I have redundant paths to Brisbane - one 9G connection to #NextDC B1, and a 10G Wave to B2.
I peer with #EdgeIX in both B1 and B2, #IXAU in B1, and Aussie Broadband and Telstra up here.
All this networking and redundancy and stuff is why it is REALLY REALLY ANNOYING that something dumb like 'I forgot to enable #VRRP on the Mastodon edge router' causes everything to fall apart.
The backend is #VyOS and #Mikrotik routers/switches, with #Proxmox for the virtualization, #Ubuntu for the OS's and #ZFS for the storage.
I'm also an #IPv6 guy, so everything is IPv6 enabled. In fact, I love it so much, I own the website. https://ipv6.com.au/
Oh, the reason all this infrastructure exists is because of my real job - https://aussievoip.com.au/ - So if you want #VoIP, or #SIP phones, you know a guy.
-
As I sent out a personalised invite to the #AusNOG crowd to join MAU, I thought you lot may be interested in how #MastodonAU (and #MastodonNZ) connects to the interwebs!
MAU and MNZ (and related things) are hosted mostly in a datacenter up here in #Gladstone (Central #Queensland, #Australia).
I have redundant paths to Brisbane - one 9G connection to #NextDC B1, and a 10G Wave to B2.
I peer with #EdgeIX in both B1 and B2, #IXAU in B1, and Aussie Broadband and Telstra up here.
All this networking and redundancy and stuff is why it is REALLY REALLY ANNOYING that something dumb like 'I forgot to enable #VRRP on the Mastodon edge router' causes everything to fall apart.
The backend is #VyOS and #Mikrotik routers/switches, with #Proxmox for the virtualization, #Ubuntu for the OS's and #ZFS for the storage.
I'm also an #IPv6 guy, so everything is IPv6 enabled. In fact, I love it so much, I own the website. https://ipv6.com.au/
Oh, the reason all this infrastructure exists is because of my real job - https://aussievoip.com.au/ - So if you want #VoIP, or #SIP phones, you know a guy.
-
These spam accounts are getting VERY Random, just to avoid filters. Here's the latest one (already suspended)
-
These spam accounts are getting VERY Random, just to avoid filters. Here's the latest one (already suspended)
-
These spam accounts are getting VERY Random, just to avoid filters. Here's the latest one (already suspended)
-
These spam accounts are getting VERY Random, just to avoid filters. Here's the latest one (already suspended)
-
These spam accounts are getting VERY Random, just to avoid filters. Here's the latest one (already suspended)
-
More random nerding, I just replaced my last HP G7 blade with a 'v zero' CPU. This is the new one, a G9 with a v4 CPU, but only 128gb of RAM - RAM is crazy expensive these days. However, if you notice that #MastodonAU and #MastodonNZ are unusually fast and responsive, you now know why!
-
More random nerding, I just replaced my last HP G7 blade with a 'v zero' CPU. This is the new one, a G9 with a v4 CPU, but only 128gb of RAM - RAM is crazy expensive these days. However, if you notice that #MastodonAU and #MastodonNZ are unusually fast and responsive, you now know why!
-
More random nerding, I just replaced my last HP G7 blade with a 'v zero' CPU. This is the new one, a G9 with a v4 CPU, but only 128gb of RAM - RAM is crazy expensive these days. However, if you notice that #MastodonAU and #MastodonNZ are unusually fast and responsive, you now know why!
-
More random nerding, I just replaced my last HP G7 blade with a 'v zero' CPU. This is the new one, a G9 with a v4 CPU, but only 128gb of RAM - RAM is crazy expensive these days. However, if you notice that #MastodonAU and #MastodonNZ are unusually fast and responsive, you now know why!
-
More random nerding, I just replaced my last HP G7 blade with a 'v zero' CPU. This is the new one, a G9 with a v4 CPU, but only 128gb of RAM - RAM is crazy expensive these days. However, if you notice that #MastodonAU and #MastodonNZ are unusually fast and responsive, you now know why!
-
Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster!
The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM.
Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves.
I am doing my best to avoid any more visits from the #fuckup-fairy
Now to finish #MastodonNZ
-
Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster!
The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM.
Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves.
I am doing my best to avoid any more visits from the #fuckup-fairy
Now to finish #MastodonNZ
-
Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster!
The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM.
Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves.
I am doing my best to avoid any more visits from the #fuckup-fairy
Now to finish #MastodonNZ
-
Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster!
The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM.
Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves.
I am doing my best to avoid any more visits from the #fuckup-fairy
Now to finish #MastodonNZ
-
Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster!
The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM.
Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves.
I am doing my best to avoid any more visits from the #fuckup-fairy
Now to finish #MastodonNZ
-
Sorry about that long-ish outage, but we were YET AGAIN getting smashed by AI bots. I have given up fighting them, and I'm just scaling up until #mastodonAU can handle the load. This time the #PostgreSQL database was running at a load of 100+ because it was disk IO bound. I did *try* to live migrate it, but it was just insanely slow so I made the call to shut it down (hence the 'Something went wrong' errors) and move it offline. It's moved. Now it's running on a pure SSD-only backing store, which hopefully will handle everything the #AI scrapers throw at us.
I'll be doing the same thing with #MastodonNZ too, shortly (but hopefully without the outage, as that's a much more lightweight instance)
I also really need to fix the status page that comes up when mau is down.
-
Sorry about that long-ish outage, but we were YET AGAIN getting smashed by AI bots. I have given up fighting them, and I'm just scaling up until #mastodonAU can handle the load. This time the #PostgreSQL database was running at a load of 100+ because it was disk IO bound. I did *try* to live migrate it, but it was just insanely slow so I made the call to shut it down (hence the 'Something went wrong' errors) and move it offline. It's moved. Now it's running on a pure SSD-only backing store, which hopefully will handle everything the #AI scrapers throw at us.
I'll be doing the same thing with #MastodonNZ too, shortly (but hopefully without the outage, as that's a much more lightweight instance)
I also really need to fix the status page that comes up when mau is down.
-
Sorry about that long-ish outage, but we were YET AGAIN getting smashed by AI bots. I have given up fighting them, and I'm just scaling up until #mastodonAU can handle the load. This time the #PostgreSQL database was running at a load of 100+ because it was disk IO bound. I did *try* to live migrate it, but it was just insanely slow so I made the call to shut it down (hence the 'Something went wrong' errors) and move it offline. It's moved. Now it's running on a pure SSD-only backing store, which hopefully will handle everything the #AI scrapers throw at us.
I'll be doing the same thing with #MastodonNZ too, shortly (but hopefully without the outage, as that's a much more lightweight instance)
I also really need to fix the status page that comes up when mau is down.
-
Sorry about that long-ish outage, but we were YET AGAIN getting smashed by AI bots. I have given up fighting them, and I'm just scaling up until #mastodonAU can handle the load. This time the #PostgreSQL database was running at a load of 100+ because it was disk IO bound. I did *try* to live migrate it, but it was just insanely slow so I made the call to shut it down (hence the 'Something went wrong' errors) and move it offline. It's moved. Now it's running on a pure SSD-only backing store, which hopefully will handle everything the #AI scrapers throw at us.
I'll be doing the same thing with #MastodonNZ too, shortly (but hopefully without the outage, as that's a much more lightweight instance)
I also really need to fix the status page that comes up when mau is down.
-
Sorry about that long-ish outage, but we were YET AGAIN getting smashed by AI bots. I have given up fighting them, and I'm just scaling up until #mastodonAU can handle the load. This time the #PostgreSQL database was running at a load of 100+ because it was disk IO bound. I did *try* to live migrate it, but it was just insanely slow so I made the call to shut it down (hence the 'Something went wrong' errors) and move it offline. It's moved. Now it's running on a pure SSD-only backing store, which hopefully will handle everything the #AI scrapers throw at us.
I'll be doing the same thing with #MastodonNZ too, shortly (but hopefully without the outage, as that's a much more lightweight instance)
I also really need to fix the status page that comes up when mau is down.
-
I've been doing some nerding today, replacing and upgrading routers. Here's some before-and-after CPU utilization graphs (higher is worse). I wasn't at capacity, but it was getting close which is what brought on the upgrade.
For the nerds, it's routing about 400k PPS at 5Gb-ish. For the REAL nerds, who I am certain will comment, YES I KNOW THAT'S NOT THAT MUCH!
-
I've been doing some nerding today, replacing and upgrading routers. Here's some before-and-after CPU utilization graphs (higher is worse). I wasn't at capacity, but it was getting close which is what brought on the upgrade.
For the nerds, it's routing about 400k PPS at 5Gb-ish. For the REAL nerds, who I am certain will comment, YES I KNOW THAT'S NOT THAT MUCH!
-
I've been doing some nerding today, replacing and upgrading routers. Here's some before-and-after CPU utilization graphs (higher is worse). I wasn't at capacity, but it was getting close which is what brought on the upgrade.
For the nerds, it's routing about 400k PPS at 5Gb-ish. For the REAL nerds, who I am certain will comment, YES I KNOW THAT'S NOT THAT MUCH!
-
I've been doing some nerding today, replacing and upgrading routers. Here's some before-and-after CPU utilization graphs (higher is worse). I wasn't at capacity, but it was getting close which is what brought on the upgrade.
For the nerds, it's routing about 400k PPS at 5Gb-ish. For the REAL nerds, who I am certain will comment, YES I KNOW THAT'S NOT THAT MUCH!
-
I've been doing some nerding today, replacing and upgrading routers. Here's some before-and-after CPU utilization graphs (higher is worse). I wasn't at capacity, but it was getting close which is what brought on the upgrade.
For the nerds, it's routing about 400k PPS at 5Gb-ish. For the REAL nerds, who I am certain will comment, YES I KNOW THAT'S NOT THAT MUCH!