#mastodonnz — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #mastodonnz, aggregated by home.social.
-
Phew. Well #MastodonNZ it looks like your database is mostly fixed. The only thing still being repaired is the stats table, which I ended up deciding to just rebuild from scratch. This means your post stats will be inaccurate for a day or so whilst all 170 million are recalculated.
-
Phew. Well #MastodonNZ it looks like your database is mostly fixed. The only thing still being repaired is the stats table, which I ended up deciding to just rebuild from scratch. This means your post stats will be inaccurate for a day or so whilst all 170 million are recalculated.
-
Phew. Well #MastodonNZ it looks like your database is mostly fixed. The only thing still being repaired is the stats table, which I ended up deciding to just rebuild from scratch. This means your post stats will be inaccurate for a day or so whilst all 170 million are recalculated.
-
Phew. Well #MastodonNZ it looks like your database is mostly fixed. The only thing still being repaired is the stats table, which I ended up deciding to just rebuild from scratch. This means your post stats will be inaccurate for a day or so whilst all 170 million are recalculated.
-
#MastodonNZ is aware of 147 million posts. That's why it's taking a good chunk of time to back up.
-
#MastodonNZ is aware of 147 million posts. That's why it's taking a good chunk of time to back up.
-
#MastodonNZ is aware of 147 million posts. That's why it's taking a good chunk of time to back up.
-
#MastodonNZ is aware of 147 million posts. That's why it's taking a good chunk of time to back up.
-
The good news is that it looks like it is only the status_stats table that is messed up. I've ALMOST finished a complete raw dump of the db (which takes an hour++) to confirm it's just that table.
After it's all safely backed up, I'll write some scripts to rebuild it all and we HOPEFULLY shouldn't need a long outage
-
The good news is that it looks like it is only the status_stats table that is messed up. I've ALMOST finished a complete raw dump of the db (which takes an hour++) to confirm it's just that table.
After it's all safely backed up, I'll write some scripts to rebuild it all and we HOPEFULLY shouldn't need a long outage
-
The good news is that it looks like it is only the status_stats table that is messed up. I've ALMOST finished a complete raw dump of the db (which takes an hour++) to confirm it's just that table.
After it's all safely backed up, I'll write some scripts to rebuild it all and we HOPEFULLY shouldn't need a long outage
-
The good news is that it looks like it is only the status_stats table that is messed up. I've ALMOST finished a complete raw dump of the db (which takes an hour++) to confirm it's just that table.
After it's all safely backed up, I'll write some scripts to rebuild it all and we HOPEFULLY shouldn't need a long outage
-
31 seconds is surprisingly fast for 100k inserts. Oh well, I'll just keep going and hope I find the derped up records.
-
31 seconds is surprisingly fast for 100k inserts. Oh well, I'll just keep going and hope I find the derped up records.
-
31 seconds is surprisingly fast for 100k inserts. Oh well, I'll just keep going and hope I find the derped up records.
-
31 seconds is surprisingly fast for 100k inserts. Oh well, I'll just keep going and hope I find the derped up records.
-
RE: https://mastodon.au/@xrobau/117300844625946496
Edit: We have figured it out, and are manually importing to a new temp table.
---
Any #PostgreSQL exports around to help me out here? After a storage server crash, I've ended up with (at least one) table having corrupt time data in it. There is, apparently, no way to pg_dump and ignore errors, and \COPY also doesn't appear to continue-on-error.
What I want to do is just get a list of the IDs of the rows with the corrupt time entries so I can either fix them, or just delete them.
This is the 'status_stats' table from #MastodonNZ, and has about 70m rows.
My current plan is that I've copied the schema of status_stats to status_stats_new, changing the time rows to just 'text' and I'm doing 'insert into status_stats_new select * from status_stats ON CONFLICT DO NOTHING;' then a left join to find missing ones.
IS THERE A BETTER WAY? Halp!
-
RE: https://mastodon.au/@xrobau/117300844625946496
Edit: We have figured it out, and are manually importing to a new temp table.
---
Any #PostgreSQL exports around to help me out here? After a storage server crash, I've ended up with (at least one) table having corrupt time data in it. There is, apparently, no way to pg_dump and ignore errors, and \COPY also doesn't appear to continue-on-error.
What I want to do is just get a list of the IDs of the rows with the corrupt time entries so I can either fix them, or just delete them.
This is the 'status_stats' table from #MastodonNZ, and has about 70m rows.
My current plan is that I've copied the schema of status_stats to status_stats_new, changing the time rows to just 'text' and I'm doing 'insert into status_stats_new select * from status_stats ON CONFLICT DO NOTHING;' then a left join to find missing ones.
IS THERE A BETTER WAY? Halp!
-
RE: https://mastodon.au/@xrobau/117300844625946496
Edit: We have figured it out, and are manually importing to a new temp table.
---
Any #PostgreSQL exports around to help me out here? After a storage server crash, I've ended up with (at least one) table having corrupt time data in it. There is, apparently, no way to pg_dump and ignore errors, and \COPY also doesn't appear to continue-on-error.
What I want to do is just get a list of the IDs of the rows with the corrupt time entries so I can either fix them, or just delete them.
This is the 'status_stats' table from #MastodonNZ, and has about 70m rows.
My current plan is that I've copied the schema of status_stats to status_stats_new, changing the time rows to just 'text' and I'm doing 'insert into status_stats_new select * from status_stats ON CONFLICT DO NOTHING;' then a left join to find missing ones.
IS THERE A BETTER WAY? Halp!
-
RE: https://mastodon.au/@xrobau/117300844625946496
Edit: We have figured it out, and are manually importing to a new temp table.
---
Any #PostgreSQL exports around to help me out here? After a storage server crash, I've ended up with (at least one) table having corrupt time data in it. There is, apparently, no way to pg_dump and ignore errors, and \COPY also doesn't appear to continue-on-error.
What I want to do is just get a list of the IDs of the rows with the corrupt time entries so I can either fix them, or just delete them.
This is the 'status_stats' table from #MastodonNZ, and has about 70m rows.
My current plan is that I've copied the schema of status_stats to status_stats_new, changing the time rows to just 'text' and I'm doing 'insert into status_stats_new select * from status_stats ON CONFLICT DO NOTHING;' then a left join to find missing ones.
IS THERE A BETTER WAY? Halp!
-
So far it looks like ONLY the 'status_stats' table is corrupt. This is not a critical table, and fingers crossed I can just skip the bad bits and re-import it.
-
So far it looks like ONLY the 'status_stats' table is corrupt. This is not a critical table, and fingers crossed I can just skip the bad bits and re-import it.
-
So far it looks like ONLY the 'status_stats' table is corrupt. This is not a critical table, and fingers crossed I can just skip the bad bits and re-import it.
-
So far it looks like ONLY the 'status_stats' table is corrupt. This is not a critical table, and fingers crossed I can just skip the bad bits and re-import it.
-
Sorry #MastodonNZ but there is a good chance I'm going to have to do a complete database dump and restore, as there's still some bits of corruption left over from the crash the other day. This is probably an hour or so outage. I'm actually not even totally sure that #PostgreSQL will even restart properly when I shut it down. More information will be in this thread (so you can look at it on #MastodonAU directly via the web interface.)
-
Sorry #MastodonNZ but there is a good chance I'm going to have to do a complete database dump and restore, as there's still some bits of corruption left over from the crash the other day. This is probably an hour or so outage. I'm actually not even totally sure that #PostgreSQL will even restart properly when I shut it down. More information will be in this thread (so you can look at it on #MastodonAU directly via the web interface.)
-
Sorry #MastodonNZ but there is a good chance I'm going to have to do a complete database dump and restore, as there's still some bits of corruption left over from the crash the other day. This is probably an hour or so outage. I'm actually not even totally sure that #PostgreSQL will even restart properly when I shut it down. More information will be in this thread (so you can look at it on #MastodonAU directly via the web interface.)
-
Sorry #MastodonNZ but there is a good chance I'm going to have to do a complete database dump and restore, as there's still some bits of corruption left over from the crash the other day. This is probably an hour or so outage. I'm actually not even totally sure that #PostgreSQL will even restart properly when I shut it down. More information will be in this thread (so you can look at it on #MastodonAU directly via the web interface.)
-
I just watched the first episode of #TaskmasterNZ Season 7 - Rose Matafeo is brilliant as always but David Fane is AWESOME. This is going to be a brilliant season. We need to get them all on #MastodonNZ
-
I just watched the first episode of #TaskmasterNZ Season 7 - Rose Matafeo is brilliant as always but David Fane is AWESOME. This is going to be a brilliant season. We need to get them all on #MastodonNZ
-
I just watched the first episode of #TaskmasterNZ Season 7 - Rose Matafeo is brilliant as always but David Fane is AWESOME. This is going to be a brilliant season. We need to get them all on #MastodonNZ
-
I just watched the first episode of #TaskmasterNZ Season 7 - Rose Matafeo is brilliant as always but David Fane is AWESOME. This is going to be a brilliant season. We need to get them all on #MastodonNZ
-
Unfun morning, everyone. The storage server that held #MastodonAU and #MastodonNZ s postgres database server dropped TWO hard drives (it could tolerate losing two), and then decided that almost everything written to it was corrupt and needed to be validated. I tried to keep mau and mnz up, but it was just taking too long to try to repair AND move the data off. MAU is back up (obviously!), and mnz is next. If you were unlucky, you saw this page saying (effectively) everything was on fire.
-
Unfun morning, everyone. The storage server that held #MastodonAU and #MastodonNZ s postgres database server dropped TWO hard drives (it could tolerate losing two), and then decided that almost everything written to it was corrupt and needed to be validated. I tried to keep mau and mnz up, but it was just taking too long to try to repair AND move the data off. MAU is back up (obviously!), and mnz is next. If you were unlucky, you saw this page saying (effectively) everything was on fire.
-
Unfun morning, everyone. The storage server that held #MastodonAU and #MastodonNZ s postgres database server dropped TWO hard drives (it could tolerate losing two), and then decided that almost everything written to it was corrupt and needed to be validated. I tried to keep mau and mnz up, but it was just taking too long to try to repair AND move the data off. MAU is back up (obviously!), and mnz is next. If you were unlucky, you saw this page saying (effectively) everything was on fire.
-
Unfun morning, everyone. The storage server that held #MastodonAU and #MastodonNZ s postgres database server dropped TWO hard drives (it could tolerate losing two), and then decided that almost everything written to it was corrupt and needed to be validated. I tried to keep mau and mnz up, but it was just taking too long to try to repair AND move the data off. MAU is back up (obviously!), and mnz is next. If you were unlucky, you saw this page saying (effectively) everything was on fire.
-
Whoops. I forgot I needed a nodejs upgrade. Post/notification streaming is fixed on #MastodonAU and #MastodonNZ so you don't need to keep refreshing the page. Sorry!
-
Whoops. I forgot I needed a nodejs upgrade. Post/notification streaming is fixed on #MastodonAU and #MastodonNZ so you don't need to keep refreshing the page. Sorry!
-
Whoops. I forgot I needed a nodejs upgrade. Post/notification streaming is fixed on #MastodonAU and #MastodonNZ so you don't need to keep refreshing the page. Sorry!
-
Whoops. I forgot I needed a nodejs upgrade. Post/notification streaming is fixed on #MastodonAU and #MastodonNZ so you don't need to keep refreshing the page. Sorry!
-
RE: https://mastodon.au/@xrobau/117273937372351034
And #MastodonNZ database upgrades are finishing up now. Both instances are on 4.7.1, so I can start working on some more anti-ai bot stuff.
-
RE: https://mastodon.au/@xrobau/117273937372351034
And #MastodonNZ database upgrades are finishing up now. Both instances are on 4.7.1, so I can start working on some more anti-ai bot stuff.
-
RE: https://mastodon.au/@xrobau/117273937372351034
And #MastodonNZ database upgrades are finishing up now. Both instances are on 4.7.1, so I can start working on some more anti-ai bot stuff.
-
RE: https://mastodon.au/@xrobau/117273937372351034
And #MastodonNZ database upgrades are finishing up now. Both instances are on 4.7.1, so I can start working on some more anti-ai bot stuff.
-
RE: https://mastodon.au/@xrobau/117273777972336409
Upgrades finished for #MastodonAU , now to finish up #MastodonNZ 's upgrades.
-
RE: https://mastodon.au/@xrobau/117273777972336409
Upgrades finished for #MastodonAU , now to finish up #MastodonNZ 's upgrades.
-
RE: https://mastodon.au/@xrobau/117273777972336409
Upgrades finished for #MastodonAU , now to finish up #MastodonNZ 's upgrades.
-
RE: https://mastodon.au/@xrobau/117273777972336409
Upgrades finished for #MastodonAU , now to finish up #MastodonNZ 's upgrades.
-
As I sent out a personalised invite to the #AusNOG crowd to join MAU, I thought you lot may be interested in how #MastodonAU (and #MastodonNZ) connects to the interwebs!
MAU and MNZ (and related things) are hosted mostly in a datacenter up here in #Gladstone (Central #Queensland, #Australia).
I have redundant paths to Brisbane - one 9G connection to #NextDC B1, and a 10G Wave to B2.
I peer with #EdgeIX in both B1 and B2, #IXAU in B1, and Aussie Broadband and Telstra up here.
All this networking and redundancy and stuff is why it is REALLY REALLY ANNOYING that something dumb like 'I forgot to enable #VRRP on the Mastodon edge router' causes everything to fall apart.
The backend is #VyOS and #Mikrotik routers/switches, with #Proxmox for the virtualization, #Ubuntu for the OS's and #ZFS for the storage.
I'm also an #IPv6 guy, so everything is IPv6 enabled. In fact, I love it so much, I own the website. https://ipv6.com.au/
Oh, the reason all this infrastructure exists is because of my real job - https://aussievoip.com.au/ - So if you want #VoIP, or #SIP phones, you know a guy.
-
As I sent out a personalised invite to the #AusNOG crowd to join MAU, I thought you lot may be interested in how #MastodonAU (and #MastodonNZ) connects to the interwebs!
MAU and MNZ (and related things) are hosted mostly in a datacenter up here in #Gladstone (Central #Queensland, #Australia).
I have redundant paths to Brisbane - one 9G connection to #NextDC B1, and a 10G Wave to B2.
I peer with #EdgeIX in both B1 and B2, #IXAU in B1, and Aussie Broadband and Telstra up here.
All this networking and redundancy and stuff is why it is REALLY REALLY ANNOYING that something dumb like 'I forgot to enable #VRRP on the Mastodon edge router' causes everything to fall apart.
The backend is #VyOS and #Mikrotik routers/switches, with #Proxmox for the virtualization, #Ubuntu for the OS's and #ZFS for the storage.
I'm also an #IPv6 guy, so everything is IPv6 enabled. In fact, I love it so much, I own the website. https://ipv6.com.au/
Oh, the reason all this infrastructure exists is because of my real job - https://aussievoip.com.au/ - So if you want #VoIP, or #SIP phones, you know a guy.
-
As I sent out a personalised invite to the #AusNOG crowd to join MAU, I thought you lot may be interested in how #MastodonAU (and #MastodonNZ) connects to the interwebs!
MAU and MNZ (and related things) are hosted mostly in a datacenter up here in #Gladstone (Central #Queensland, #Australia).
I have redundant paths to Brisbane - one 9G connection to #NextDC B1, and a 10G Wave to B2.
I peer with #EdgeIX in both B1 and B2, #IXAU in B1, and Aussie Broadband and Telstra up here.
All this networking and redundancy and stuff is why it is REALLY REALLY ANNOYING that something dumb like 'I forgot to enable #VRRP on the Mastodon edge router' causes everything to fall apart.
The backend is #VyOS and #Mikrotik routers/switches, with #Proxmox for the virtualization, #Ubuntu for the OS's and #ZFS for the storage.
I'm also an #IPv6 guy, so everything is IPv6 enabled. In fact, I love it so much, I own the website. https://ipv6.com.au/
Oh, the reason all this infrastructure exists is because of my real job - https://aussievoip.com.au/ - So if you want #VoIP, or #SIP phones, you know a guy.
-
As I sent out a personalised invite to the #AusNOG crowd to join MAU, I thought you lot may be interested in how #MastodonAU (and #MastodonNZ) connects to the interwebs!
MAU and MNZ (and related things) are hosted mostly in a datacenter up here in #Gladstone (Central #Queensland, #Australia).
I have redundant paths to Brisbane - one 9G connection to #NextDC B1, and a 10G Wave to B2.
I peer with #EdgeIX in both B1 and B2, #IXAU in B1, and Aussie Broadband and Telstra up here.
All this networking and redundancy and stuff is why it is REALLY REALLY ANNOYING that something dumb like 'I forgot to enable #VRRP on the Mastodon edge router' causes everything to fall apart.
The backend is #VyOS and #Mikrotik routers/switches, with #Proxmox for the virtualization, #Ubuntu for the OS's and #ZFS for the storage.
I'm also an #IPv6 guy, so everything is IPv6 enabled. In fact, I love it so much, I own the website. https://ipv6.com.au/
Oh, the reason all this infrastructure exists is because of my real job - https://aussievoip.com.au/ - So if you want #VoIP, or #SIP phones, you know a guy.
-
More random nerding, I just replaced my last HP G7 blade with a 'v zero' CPU. This is the new one, a G9 with a v4 CPU, but only 128gb of RAM - RAM is crazy expensive these days. However, if you notice that #MastodonAU and #MastodonNZ are unusually fast and responsive, you now know why!
-
More random nerding, I just replaced my last HP G7 blade with a 'v zero' CPU. This is the new one, a G9 with a v4 CPU, but only 128gb of RAM - RAM is crazy expensive these days. However, if you notice that #MastodonAU and #MastodonNZ are unusually fast and responsive, you now know why!
-
More random nerding, I just replaced my last HP G7 blade with a 'v zero' CPU. This is the new one, a G9 with a v4 CPU, but only 128gb of RAM - RAM is crazy expensive these days. However, if you notice that #MastodonAU and #MastodonNZ are unusually fast and responsive, you now know why!
-
More random nerding, I just replaced my last HP G7 blade with a 'v zero' CPU. This is the new one, a G9 with a v4 CPU, but only 128gb of RAM - RAM is crazy expensive these days. However, if you notice that #MastodonAU and #MastodonNZ are unusually fast and responsive, you now know why!
-
Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster!
The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM.
Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves.
I am doing my best to avoid any more visits from the #fuckup-fairy
Now to finish #MastodonNZ
-
Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster!
The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM.
Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves.
I am doing my best to avoid any more visits from the #fuckup-fairy
Now to finish #MastodonNZ
-
Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster!
The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM.
Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves.
I am doing my best to avoid any more visits from the #fuckup-fairy
Now to finish #MastodonNZ
-
Well I gotta say, moving #MastodonAU 's database to a pure SSD storage backend has made a massive difference. Whooohoo, she's much faster!
The storage servers are a pair of #Dell R640s, with 2x Gold 6150 (2.7GHz 18 Cores each) and 256GB of RAM.
Each one has 8 x 1TB SSDs, in a #ZFS raidz2 pool, and they automatically replicate between themselves.
I am doing my best to avoid any more visits from the #fuckup-fairy
Now to finish #MastodonNZ