#bldrweb — Public Fediverse posts

Hey #bldrweb - per my previous messages[1], I'll be SHUTTING DOWN https://toot.bldrweb.org TOMORROW (originally planned for Tuesday but life's busy).

If you need to export any last backups, do so RIGHT NOW!

I'm going to make a last-ditch effort to upgrade to fix some of the redirect problems, but this could very well break some things.

[1] https://toot.bldrweb.org/@thomasw/114337968784737969

Also, if any other #colorado or #boulder related instances want to send out an invitation to our users, please let me know!

I'll also be happy to provide backup admin support for other small, public instances once this I get through closing up all the loose ends with #bldrweb

**Important announcement for my toot.bldrweb.org users**

TLDR: #bldrweb will be shutting down on July 15th (three months from now).

Time has come to close the doors on bldrweb. I've enjoyed doing this, but I'd now like to turn my attention elsewhere. Happy to share more thoughts in the replies!

Per the covenant[1], I'm giving a 3 month notice to allow for switching to a new instance.

(continued)

[1] https://joinmastodon.org/covenant

Hey #bldrweb - we need to chat about financial sustainability of our instance.

Monthly costs: $50
Monthly income: $20

I could probably get the monthly costs down a bit by reworking the config, but at the moment, I'd have to take time away from paid work to do that.

So, what do you think? Do we have enough users/funds to justify keeping this instance up and running or should we plan to consolidate with another server?

https://opencollective.com/bldrweb

#fediadmin

Hey #bldrweb - I need to so some upgrades to the server. I plan to do them sometime tomorrow. Hopefully in the morning, but TBD. If timing and schedule allows, I might do it tonight. I'll update again before I start the update.

Hey #bldrweb - we'll be performing a reboot to install some updates at 1 pm today.

Hey #bldrweb - I need to put together a mailing list of the active users of our instance so I can communicate about down times and future changes to the server.

Please DM or email me your preferred email address:
[email protected]

We also need to talk about financials and server funding. Here's the summary:

- Monthly expenses: ($50)
- Monthly contributions: $35
- 2024 deficit: ($320)

TL;DR need more contributors! Contributions page:
https://opencollective.com/bldrweb

Hey friends, asking for some confirmation here if you’re outside of #bldrweb that our posts are visible and you can interact with them.

I’ve had a couple of interactions with mutuals on other instances but it does seem like “less” than before our server was down a couple days for maintenance.

TIA https://toot.bldrweb.org/@thomasw/113813170052315712

#bldrweb - gotten a few upgrades and improvements implemented, but I think there's still an issue with the email. Still working on things.

Hey again #bldrweb - I'm going to run an upgrade today to see if that fixes anything. TBD on how long this will take.

Hey #bldrweb - Seems our Mastodon server is not sending out emails (new accounts, password resets, missed notifications, etc.). I'm working on the issue, but it's a tricky one, so I'm not sure how long it'll take to resolve. Please DM or email me ([email protected]) if you run into issues.

Server was running slow, so that took a lot longer than expected, but seems we're back online. #bldrweb

Hey #bldrweb - need to apply an upgrade, so we'll have a brief downtime in a few minutes.

Hey #bldrweb - need to reboot the server for some maintenance things. Hoping this will be a quick downtime.

Hey there #bldrweb - I created a Matrix room for us to use as a communications back channel in case the server is ever down. Of course you can always reach me by email, but there are some cases where Matrix (a slack- or discord-like interface) might be better. Go check it out and say hello:
https://matrix.to/#/#bldrweb:matrix.org

Hey #bldrweb - looks like we're back online! Sorry for the downtime, the upgrade process ran smoothly, but it just took a long time to copy/move all the files.

#bldrweb - going offline in ~15 minutes

#bldrweb - Upgrade with some downtime happening today!

Sometime midday today I'm going to take the server offline so I can run a backup then apply some upgrades. Tentatively planning for ~11AM, but I'll try to give you all a 30min warning.

we might be back online!
#bldrweb

Progress continues. #bldrweb

Server going offline now. Wish me luck!

#bldrweb

Hey #bldrweb - around 2:30p today, I'm going to make another attempt at the improvements to the server (more disk space, separate data volume, smaller/cheaper VPS node).

I'm estimating about 2h of downtime if everything goes smoothly.

I setup a chat (mattermost) on a separate server for side-channel comms and Q&A: https://smart-seriema.pikapod.net/bldrweb/

And server status dashboard is here: https://stats.uptimerobot.com/72VWwuzKPV

Well #bldrweb , the update failed again. Apologies for the long down times, but restoring from the backup takes at least 2 or 3 hours (there's a lot of small files).

But this time I found a new clue! I think something is broken with the Yunhost updater for glitch social. Others are reporting upgrade errors as well. https://github.com/YunoHost-Apps/glitchsoc_ynh/issues/291

I'll probably need to do another couple reboots sometime today. I'm not sure of the best way to announce down times (we need a secondary comms channel). I'll put some thought into it and share here after some morning caffeine.

Hey #bldrweb - still working on upgrades here. Will need to take the server offline again sometime soon. Sorry for all the up/down.

Hey #bldrweb 🚨 - I'm going to take the server offline sometime in the next 30min or so to do some upgrades. I'm hoping this will be ~1h downtime.

Fear not - we have a more recent backup this time around, so data loss should be pretty unlikely!

Hey #bldrweb - our @phanpy interface is back up and running again (it was a casualty of yesterday's upgrade misadventures). Anyone can use it, bldrweb or otherwise:
https://phanpy.bldrweb.org/

If you're not familiar with Phanpy:
https://github.com/cheeaun/phanpy

Hey #bldrweb: I'm very sorry to share that a server upgrade earlier today failed, and I was forced to restore from a backup that had been taken earlier this morning. Any posts/messages/interactions created between approx 2AM and noon today might have been lost. Please check your toots/likes/DMs.

I'm happy to share details with anyone who wants more info, but just know that I have some short-term fixes already applied and later this week I'm going to work on improving our server setup to prevent this sort of thing in the future.

Please accept my humble apologies!

Hey #bldrweb - I'll be applying a server upgrade shortly. Possibly momentary downtime. Will reply here when I'm done.

Heyo #bldrweb - I just updated our local version of @phanpy to the latest release. Check it out:
https://phanpy.bldrweb.org/#/

And we're back online #bldrweb !