#footimac — Public Fediverse posts

@EdwinG ya I only have 1 thread ever stuck it seems, so I’m not too worried about it. @jfmezei found a very relevant bugreport relating to latency which would make sense. Worst case scenario, I could move the redis container back onto #footiMac. That should take the latency down to near zero.

ooo! #footiMac and its newly relocated docker redis/db containers up and running just in time for the new Mastodon v4.7.1! OK! :)

#timeToUpdate #mastoAdmin #selfhost

https://github.com/mastodon/mastodon/releases/tag/v4.7.1

@jfmezei this is good info. It would make sense that I increased the latency by moving the redis work across the continent. So if latency impacts the scheduler that would make sense. I am not too worried because I never see more than one busy thread at a time in the scheduler queues so it doesn't seem to be a critical issue. But if it does become an issue, I could try moving the redis docker container back onto #footiMac. It's less than 1GB so it's easy to move around and doesn't add a lot to the impact on the old mac.

Here’s a #mastoadmin #selfhost question.

Why would the scheduler process be up around 1000 seconds for average completion time when all others are under 5 seconds?

Is it something to be concerned about?

#footiMac #debian #mastodon

Oooh. And it dawned on me that since there is no db and redis taking up space on #footiMac I could probably add another set of processes, so I am up to 3 processes now with 15 queues each doing the push/pull/ingress/default which helps keep the latency down!

FootiMac feeling snappy again!

#selfhost #mastodon #docker #debian

by jove. i think we have liftoff!

#footiMac is now doing the webserver and other frontend mastodon stuff while passing the redis and db functions to my external host running docker containers monitored in portainer!

not too shabby!

#selfhost #mastodon #docker #debian

backing up takes a minute… #footiMac #selfhost

well *&^t… it worked on the first try!

REDIS now hosted from my external (leasedweb) host!
#portainer #selfhost #mastoadmin #footiMac

trying out #portainer to run docker on my external host and... possibly... move my redis and db containers to the external.

trying to move redis first!

the portainer interface is not bad.
#selfhost #mastoadmin #footiMac

just separated the stream and mastodon services a little by having the webcams send to the #newIMac ’s wifi address instead of the same ethernet address everything else is using.

Hopefully that stabilizes things a little. The whole machine seemed to be losing its connectivity periodically, which made things freak out and eventually die.

#Selfhost #FootiMac

@EdwinG complete and successful at 14:40.

So that's 35 minutes on old #footiMac

it always takes so much longer than you want…. lol zzzzz #newOldiMac #selfhost #footiMac #newOldiMac #mastoAdmin #qemu #colima #cantHelpMyself

“what are you trying to do now?”
you might be asking.

Colima on macOS Ventura and later uses the Apple-supplied 'vz’ engine.

#newOldiMac is running Ventura.

So maybe docker was interacting poorly with that engine, especially because it was the first macOS version with it… maybe it's buggy?

So I'm installing ‘qemu', which is the native colima engine on all pre-Ventura Macs and Linux.

building it is taking a bit…

#selfhost #footiMac #newOldiMac #mastoAdmin #qemu #colima #cantHelpMyself

I said I wasn't going to fiddle around with my #selfhost #footiMac #newOldiMac anymore... well.

that didn't last. lol

#mastoAdmin #qemu #colima #cantHelpMyself

Time to update to v4.6.1!
#SelfHost #MastoAdmin #footiMac

so far so good. #footiMac and #newOldimac are stable and services are being restarted periodically.

ok. *this time* i think my #selfhost is going to be stable or at least self-recoverable. Both #footiMac and #newOldiMac are now set to periodically restart to catch any times when the become unresponsive or colima dies.
#mastoadmin

Had to do a revert back to v4.5.11 on #selfhost as I messed up the update steps. lol. Which is a first. Totally brain cramped. Anyway, after restoring the db to this morning just in case I am happy to be back at the point of updating to v4.6!

Because…. Eeeeeee new things are oouuuuttt and I want them nooowww… and yes I am doing it on the bus on my ippaaaaaaaad. 🤓 😱 😀
#v46 #mastodon #mastoadmin #footiMac #newOldiMac

oo.. good sign at least... seeing Puma has updated as is ruby in the mastodon-web process, so that's encouraging. But it was the db:migrate commands I flubbed... so we'll see.

#selfhost #footiMac

hmm... a little worried I may have messed up the sequencing on the v4.6 upgrade path on #selfhost #footiMac… fingers extra crossed that this works!

Checking to make sure #Docker on #newOldiMac is still running. It is still inexplicably unstable inside the Colima VM. I think it is the database taking it down when it gets too busy. But I’ve added RAM, taken it away, ensured the volumes have lots of free space, etc etc.

If anyone has experience running docker in Colima and has ideas, I’m all ears.

The good news is #footiMac (running Debian) is rock solid. As soon as the database returns, it resumes sending requests.

#mastoAdmin

I think I can confidently say the transition from failing #footiMac trying to handle too many tasks and especially the mastodon database, to a tag team between footiMac and the ten-year younger #newoldiMac running the db and redis process under Colima and docker is complete and successful.

It has been solid for 24 hours now. It’s responsive. It’s almost fast. 😂 even have elasticsearch reactivated.

And the best part: I learned more about docker and feel much more comfortable with it.

Yay
#docker #mastoadmin #selfhost

hmmm... just thought of a new purpose for #footiMac now that it has a little extra CPU/RAM headspace but is hurting on the SSD front.

We have so many devices in the house now, and only one (my laptop) gets backed up regularly.

Maybe I could turn on an SMB service and servce up a remote NAS/Time Machine with a few large HDDs.

#homelab #NAS

I think it is funny, ironic, nerdy, and meta (the proper literal sense) that I have maxed out the 2017 iMac by wanting to install btop while watching it be maxed out on htop.

And yet... it's still posting this, so if stuff doesn't break down now, surely it never will? Even the swap is having a moment. Ooof. I need to be nicer to my machines. lol

#newOldimac #selfhost #mastoAdmin #footiMac #docker #colima #homebrew

just tweaked/upped web concurrrency and max threads settings in my .env.production settings for Mastodon to give #footiMac a few more processes to handle requests and things are humming along nicely.

I think a big takeway for me on this #selfhost journey is that except on the very smallest one person and minimal followers instance, the database (and possibly redis) function should always be on a different machine or container from mastodon.

#footiMac was clearly having real trouble handling the database activity once it got busy. Even with an SSD now with two bad blocks (which may have been caused by that over-activity), it is way more stable.

#newOldiMac #docker #mastoAdmin #colima #iMac

@fname oh yes, ElasticSearch is definitely a memory hog and it has been enable since Monday so should already be counted in. That hit should be on #footimac i would think as that is where the mastodon processes and media files are stored? But it is possible that there is a knock on effect on the database and it could have delayed activating until now. A lot of mysteries still in this system!

It's a day later and #footiMac and #newOldiMac are still up and running. Disk usage is stable. Current RAM usage for the database has gone up to 3GB from 1GB since 9PM last night. I'd like to know why. It's not any busier.

I'm running the backup script then later today I'll see if I can investigate more on why the RAM usage fluctuates so much.

I would like to know why it is more stable. Is it because I've simply given it enough room to expand and contract in the colima VM?

Is it because I set the clocks to the same timezone?

Inquiring minds... but I am happy that it is happy. And I have learned a lot and gotten more comfortable with docker. Which has been a goal almost since I joined Mastodon in Nov 2022.

I’m hoping to look for a new macMini now to replace #newOldiMac. If I can get that an SSD to replace the failing SSD in #footiMac and a large HDD for backup that'd be great.

#docker #mastoAdmin #colima #iMac

dare I say the #footiMac + #newOldiMac combo are currently delivering snappy mastodon performance.

#mastoAdmin #selfhost

ok, I've managed to clear 250GB of space on my iMac and give colima access to 200GB.

Now that I have a handle on how to monitor disk space both in colima and in each docker container, I'll be monitoring it closely.

The bad news is I had to revert back to a database from Friday night so I've lost some content. But I'm again increasing my understanding of working within a docker environment within a colima virtualization.

I’m updating the docker images for both the db and the redis now and will take current backups from both so I can be closer to realtime.

I'll get there! I refuse to let this beat me. I feel like I'm close!

Kind of in the background of all the drama on #newOldiMac trying to get this colima and docker stuff to behave, trusty #footiMac is much happier now that some load is off. But I still have my eye on a new Samsung branded and hopefully reliable 2TB SSD to replace the one that is failing that gave rise to all of this.

screenshot of realtime “hacking" according to @Jade lol.

#mastoAdmin #selfhost #colima #macos #linux #docker #footiMac #newOldiMac

Under threat of jinxing myself, I think the server has stabilized a little. Or at least redis is restarting automatically now when it dies. I flushed the redis cache thanks to some helpful advice as well, so hopefully it will work itself out. It is still handingly old posts but is catching up.

I am going to try to post a gardening #poolpond post now! Stay tuned. And thank you everyone for your support, assistance and patience as I nerd about with my personal
#selfhost #footiMac #newoldiMac #mastodon instance