Live and recent posts from across the Fediverse tagged #keyboardvagabond, aggregated by home.social.
As part of the constant tinkering on #keyboardvagabond, today I did a few fixes:
1) the talos registry mirror fallbacks were wrong, so #harborregistry went down and so did #cloudflared and #tailscale. Scary. I got back in to the system and separated the mirror and fallback configs (for disaster only) and then scaled harbor to 2 replicas, hopefully to prevent this failure in the future
2) added a #piefed watchdog script that restarts if the queue is stuck (cont'd) #kubernetes #selfhosting
More progress on updating #keyboardvagabond! I finally fully removed the last vestiges on ingress-nginx. I needed it because I still had harbor registry available externally so I could push images, but I now host #gitea and have #cicd internally. Today I completely removed ingress-nginx in favor of #istio and #harborregistry is behind the same tunnels as everything else.
#kubernetes #selfHosting #nginx #softwareEngineering #softwareDevelopment #k8s
#keyboardvagabond is now migrated from using #ingress-nginx to #gatewayApi. It took most of the day, but almost the entire networking stack has been swapped out for what I should have been using anyway.
I'll try to get a write-up or something on it later to go into more detail. Overall, it wasn't too bad.
#keyboardvagabond just upgraded #writefreely from v0.15.1 to v0.16.0. I was previously using someone else's image for the docker container, but just switched to using official images as part of my own build for my #kubernetes #cluster. Everything is up and running with the latest features! Woo!
I finally got to add #DeepL #translations to this #mastodon instance! It took a while because they flagged me as suspicious, but this week it finally went through and configuration was super easy!
Moar adventures in #selfhosting for #keyboardvagabond! I think that I finally got the longhorn too-many-s3-calls networking issue resolved https://blog.keyboardvagabond.com/michael-dileos-software-and-tech-blog/longhorns-backup-cost-savings-or-not-getting-surprise-bills
I also found that the #bitnami #helmchart for #harborRegistry didn't get updated with the latest migration they did, so I had to find out that they moved their repositories to `bitnamilegacy` and update my helm chart. That took a while to figure out why the image pulls failed!
[Blog Post] - Longhorn's Backup Cost Savings – or not getting surprise bills
Longhorn's Backup Cost Savings - or not getting surprise bills
I'm experimenting with where to put these types of blog posts. I have a copy over at the Keyboard Vagabond blog
Longhorn, the system that is used to manage storage for Keyboard Vagabond, performs regular backups and disaster recovery management. I noticed that on the last few billing cycles, the costs for S3 cloud storage with BackBlaze was about $25 higher than expected, and given that the last two bills were like this, it's not a fluke.
The costs are from s3_list_objects, over 5M calls last month. It turns out this is a common thing that has been mentioned in github, reddit, Stack Overflow, etc. The solution seems to be just to turn it off. It doesn't seem to be required for backups and disaster recovery to work and Longhorn seems to be doing something very incorrectly to be making all of these calls.
...
data:
default-resource.yaml: |-
...
"backupstore-poll-interval": "0"
My expectation is that future billing cycles should be well under $10/month for storage. The current daily average storage size is 563GB, or $3.38 per month.
#kubernetes #longhorn #s3 #programming #selfhosting #cloudnative #keyboardvagabond
Adventures in #selfhosting! I'm a dummy.
I was wondering for a while why pixelfed.keyboardvagabond.com was only using large images and just being overall a terribly slow experience. I saw that image upload processing jobs would run, but would not do anything.
I verified that things were being uploaded to #backblaze #s3 and that the #cloudflare #cdn cache was showing HIT, but downloading 5MB images just takes a while, especially when there's a bunch of them and you're working out of hostels that limit everyone to 5Mbps 😭
After debugging for a while with #claude - it's actually quite good at doing investigations, especially when you need a bunch of #kubectl #kubernetes commands - and went through a ton of options.
I had at one point messaged in the #matrixchat that I thought that pixelfed wasn't using thumbnails and @dansup responded. (I'm really sorry for my dumb!)
Finally, it found what I had done. I got confused about where the media was being stored... I was mounting one level too high, so the job never found the files, but they still got uploaded to S3...ok.
Anyway, here's my fix, change the volume mounts from - notice the additional /pixelfed:
- name: app-storage
mountPath: /var/www/storage
- name: cache-storage
mountPath: /var/www/bootstrap/cache
to
- name: app-storage
mountPath: /var/www/pixelfed/storage
- name: cache-storage
mountPath: /var/www/pixelfed/bootstrap/cache
I'm now feeling a lot better about the coming soon (TM) announcement of keyboard vagabond joining the fediverse!
Adventures in #selfhosting!
I just finished failing to do a "correct" and "proper" upgrade of cloud-native postgres, #cnpg, from using standard #longhorn volume backups to the barman cloud plugin.
I got the plugin loaded accord to the migration docs, but couldn't get it to write to #s3, nor could the pods become ready. I worked at it for hours, but I saw lots of other people online and recently having the same issues and log messages that I was.
The reason I did this in the first place was that I noticed that I had some duplicate backup jobs causing issues with #fluxcd reconciliation.
In the end, I gave up and went back to the original longhorn backups, which have worked and I've already done disaster recovery with (don't ask), and deleted the duplicates.
Currently I'm waiting for the previous primary/write node to fully restart and clear out the barman side car. Then I'll turn flux back on and hopefully things will be good.
More adventures in #selfhosting! I'm going through more tests before I get to launch #keyboardvagabond! Woo! I just tried to do some uploads to #pixelfed like I did with #instagram. The results are...frustrating.
I can't add any text to or over a story, so then I tried to upload the images that my phone made before instagram and I found a few issues right off of the bat that eventually caused me to give up the post.
First, I couldn't upload the same photo or video files because pixelfed doesn't think they're images or videos.
Second, I tried uploading the originals, but couldn't because the apps and website don't compress them first, so I had to download a video compressor on my phone.
Then, when I tried to upload all of them together in a post, it uploaded then gave me an error that said a post can be either all photos or a single video.
I gave up. I think I might have to put that pixelfed mainly focuses on images. As it is now, I don't think that I'll be able to get too many people to switch off of instagram for keeping in touch. It's still a cool place though, and I can see it as a break from the rest of corporate media while still using instagram to keep in touch with friends and their stories.
I am taking a break off of work for a bit, so maybe I can get to know #PHP a bit and propose or make contributions. It's not that easy diving into new languages and frameworks, though.
More progress towards #keyboardvagabond, a fediverse space for digital nomads and backpackers! It's a baby step and a place holder, but www.keyboardvagabond.com is a thing that loads.
I already have 7 issues on my #gitea instance. It's more of an outline at this point, but things are getting closer. There's still a decent amount of work to do for prepping to go live in general for all of the services.
#fediverse #selfhost #kubernetes #piefed #mastodon #pixelfed #writefreely #picsur #digitalnomad #vagabond
It turns out that there's a problem when you want to run everything in the #fediverse.
I guess it's time to purchase a 3rd node ( ^ . ^ ;) ( T _ T )
#kubernetes #keyboardvagabond #outofmemory #dudewheresmyram.
Today's #adventuresInSelfHosting, I was having trouble with #cloudnativePG where I'd always have n-1 pods stable and they'd constantly restart between the replicas in a massive loop, then eventually switch with the primary and continue the process. So, I set the instance count to 3 so there'd at least be one stable read replica at a time.
I finally found out what was wrong. I have my #cnpg cluster in the postgresql-system namespace and I happened to see that I had an operator running in the default cnpg-system namespace. I don't know how long it had been there, so both it and the one in my namespace were both competing for the state. Deleting and cleaning up that old cluster brought immediate stability.
I also realized that I wasn't overriding the default #php configuration for #pixelfed, so when I uploaded an image taking on my phone, the web server would restart. I bumped the php memory up to 1GB for now. For the expected userbase for the upcoming #keyboardvagabond #fediverse space, this should be fine.
Right now the services are running well, but I need to do more testing and get mastodon into an "interesting" state for new visitors. Pixelfed seems the hardest for me in terms of getting content onto the server so that it doesn't look barren.
The todo list for now is:
It's getting close! The services are essentially ready, just not necessarily turned on for signups until I'm ready for a pre-launch or full launch. I want to make sure things are in a good state.
But with the 2 node #kubernetes #cluster, I think things should be good!
By then end, it should look like:
More adventures in self-hosting the fediverse
More adventures in standing up fediverse services
More updates on the upcoming #keyboardvagabond #fediverse #isntance! Woo! I'm getting excited!
I've gotten good progress with #piefed, #mastodon, and #pixelfed. I'm still trying to get #hcaptcha set up and working. I did accomplish correctly working out how I should mount volumes for piefed so that imports would work correctly (unexpected directory). I also fixed an issue with nginx metrics exports and #cilium network policies, so now I have some nice new dashboards. I also spoke with a friend and #nomad who does #graphicdesign about a new mascot!
We're getting close, there's still more things that I want to finalize before going live, but some of the big ones are:
Today's setup for the #fediverse was mostly for #mastodon - finding some default block lists, setting policies and rules, etc. But I did get to set up a #relay for some travel themed topics for the soon to arrive #keyboardvagabond collection of fediverse apps. I'm getting closer, though there's still more working to do, such as getting hcaptcha to work properly.
But what I do have is:
Upcoming:
#Piefed, #Bookwyrm, #Matrix, and a landing page.
I have some other ideas in the works as well, but my goal is to have a proper launch within a month. Let's hope. I hate setting timelines, but I have reasons haha.