#rukiinet — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #rukiinet, aggregated by home.social.
-
Migrated #RukiiNet from Ingress-Nginx to #traefik
https://kubernetes.io/blog/2026/01/29/ingress-nginx-statement/
-
Migrated #RukiiNet from Ingress-Nginx to #traefik
https://kubernetes.io/blog/2026/01/29/ingress-nginx-statement/
-
Migrated #RukiiNet from Ingress-Nginx to #traefik
https://kubernetes.io/blog/2026/01/29/ingress-nginx-statement/
-
Migrated #RukiiNet from Ingress-Nginx to #traefik
https://kubernetes.io/blog/2026/01/29/ingress-nginx-statement/
-
Migrated #RukiiNet from Ingress-Nginx to #traefik
https://kubernetes.io/blog/2026/01/29/ingress-nginx-statement/
-
#RukiiNet was down for 9 hours 45 minutes today because we installed a new #ElectricCar charger, which cut the power momentarily which apparently killed my UPS. It took me a while to notice.
-
#RukiiNet was down for 26 hours because five cities lost the internet due to cable damage:
https://www.granadahoy.com/granada/averia-caida-internet-provocada-obras-metro-area-metropolitana-granada-solucionada_0_2004026651.html -
Yesterday and today, the whole of Spain, Portugal and parts of France, and possibly as per some reports, parts of Italy and Belgium lost their power inexplicably. Italy and Belgium reports might have been spurious. This happened about 12:30 local time here in Spain.
The power was out for about 14 hours here, a bit less for some areas as they brought up the grid piece by piece.
The mobile base stations were the first to go.
We have solar panels and batteries which carry over the night, and it was a sunny day, so we maintained power here in the house over the event.
This Mastodon instance was inaccessible for 11 hours, because apparently the fiber optic substations lost their emergency battery power after 6 hours or so, and the final internet connection we had was cut.
There is still no conclusive determination for what happened or why. It seems the most likely explanation currently is some sort of an atmospheric event causing resonance in the electric grid.
https://elpais.com/economia/2025-04-28/apagon-electrico-masivo-en-espana.html
At night, our house was pretty much the only one with power. The other lights visible in the photo are largely cars and flashlights. We still have our Christmas lights in the yard...
The stars were well visible in the sky.
So, lessons learned:
1. Apocalypses (or pseudo-apocalypses) don't necessarily come with a bang. They come with an eery silence and darkness, and no one knows what is happening.
2. I am still unable to play my local Plex series with the good TV, because everything seems to require internet connectivity for no good reasons. I can play them with my phone and on laptop (but with proprietary codecs and without hardware accelerations it's in practice impossible). Playstation Plex simply refuses to function without internet.
3. Don't run multiple high power cooking appliances at once because it overloads the inverter.
4. The house water pump isn't in the emergency power circuit, which is inconvenient and needs fixing.
5. Need to have physical keys for things easily available, and not hidden in places you'd look in the last.
6. Even if the local internet networks would still work, if the global networks aren't available the ISPs happily just cut off all the networks and don't even try to maintain just local networks for convenient local emergency communications purposes. Ditto for electricity.#SelfHosting #RukiiNet #apocalypse #resilience #blackout #Spain
-
#RukiiNet relatively uneventfully upgraded to v4.3.4.
Needed setting up some new cryptokeys and mastodon-streaming is apparently a separate Docker image nowadays.
-
Started investigating why I was still getting HTTP 413 Entity Too Large errors for uploading larger image files on this instance. Caddy reverse proxy doesn't impose limits and Mastodon default limits are way higher. What was the cause then?
The Kubernetes nginx-ingress has a default limit of 1 MB requests, and that was giving the error. It was easy to fix by adding an annotation to the ingress.
-
Upgraded #RukiiNet to the newest Mastodon, 4.2.10.
Everything went painlessly.
#SelfHosting -
Rukii.net was down for 7 hours 28 minutes because NAS got a new IP address even though it was statically bound. Changed it to a static IP.
#RukiiNet #SelfHosting -
Rukii.net was down for 7 hours 28 minutes today because NAS decided to get a new IP address even though it was bound statically. Switched it to a static IP address.
#RukiiNet #SelfHosting -
Rukii.net was down for 12 hours as I was installing a new NAS. This is now done, gives more space and stops all the old issues we had with OpenEBS Jiva.
#RukiiNet #SelfHosting -
Rukii.net will go down shortly tomorrow, Saturday morning Spanish time due to solar panel installation. Need to cut the power to rewire it in a more sensible manner.
#RukiiNet #SelfHosting -
#RukiiNet #SelfHosting: I had an outage of one and a half days because there was a small fire in the house. The pellet boiler caught fire. Lots of losses of electricity in the event itself and the related fixes.
-
#RukiiNet #SelfHosting update:
Rukii.net has moved from #Benalmádena to #LasGabias! Everything works again, after an outage of 5 days and 13 hours due to the move. -
#RukiiNet #SelfHosting: rukii.net will have an outage 9.11 as I will be moving to Granada. I will try to set things up as fast as possible there, but some downtime will be unavoidable.
-
#RukiiNet #SelfHosting, rukii.net will go down for a day or two 24.11 or so when we're moving to Granada. The plan is to get the servers up again as soon as possible, but it will necessarily cause some downtime.
-
#SelfHosting #RukiiNet update:
Updated Mastodon to version v4.2.0. Lots of new features! Be sure to set your settings to allow your public posts to be indexed and searched if you desire so!
Relating to that, there was a jam-tangle with the database migrations, which at first corrupted the database and then with backups it got stuck in an infinite iteration in the duplicate removal of some preview images. I had to manually remove the duplicate relations, which allowed the migration to proceed. The site was down for some 2 hours, and I lost two posts of mine, one of which I rewrote and the other shall remain as if deleted. -
#RukiiNet #SelfHosting update:
Migrated away from NFS and Jiva for Postgresql. Now using high-availability Postgres with two replicas instead. Works much faster and is more robust to failures. -
#RukiiNet: Backups are now tested, added a cronjob which checks that everything is fine in the cluster and emails me immediately if not. I am also getting alarms from StatusCake if the site is unreachable. Will add an automatic scheduled test for backups being good as well to make sure everything is good in the future as well.
-
#RukiiNet update:
Sad news: A large amount of data was lost. I was doing maintenance and deleted a jiva pod to make it remount stuff. This was a mistake as it was actually the controller pod, not a replica pod. It caused the whole Postgres volume to be nuked.Anyhow, I have backups for just this thing, the latest from the day before. In two different instances to be sure. And from the last four days to be very sure. No worries? Murphy's law strikes again.
Turns out I had two separate backup scripts running on different times on two different nodes, to be extra sure that I had backups. The thing is, these processes took a long time and it seems they actually co-occurred, and using the same sqlc dump file name on the pod, they both wrote to it at the same time.
All backups were corrupted. On all the nodes. On all the days. The last backup which wasn't was from the end of January... I restored that one, and everything I could recover from the latter backups, which sadly didn't include all the posts. Luckily pretty much no one on my instance had posted much, except me. Sadly the follow lists were gone as well.
-
#RukiiNet update: Still keeps happening. 1 hour downtime today. This seems to have been because of a Linux kernel upgrade, and some problem with the Realtek driver. It fails due to a netdev watchdog.
Trying to use the official Realtek DKMS drivers instead, conveniently in Ubuntu distribution but not installed by default, r8168-dkms.
Edit:
This caused problems, had to add a kernel parameter pci=nommconf. Should work now... -
#RukiiNet: The nodes kept going down all the time. Upgraded Ubuntus, Kuberneteses and all those things on everything. Caused a downtime of some 8 hours.
-
Another downtime for #RukiiNet, for 6 hours. One of my NUC Kubernetes nodes died. Couldn't SSH in. Rebooting helped. Hope it won't happen again.
-
#RukiiNet had another outage of 3 hours because of Bitnami Postgresql file system permissions again. Now it should never reoccur as I am manually fixing the permissions at every container boot.
-
#RukiiNet had another 4.5 hour outage, which I used to analyze why it gets stuck.
It is because for some reason the dynamic NFS server changes the mounted directory fsGroup from correct 1001 to 1000, no clue why. All the settings are set to make it 1001, and it typically works. However, when the NFS volume is mounted, it automatically changes the ownership back to 1000, and hence Postgres fails to start.
Anyhow, hacked around fsGroup setting here and there, and now it seems to (correctly?) set the S (setgid) bit for group on for the directory in question. Hopefully it now works and doesn't randomly die.
-
#RukiiNet had another 10.5 hour outage due to some glitch in the volume mounts I haven't yet managed to pin-point. Doesn't seem like my fixes so far have made a difference.
Rebooting all the nodes tends to fix the issue, it's some weird mounting order thing. Need to get it once more on some day I am not completely exhausted and deep dive into causes and effects.
-
#RukiiNet had another 7.5 hour outage because of a regression in NFS server pods. They used to be assigned to one specific node, to prevent the irritating issue of failing to mount an "already bound" volume. This configuration disappeared in the recent NFS upgrade, in a way I was hoping that upgrade would have fixed the issue. It did not. Now all the NFS pods are nodeName'd to a specific node, which in practice makes the uptime better than the counterfactual...
-
#RukiiNet: Will be upgrading the instance to v4.1.4.
Edit: Upgraded without an incident. -
#RukiiNet #SelfHosting update:
Going to update Mastodon from 4.1.0 -> 4.1.2 and also deploy sitemap. -
#RukiiNet #SelfHosting update:
Here's an all-time uptime graph from #StatusCake. Managed the first week with a 100% uptime recently, and the weekly uptime is now in single-nine. Monthly uptime isn't quite there yet, but improving steadily. -
#RukiiNet #SelfHosting update:
I lost the last update in a database clean up, damn.
Stuff is stabilizing, there are now three cluster nodes all of which are rock solid, no more daily outages.
Everything works otherwise, but ElasticSearch pods need some tuning with the permissions which is probably caused by some upstream change in Bitnami container images.
After that, I need to dive deep in that file permissions issue as it spans all the pods, some of which I temporarily worked around. It seems volume mounts and all are ok, but when the container entrypoint creates some directories, it tends to create them so that it doesn't itself have access to them anymore. Created with 0000 permissions...
Anyhow, my backups were run by #Curie node, which was the unstable one, so I'll need to migrate those to some other node for it to work now that Curie isn't a node anymore.
I would hope a new era of stability for rukii.net would start from here. -
#RukiiNet #SelfHosting update:
Just after writing this #Curie went down again, and it didn't help that the #NFS pods were all on a different node. It all went down regardless.Even got some data corruption again, it's always a huge manual hassle to bring everything back up. I read somewhere that #MicroK8S tends to be bad with hard reboots if some specific singleton cluster pods like coredns or calico or nfc controller or hostpath provisioner are on the node which goes down. I wonder if it's possible to just add replicas for those...
I found a new (old and known) bug with #OpenEBS, and a mitigation. In some cases, #Jiva has replicas in a readonly state for a moment as it syncs the replicas, and if the moon phase is correct, there's an apparent race condition where the iSCSI mounts become read-only, even though the underlying volume has already become read-write.
To fix this is to go to the node which mounted these, do "mount | grep ro,", and ABSOLUTELY UNDER NO CIRCUMSTANCE UNMOUNT (learned the hard way). Instead, I think it's possible to just remount these rw.
There's also an irritating thing where different pods run their apps with different UIDs, and the Dynamic NFS Provisioner StorageClass needs to be configured to mount the stuff with the same UID. I originally ran this by just setting chmod 0777, but the apps insist on creating files with a different permission set, so when their files get remounted, their permissions stay but the UID changes, and after a remount they don't have write access to the files anymore.
This compounds with the fact that each container runs on its own UID, so each needs its own special StorageClass for that UID... Gods.
I got the new #IntelNUC for the fourth node in the cluster to replace the unstable Curie node, but memories for it are coming Thursday.
-
#RukiiNet #SelfHosting update:
Ok, the issue with Dynamic NFS Provisioner that it tends to leave some weird zombie containers running after hard reboots, perhaps a #Containerd issue, tends to leave these very important pods hanging in ContainerCreating, because "Volume already mounted at more than one place":
https://github.com/openebs/dynamic-nfs-provisioner/issues/153So, I can fix that manually by assigning/moving these pods to specific hosts it seems, because ReadWriteOnce these things lean upon is node specific. But it's all so very fragile.
I would love it if #MicroK8S #Kubernetes and #Containerd could get over their issues and just kill the zombie containers which shouldn't exist as far as Kubernetes knows. Or if only they would somehow show me which container is mounting the offending share, I could make a script to kill those...
Also, Redis volumes are now moved back to being hostpaths. These don't need to be persistent, actually if they are it just produces more trouble for hard reboots, because the AOF file gets corrupted and needs manual fixing or just deleting and recreating the PVC which is easier actually. I tried Jiva volumes for those which was insane, it took so much resources that my Mastodon became very slow, and it doesn't make sense to replicate Redis disk backs anyhow.
Still waiting for a new #IntelNUC node to maybe swap out #Curie which keeps being unstable due to hardware issues. It tends to go down daily now. And when it goes down, pods migrate around and take a lot of memory, and this interestingly kills the kubelite daemons on the two other nodes, and I have to reboot them as well manually when that happens...
-
#RukiiNet #SelfHosting update:
After fighting with an unstable host due to memories I believe, and the whole cluster always going down when one node went down, I deep dove into what actually happens.Turned out as I had installed #OpenEBS #Jiva for replicating volumes on my #MicroK8S using their official Helm charts, it didn't work at all. It just made all the replicas correctly, went through all the motions and then stored all the data on a single pod ephemeral store! I had to take the cluster down to investigate, that took a weekend more or less.
I found out that if OpenEBS Jiva is installed as a MicroK8S plug-in, pointing it specifically to their Git main, and not to a tagged release which doesn't work, then it works. I tried to find out the difference between the Helm chart this installs and the one I had installed, with no luck. I think I installed OpenEBS Jiva Helm chart before and that didn't work, while MicroK8S plug-in installs OpenEBS chart with Jiva enabled as a setting.
Anyhow, ordered a new #IntelNUC again, to reduce my maintenance actions due to one flaky node as well. But as I recreated basically the whole cluster with functioning OpenEBS now, and restored all the (daily) backups once again, it seems everything works and probably single node going down shouldn't take the whole Mastodon down anymore regardless.
During all this I have also filed a lot of issues to the relevant projects on GitHub and documented my findings there, so that people getting the same errors can find solutions.
-
#RukiiNet #SelfHosting update:
I think there is an issue with #OpenEBS #Jiva replication on the #Kubernetes cluster.It seems all the volume data goes to the Jiva controller pod for that PVC, and it stores all the data in /var/snap/microk8s/common/var/lib/kubelet/pods/PODID/volumes/kubernetes.io~csi/PVCID/mount.
That directory perhaps should be a mount to somewhere, but isn't. It just stores the plain files there on a single node.
The Jiva replicas, three per every volume claim, are set up correctly but the file data doesn't seem to go to those...