#openebs — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #openebs, aggregated by home.social.
-
I've been a little rough and irresponsible with my #baremetal #Kubernetes cluster, especially when it comes to randomly rebooting nodes. Today I fixed that.
I'm running a bunch of somewhat delicate workloads, including database clusters with CSIs like #Longhorn and #OpenEBS. Checking if everything is in working order has been demanding task and often something I've skipped before rebooting or upgrading nodes - occasionally with horrific results.
Last night I finally took the time and wrote a pretty thorough script that checks that everything is working and healthy, before politely cordoning off a node, draining it and applying upgrades.
I felt so confident today that I tested it by running this new safe upgrade script for all the nodes in the cluster - and it worked! All nodes are now fully upgraded and running kernel 6.12.73 on Debian 13.
This also fixes the outstanding issue caused by #Hetzner no longer supporting obtaining IP addresses through DHCP.
-
I've been a little rough and irresponsible with my #baremetal #Kubernetes cluster, especially when it comes to randomly rebooting nodes. Today I fixed that.
I'm running a bunch of somewhat delicate workloads, including database clusters with CSIs like #Longhorn and #OpenEBS. Checking if everything is in working order has been demanding task and often something I've skipped before rebooting or upgrading nodes - occasionally with horrific results.
Last night I finally took the time and wrote a pretty thorough script that checks that everything is working and healthy, before politely cordoning off a node, draining it and applying upgrades.
I felt so confident today that I tested it by running this new safe upgrade script for all the nodes in the cluster - and it worked! All nodes are now fully upgraded and running kernel 6.12.73 on Debian 13.
This also fixes the outstanding issue caused by #Hetzner no longer supporting obtaining IP addresses through DHCP.
-
Just when I thought I was almost out of yaks (and finally installing #Airflow!), #terragrunt got confused and started demanding to create resources that already exist, which broke #openEBS, which broke... sigh ...
Another 2 days of work later, #argocd is installed in my neurons and my cluster, and most of my config is refactored "enough". I swear we'll actually get to do some #datascience someday folks...
Big data on a tiny budget is hard!
-
A quick follow-up: Given that caching an #LVM volume requires setting up a cache and a cache meta volume per data volume you need cached, this just gets needlessly complicated when dealing with a storage provisioner that constantly creates new volumes. Caching seems absolutely necessary with the hardware I have available and LVM seems like the only way to go. There are alternatives, like #flashcache by #Synology, but they're not part of the main stream #kernel.
The solution? It ain't pretty, and I have still to test the performance implications of it. I've created a 4TB file on a filesystem cached by LVM, created a #loopback device from it, initialized that as an LVM physical volume, then added a volume group. The #OpenEBS Local LVM provisioner now uses this group to provision persistent volumes in #Kubernetes.
Not sure about the implications of nested LVM'ing, but it solves the #caching issue.
-
A quick follow-up: Given that caching an #LVM volume requires setting up a cache and a cache meta volume per data volume you need cached, this just gets needlessly complicated when dealing with a storage provisioner that constantly creates new volumes. Caching seems absolutely necessary with the hardware I have available and LVM seems like the only way to go. There are alternatives, like #flashcache by #Synology, but they're not part of the main stream #kernel.
The solution? It ain't pretty, and I have still to test the performance implications of it. I've created a 4TB file on a filesystem cached by LVM, created a #loopback device from it, initialized that as an LVM physical volume, then added a volume group. The #OpenEBS Local LVM provisioner now uses this group to provision persistent volumes in #Kubernetes.
Not sure about the implications of nested LVM'ing, but it solves the #caching issue.
-
I'm running my own #Kubernetes cluster on bare metal at #Hetzner. I'm going through different iterations of node configurations, and trying to figure out how to squeeze the most out of a hardware configuration with a 10TB+ HDD drive and two 512GB #NVME drives. #bcache has apparently been discontinued, which leaves me with #dmcache, which is nicely built into #LVM.
One catch though: you need to create NVME cache volumes for each of the data volumes you create. This is all well and good if you're using a single volume for everything, and tools like #Longhorn work well in this kind of setup. Others, like the #OpenEBS LVM provisioner don't. There is a version in the making that just provisions raw disk images as files on whatever filesystem you have, but it's still lacking some features that are vital to me - primarily the ability to offer up StorageClasses that provision on different paths of the node, so I can pick a fast or slow filesystem to store it on.
Tips for other solutions?
-
I'm running my own #Kubernetes cluster on bare metal at #Hetzner. I'm going through different iterations of node configurations, and trying to figure out how to squeeze the most out of a hardware configuration with a 10TB+ HDD drive and two 512GB #NVME drives. #bcache has apparently been discontinued, which leaves me with #dmcache, which is nicely built into #LVM.
One catch though: you need to create NVME cache volumes for each of the data volumes you create. This is all well and good if you're using a single volume for everything, and tools like #Longhorn work well in this kind of setup. Others, like the #OpenEBS LVM provisioner don't. There is a version in the making that just provisions raw disk images as files on whatever filesystem you have, but it's still lacking some features that are vital to me - primarily the ability to offer up StorageClasses that provision on different paths of the node, so I can pick a fast or slow filesystem to store it on.
Tips for other solutions?
-
Having recently experienced a rather horrible #Kubernetes crash, I'm looking for #backup solutions. We're good with PostgreSQL since we're using #CNPG with remote transaction logs to an offsite #S3 bucket. I need something for volumes and maybe Kubernetes resources. #Longhorn offers S3 backups for it's own volumes, but for other #CSI like local #OpenEBS, maybe #Velero? Thoughts?
-
Having recently experienced a rather horrible #Kubernetes crash, I'm looking for #backup solutions. We're good with PostgreSQL since we're using #CNPG with remote transaction logs to an offsite #S3 bucket. I need something for volumes and maybe Kubernetes resources. #Longhorn offers S3 backups for it's own volumes, but for other #CSI like local #OpenEBS, maybe #Velero? Thoughts?
-
I've been doing things I shouldn't with #Kubernetes. We're using a replicated #MinIO cluster as the storage backend on #mstdndk, which requires a boat load of storage, especially if you forget to specify any kind of retention. So far, the quick workaround for a full disk, was just to expand the filesystem. Since we're replicating across nodes, we're using #OpenEBS #LVM for local storage. Poor partitioning means we're running out of storage on the volume group, but even worse - PVCs sizes were increased before checking if we had space for it. Kubernetes is now stuck in a most unfortunate situation - it can't grow the local filesystem, as the volume group is full and you're not allowed to decrease the size request. What then? Cue https://github.com/etcd-io/auger - a tools that allows you to edit #K8s resources directly in #etcd. Obviously you should never do this, but with steady hands and clinical precision, you can get yourself out of a pickle like mine. Size was reverted and PVCs were unstuck.
-
I've been doing things I shouldn't with #Kubernetes. We're using a replicated #MinIO cluster as the storage backend on #mstdndk, which requires a boat load of storage, especially if you forget to specify any kind of retention. So far, the quick workaround for a full disk, was just to expand the filesystem. Since we're replicating across nodes, we're using #OpenEBS #LVM for local storage. Poor partitioning means we're running out of storage on the volume group, but even worse - PVCs sizes were increased before checking if we had space for it. Kubernetes is now stuck in a most unfortunate situation - it can't grow the local filesystem, as the volume group is full and you're not allowed to decrease the size request. What then? Cue https://github.com/etcd-io/auger - a tools that allows you to edit #K8s resources directly in #etcd. Obviously you should never do this, but with steady hands and clinical precision, you can get yourself out of a pickle like mine. Size was reverted and PVCs were unstuck.
-
A unfortunate side affect of Longhorn experiencing I/O latency and saturation, is that volumes attached to pods become remounted read-only. This has very very unfortunate side effects on running databases, caches etc. Any tips on making #Longhorn behave would be greatly appreciated. I've looked briefly into OpenEBS' distributed, replicated storage but the requirements are currently not allowing it - specifically, replicated #OpenEBS needs an entire physical disk for itself.
-
A unfortunate side affect of Longhorn experiencing I/O latency and saturation, is that volumes attached to pods become remounted read-only. This has very very unfortunate side effects on running databases, caches etc. Any tips on making #Longhorn behave would be greatly appreciated. I've looked briefly into OpenEBS' distributed, replicated storage but the requirements are currently not allowing it - specifically, replicated #OpenEBS needs an entire physical disk for itself.
-
#kubernetes question: our deployment stack is #skaffold plus #helm. We have #OpenEBS deployed, I'm trying to add their monitoring mixin to deployment. But they want to deploy whole kube-prometheus, which we already have. I have a script to generate their manifests and grab just those 4 files we're interested in, and have helm grab those. But here comes the question: how do I tie in the generation with the rest of the deployment?
[1/2] -
#kubernetes question: our deployment stack is #skaffold plus #helm. We have #OpenEBS deployed, I'm trying to add their monitoring mixin to deployment. But they want to deploy whole kube-prometheus, which we already have. I have a script to generate their manifests and grab just those 4 files we're interested in, and have helm grab those. But here comes the question: how do I tie in the generation with the rest of the deployment?
[1/2] -
man, my #openebs jiva experience has been nightmare. it requires manual intervention like every day on my cluster, and the community is just a sea of ignored issues and slack messages. i don't know why the talos team recommends it.
am i the only one who feels this way?
-
Very Nice. There are many features of #Kubernetes that is helpful in allowing #Mastodon to scale
-
I'm not posting "days" anymore, my days aren't consecutive and I don't always have stuff to say.
I've managed to get my production #Kubernetes cluster running with #Longhorn after a period of struggles with #Openebs. I've also been working with my friend to build fresh #alpinelinux base images with #Jenkins and shipping them off to #harbor.
#selfhosted #100DaysOfHomeLab -
#RukiiNet #SelfHosting update:
Just after writing this #Curie went down again, and it didn't help that the #NFS pods were all on a different node. It all went down regardless.Even got some data corruption again, it's always a huge manual hassle to bring everything back up. I read somewhere that #MicroK8S tends to be bad with hard reboots if some specific singleton cluster pods like coredns or calico or nfc controller or hostpath provisioner are on the node which goes down. I wonder if it's possible to just add replicas for those...
I found a new (old and known) bug with #OpenEBS, and a mitigation. In some cases, #Jiva has replicas in a readonly state for a moment as it syncs the replicas, and if the moon phase is correct, there's an apparent race condition where the iSCSI mounts become read-only, even though the underlying volume has already become read-write.
To fix this is to go to the node which mounted these, do "mount | grep ro,", and ABSOLUTELY UNDER NO CIRCUMSTANCE UNMOUNT (learned the hard way). Instead, I think it's possible to just remount these rw.
There's also an irritating thing where different pods run their apps with different UIDs, and the Dynamic NFS Provisioner StorageClass needs to be configured to mount the stuff with the same UID. I originally ran this by just setting chmod 0777, but the apps insist on creating files with a different permission set, so when their files get remounted, their permissions stay but the UID changes, and after a remount they don't have write access to the files anymore.
This compounds with the fact that each container runs on its own UID, so each needs its own special StorageClass for that UID... Gods.
I got the new #IntelNUC for the fourth node in the cluster to replace the unstable Curie node, but memories for it are coming Thursday.
-
#RukiiNet #SelfHosting update:
Just after writing this #Curie went down again, and it didn't help that the #NFS pods were all on a different node. It all went down regardless.Even got some data corruption again, it's always a huge manual hassle to bring everything back up. I read somewhere that #MicroK8S tends to be bad with hard reboots if some specific singleton cluster pods like coredns or calico or nfc controller or hostpath provisioner are on the node which goes down. I wonder if it's possible to just add replicas for those...
I found a new (old and known) bug with #OpenEBS, and a mitigation. In some cases, #Jiva has replicas in a readonly state for a moment as it syncs the replicas, and if the moon phase is correct, there's an apparent race condition where the iSCSI mounts become read-only, even though the underlying volume has already become read-write.
To fix this is to go to the node which mounted these, do "mount | grep ro,", and ABSOLUTELY UNDER NO CIRCUMSTANCE UNMOUNT (learned the hard way). Instead, I think it's possible to just remount these rw.
There's also an irritating thing where different pods run their apps with different UIDs, and the Dynamic NFS Provisioner StorageClass needs to be configured to mount the stuff with the same UID. I originally ran this by just setting chmod 0777, but the apps insist on creating files with a different permission set, so when their files get remounted, their permissions stay but the UID changes, and after a remount they don't have write access to the files anymore.
This compounds with the fact that each container runs on its own UID, so each needs its own special StorageClass for that UID... Gods.
I got the new #IntelNUC for the fourth node in the cluster to replace the unstable Curie node, but memories for it are coming Thursday.
-
#RukiiNet #SelfHosting update:
After fighting with an unstable host due to memories I believe, and the whole cluster always going down when one node went down, I deep dove into what actually happens.Turned out as I had installed #OpenEBS #Jiva for replicating volumes on my #MicroK8S using their official Helm charts, it didn't work at all. It just made all the replicas correctly, went through all the motions and then stored all the data on a single pod ephemeral store! I had to take the cluster down to investigate, that took a weekend more or less.
I found out that if OpenEBS Jiva is installed as a MicroK8S plug-in, pointing it specifically to their Git main, and not to a tagged release which doesn't work, then it works. I tried to find out the difference between the Helm chart this installs and the one I had installed, with no luck. I think I installed OpenEBS Jiva Helm chart before and that didn't work, while MicroK8S plug-in installs OpenEBS chart with Jiva enabled as a setting.
Anyhow, ordered a new #IntelNUC again, to reduce my maintenance actions due to one flaky node as well. But as I recreated basically the whole cluster with functioning OpenEBS now, and restored all the (daily) backups once again, it seems everything works and probably single node going down shouldn't take the whole Mastodon down anymore regardless.
During all this I have also filed a lot of issues to the relevant projects on GitHub and documented my findings there, so that people getting the same errors can find solutions.
-
#RukiiNet #SelfHosting update:
After fighting with an unstable host due to memories I believe, and the whole cluster always going down when one node went down, I deep dove into what actually happens.Turned out as I had installed #OpenEBS #Jiva for replicating volumes on my #MicroK8S using their official Helm charts, it didn't work at all. It just made all the replicas correctly, went through all the motions and then stored all the data on a single pod ephemeral store! I had to take the cluster down to investigate, that took a weekend more or less.
I found out that if OpenEBS Jiva is installed as a MicroK8S plug-in, pointing it specifically to their Git main, and not to a tagged release which doesn't work, then it works. I tried to find out the difference between the Helm chart this installs and the one I had installed, with no luck. I think I installed OpenEBS Jiva Helm chart before and that didn't work, while MicroK8S plug-in installs OpenEBS chart with Jiva enabled as a setting.
Anyhow, ordered a new #IntelNUC again, to reduce my maintenance actions due to one flaky node as well. But as I recreated basically the whole cluster with functioning OpenEBS now, and restored all the (daily) backups once again, it seems everything works and probably single node going down shouldn't take the whole Mastodon down anymore regardless.
During all this I have also filed a lot of issues to the relevant projects on GitHub and documented my findings there, so that people getting the same errors can find solutions.
-
#RukiiNet #SelfHosting update:
I think there is an issue with #OpenEBS #Jiva replication on the #Kubernetes cluster.It seems all the volume data goes to the Jiva controller pod for that PVC, and it stores all the data in /var/snap/microk8s/common/var/lib/kubelet/pods/PODID/volumes/kubernetes.io~csi/PVCID/mount.
That directory perhaps should be a mount to somewhere, but isn't. It just stores the plain files there on a single node.
The Jiva replicas, three per every volume claim, are set up correctly but the file data doesn't seem to go to those...
-
#RukiiNet #SelfHosting update:
I think there is an issue with #OpenEBS #Jiva replication on the #Kubernetes cluster.It seems all the volume data goes to the Jiva controller pod for that PVC, and it stores all the data in /var/snap/microk8s/common/var/lib/kubelet/pods/PODID/volumes/kubernetes.io~csi/PVCID/mount.
That directory perhaps should be a mount to somewhere, but isn't. It just stores the plain files there on a single node.
The Jiva replicas, three per every volume claim, are set up correctly but the file data doesn't seem to go to those...
-
Ok, many of my #Microk8s #Kubernetes issues are apparently distro related. #Ubuntu wants to install a snap package of MicroK8S, which would be all well and fine, it's a relatively recent version, and uses Kubelite as a snap service. But there are of course some version incompatibilities.
I had previous issues with MicroK8S plug-ins which in some cases installed very old and buggy versions of the charts in question. Learned to avoid those and install manually where appropriate. In most cases it works, it's just #OpenEBS that is problematic.
Now it seems that when I've had issues with the cluster, I've followed general instructions and used microk8s start. Apparently that's a huge mistake. MicroK8S has opinions about cluster tooling versions which apparently differ from what snap wants to run, and all this leads to really weird incompatibility issues with Unix sockets where multiple things conflict over them.
Hopefully the cluster stays up a bit better now. I can't recommend trying to run #Mastodon on MicroK8S, it's a forever pain with no, or often misleading documentation.
-
Ok, many of my #Microk8s #Kubernetes issues are apparently distro related. #Ubuntu wants to install a snap package of MicroK8S, which would be all well and fine, it's a relatively recent version, and uses Kubelite as a snap service. But there are of course some version incompatibilities.
I had previous issues with MicroK8S plug-ins which in some cases installed very old and buggy versions of the charts in question. Learned to avoid those and install manually where appropriate. In most cases it works, it's just #OpenEBS that is problematic.
Now it seems that when I've had issues with the cluster, I've followed general instructions and used microk8s start. Apparently that's a huge mistake. MicroK8S has opinions about cluster tooling versions which apparently differ from what snap wants to run, and all this leads to really weird incompatibility issues with Unix sockets where multiple things conflict over them.
Hopefully the cluster stays up a bit better now. I can't recommend trying to run #Mastodon on MicroK8S, it's a forever pain with no, or often misleading documentation.
-
@tero I remember running into so many issues when I was looking at #openebs as my storage provider. My biggest thing was tying it with a backup solution #velero since openebs was creating a bunch of services for its volumes. Ended up being a nightmare. But the biggest thing was since I had single node instance, any power failure was devastating to the storage. Not all of the volumes would come back up.
Since then I switched to #longhorn from #rancher and have had 0 issues.
-
@tero I remember running into so many issues when I was looking at #openebs as my storage provider. My biggest thing was tying it with a backup solution #velero since openebs was creating a bunch of services for its volumes. Ended up being a nightmare. But the biggest thing was since I had single node instance, any power failure was devastating to the storage. Not all of the volumes would come back up.
Since then I switched to #longhorn from #rancher and have had 0 issues.
-
@mlink, I actually described the scheme here in rough detail regarding #OpenEBS #Jiva and #DynamicNFSProvisioner:
https://www.costacoders.es/news/2022-12-24_kubernetes/The persistence volume replica stores are ultimately hostpaths.
-
@mlink, I actually described the scheme here in rough detail regarding #OpenEBS #Jiva and #DynamicNFSProvisioner:
https://www.costacoders.es/news/2022-12-24_kubernetes/The persistence volume replica stores are ultimately hostpaths.
-
@tero Nice! I love my #kubernetes instance. It houses almost everything we have running including #mastodon. With the #nuc though how do you plan to use #openebs? Planning to have some external storage? #selfhosting
-
@tero Nice! I love my #kubernetes instance. It houses almost everything we have running including #mastodon. With the #nuc though how do you plan to use #openebs? Planning to have some external storage? #selfhosting
-
Day 5: Seem to be perpetually having issues with my #kubernetes cluster. Using #OpenEBS for config and database storage and the volumes keep becoming corrupted. Anyone know a fix without buying 3-4 additional servers and begging for no power outages?
#selfhosted #100DaysOfHomeLab -
Oh wow, tested back up restoration process as a side-effect of battling the whole day with #OpenEBS, #Jiva, #iscsi and #NFS. I know too much about how these things work now. Need to write a blog article.
Now all data is three-fold replicated across the cluster and back up restoration process has been tested.
#SelfHosting #Kubernetes -
Oh wow, tested back up restoration process as a side-effect of battling the whole day with #OpenEBS, #Jiva, #iscsi and #NFS. I know too much about how these things work now. Need to write a blog article.
Now all data is three-fold replicated across the cluster and back up restoration process has been tested.
#SelfHosting #Kubernetes -
Everything is incomplete, full of bugs, documentation is terrible...
I found this problem from #MicroK8S now trying to deploy #OpenEBS:
https://github.com/canonical/microk8s/issues/3639
#SelfHosting -
Everything is incomplete, full of bugs, documentation is terrible...
I found this problem from #MicroK8S now trying to deploy #OpenEBS:
https://github.com/canonical/microk8s/issues/3639
#SelfHosting -
Two #NUC nodes making the home #Mastodon #SelfHosting #Kubernetes #cluster. The third node is a boring desktop PC upstairs.
I'm currently fighting with #OpenEBS. Apparently the local hostpath provider is unable to elect itself the leader if I don't turn the default leader election off...
Still can't actually mount the replicated PersistentVolumes... -
Two #NUC nodes making the home #Mastodon #SelfHosting #Kubernetes #cluster. The third node is a boring desktop PC upstairs.
I'm currently fighting with #OpenEBS. Apparently the local hostpath provider is unable to elect itself the leader if I don't turn the default leader election off...
Still can't actually mount the replicated PersistentVolumes... -
Got a new #TinyServer for #Mastodon #Kubernetes #cluster! Now with 4 #cores and 32GB memory, in an #Intel #NUC. Now I have three nodes in the cluster and can start experimenting with high availability data storage in an #OpenEBS.
#SelfHosting -
Got a new #TinyServer for #Mastodon #Kubernetes #cluster! Now with 4 #cores and 32GB memory, in an #Intel #NUC. Now I have three nodes in the cluster and can start experimenting with high availability data storage in an #OpenEBS.
#SelfHosting -
There, now the services are on #MetalLB IPs, and the #ReverseProxy forwards rukii.net to those. Works perfectly, #ZeroDowntime.
Now everything else is #HighAvailability, except the persistent volumes. For #OpenEBS I'll need a third cluster node, which is in the mail... And of course the reverse proxy and the internet connections aren't redundant yet, in principle I could set up another internet connection e.g. over 4G, but for now the fiber and the proxy are reliable enough.
