#openebs — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #openebs, aggregated by home.social.
-
I've been a little rough and irresponsible with my #baremetal #Kubernetes cluster, especially when it comes to randomly rebooting nodes. Today I fixed that.
I'm running a bunch of somewhat delicate workloads, including database clusters with CSIs like #Longhorn and #OpenEBS. Checking if everything is in working order has been demanding task and often something I've skipped before rebooting or upgrading nodes - occasionally with horrific results.
Last night I finally took the time and wrote a pretty thorough script that checks that everything is working and healthy, before politely cordoning off a node, draining it and applying upgrades.
I felt so confident today that I tested it by running this new safe upgrade script for all the nodes in the cluster - and it worked! All nodes are now fully upgraded and running kernel 6.12.73 on Debian 13.
This also fixes the outstanding issue caused by #Hetzner no longer supporting obtaining IP addresses through DHCP.
-
I've been a little rough and irresponsible with my #baremetal #Kubernetes cluster, especially when it comes to randomly rebooting nodes. Today I fixed that.
I'm running a bunch of somewhat delicate workloads, including database clusters with CSIs like #Longhorn and #OpenEBS. Checking if everything is in working order has been demanding task and often something I've skipped before rebooting or upgrading nodes - occasionally with horrific results.
Last night I finally took the time and wrote a pretty thorough script that checks that everything is working and healthy, before politely cordoning off a node, draining it and applying upgrades.
I felt so confident today that I tested it by running this new safe upgrade script for all the nodes in the cluster - and it worked! All nodes are now fully upgraded and running kernel 6.12.73 on Debian 13.
This also fixes the outstanding issue caused by #Hetzner no longer supporting obtaining IP addresses through DHCP.
-
I've been a little rough and irresponsible with my #baremetal #Kubernetes cluster, especially when it comes to randomly rebooting nodes. Today I fixed that.
I'm running a bunch of somewhat delicate workloads, including database clusters with CSIs like #Longhorn and #OpenEBS. Checking if everything is in working order has been demanding task and often something I've skipped before rebooting or upgrading nodes - occasionally with horrific results.
Last night I finally took the time and wrote a pretty thorough script that checks that everything is working and healthy, before politely cordoning off a node, draining it and applying upgrades.
I felt so confident today that I tested it by running this new safe upgrade script for all the nodes in the cluster - and it worked! All nodes are now fully upgraded and running kernel 6.12.73 on Debian 13.
This also fixes the outstanding issue caused by #Hetzner no longer supporting obtaining IP addresses through DHCP.
-
I've been a little rough and irresponsible with my #baremetal #Kubernetes cluster, especially when it comes to randomly rebooting nodes. Today I fixed that.
I'm running a bunch of somewhat delicate workloads, including database clusters with CSIs like #Longhorn and #OpenEBS. Checking if everything is in working order has been demanding task and often something I've skipped before rebooting or upgrading nodes - occasionally with horrific results.
Last night I finally took the time and wrote a pretty thorough script that checks that everything is working and healthy, before politely cordoning off a node, draining it and applying upgrades.
I felt so confident today that I tested it by running this new safe upgrade script for all the nodes in the cluster - and it worked! All nodes are now fully upgraded and running kernel 6.12.73 on Debian 13.
This also fixes the outstanding issue caused by #Hetzner no longer supporting obtaining IP addresses through DHCP.
-
I've been a little rough and irresponsible with my #baremetal #Kubernetes cluster, especially when it comes to randomly rebooting nodes. Today I fixed that.
I'm running a bunch of somewhat delicate workloads, including database clusters with CSIs like #Longhorn and #OpenEBS. Checking if everything is in working order has been demanding task and often something I've skipped before rebooting or upgrading nodes - occasionally with horrific results.
Last night I finally took the time and wrote a pretty thorough script that checks that everything is working and healthy, before politely cordoning off a node, draining it and applying upgrades.
I felt so confident today that I tested it by running this new safe upgrade script for all the nodes in the cluster - and it worked! All nodes are now fully upgraded and running kernel 6.12.73 on Debian 13.
This also fixes the outstanding issue caused by #Hetzner no longer supporting obtaining IP addresses through DHCP.
-
Just when I thought I was almost out of yaks (and finally installing #Airflow!), #terragrunt got confused and started demanding to create resources that already exist, which broke #openEBS, which broke... sigh ...
Another 2 days of work later, #argocd is installed in my neurons and my cluster, and most of my config is refactored "enough". I swear we'll actually get to do some #datascience someday folks...
Big data on a tiny budget is hard!
-
Just when I thought I was almost out of yaks (and finally installing #Airflow!), #terragrunt got confused and started demanding to create resources that already exist, which broke #openEBS, which broke... sigh ...
Another 2 days of work later, #argocd is installed in my neurons and my cluster, and most of my config is refactored "enough". I swear we'll actually get to do some #datascience someday folks...
Big data on a tiny budget is hard!
-
A quick follow-up: Given that caching an #LVM volume requires setting up a cache and a cache meta volume per data volume you need cached, this just gets needlessly complicated when dealing with a storage provisioner that constantly creates new volumes. Caching seems absolutely necessary with the hardware I have available and LVM seems like the only way to go. There are alternatives, like #flashcache by #Synology, but they're not part of the main stream #kernel.
The solution? It ain't pretty, and I have still to test the performance implications of it. I've created a 4TB file on a filesystem cached by LVM, created a #loopback device from it, initialized that as an LVM physical volume, then added a volume group. The #OpenEBS Local LVM provisioner now uses this group to provision persistent volumes in #Kubernetes.
Not sure about the implications of nested LVM'ing, but it solves the #caching issue.
-
A quick follow-up: Given that caching an #LVM volume requires setting up a cache and a cache meta volume per data volume you need cached, this just gets needlessly complicated when dealing with a storage provisioner that constantly creates new volumes. Caching seems absolutely necessary with the hardware I have available and LVM seems like the only way to go. There are alternatives, like #flashcache by #Synology, but they're not part of the main stream #kernel.
The solution? It ain't pretty, and I have still to test the performance implications of it. I've created a 4TB file on a filesystem cached by LVM, created a #loopback device from it, initialized that as an LVM physical volume, then added a volume group. The #OpenEBS Local LVM provisioner now uses this group to provision persistent volumes in #Kubernetes.
Not sure about the implications of nested LVM'ing, but it solves the #caching issue.
-
A quick follow-up: Given that caching an #LVM volume requires setting up a cache and a cache meta volume per data volume you need cached, this just gets needlessly complicated when dealing with a storage provisioner that constantly creates new volumes. Caching seems absolutely necessary with the hardware I have available and LVM seems like the only way to go. There are alternatives, like #flashcache by #Synology, but they're not part of the main stream #kernel.
The solution? It ain't pretty, and I have still to test the performance implications of it. I've created a 4TB file on a filesystem cached by LVM, created a #loopback device from it, initialized that as an LVM physical volume, then added a volume group. The #OpenEBS Local LVM provisioner now uses this group to provision persistent volumes in #Kubernetes.
Not sure about the implications of nested LVM'ing, but it solves the #caching issue.
-
A quick follow-up: Given that caching an #LVM volume requires setting up a cache and a cache meta volume per data volume you need cached, this just gets needlessly complicated when dealing with a storage provisioner that constantly creates new volumes. Caching seems absolutely necessary with the hardware I have available and LVM seems like the only way to go. There are alternatives, like #flashcache by #Synology, but they're not part of the main stream #kernel.
The solution? It ain't pretty, and I have still to test the performance implications of it. I've created a 4TB file on a filesystem cached by LVM, created a #loopback device from it, initialized that as an LVM physical volume, then added a volume group. The #OpenEBS Local LVM provisioner now uses this group to provision persistent volumes in #Kubernetes.
Not sure about the implications of nested LVM'ing, but it solves the #caching issue.
-
A quick follow-up: Given that caching an #LVM volume requires setting up a cache and a cache meta volume per data volume you need cached, this just gets needlessly complicated when dealing with a storage provisioner that constantly creates new volumes. Caching seems absolutely necessary with the hardware I have available and LVM seems like the only way to go. There are alternatives, like #flashcache by #Synology, but they're not part of the main stream #kernel.
The solution? It ain't pretty, and I have still to test the performance implications of it. I've created a 4TB file on a filesystem cached by LVM, created a #loopback device from it, initialized that as an LVM physical volume, then added a volume group. The #OpenEBS Local LVM provisioner now uses this group to provision persistent volumes in #Kubernetes.
Not sure about the implications of nested LVM'ing, but it solves the #caching issue.
-
I'm running my own #Kubernetes cluster on bare metal at #Hetzner. I'm going through different iterations of node configurations, and trying to figure out how to squeeze the most out of a hardware configuration with a 10TB+ HDD drive and two 512GB #NVME drives. #bcache has apparently been discontinued, which leaves me with #dmcache, which is nicely built into #LVM.
One catch though: you need to create NVME cache volumes for each of the data volumes you create. This is all well and good if you're using a single volume for everything, and tools like #Longhorn work well in this kind of setup. Others, like the #OpenEBS LVM provisioner don't. There is a version in the making that just provisions raw disk images as files on whatever filesystem you have, but it's still lacking some features that are vital to me - primarily the ability to offer up StorageClasses that provision on different paths of the node, so I can pick a fast or slow filesystem to store it on.
Tips for other solutions?
-
I'm running my own #Kubernetes cluster on bare metal at #Hetzner. I'm going through different iterations of node configurations, and trying to figure out how to squeeze the most out of a hardware configuration with a 10TB+ HDD drive and two 512GB #NVME drives. #bcache has apparently been discontinued, which leaves me with #dmcache, which is nicely built into #LVM.
One catch though: you need to create NVME cache volumes for each of the data volumes you create. This is all well and good if you're using a single volume for everything, and tools like #Longhorn work well in this kind of setup. Others, like the #OpenEBS LVM provisioner don't. There is a version in the making that just provisions raw disk images as files on whatever filesystem you have, but it's still lacking some features that are vital to me - primarily the ability to offer up StorageClasses that provision on different paths of the node, so I can pick a fast or slow filesystem to store it on.
Tips for other solutions?
-
I'm running my own #Kubernetes cluster on bare metal at #Hetzner. I'm going through different iterations of node configurations, and trying to figure out how to squeeze the most out of a hardware configuration with a 10TB+ HDD drive and two 512GB #NVME drives. #bcache has apparently been discontinued, which leaves me with #dmcache, which is nicely built into #LVM.
One catch though: you need to create NVME cache volumes for each of the data volumes you create. This is all well and good if you're using a single volume for everything, and tools like #Longhorn work well in this kind of setup. Others, like the #OpenEBS LVM provisioner don't. There is a version in the making that just provisions raw disk images as files on whatever filesystem you have, but it's still lacking some features that are vital to me - primarily the ability to offer up StorageClasses that provision on different paths of the node, so I can pick a fast or slow filesystem to store it on.
Tips for other solutions?
-
I'm running my own #Kubernetes cluster on bare metal at #Hetzner. I'm going through different iterations of node configurations, and trying to figure out how to squeeze the most out of a hardware configuration with a 10TB+ HDD drive and two 512GB #NVME drives. #bcache has apparently been discontinued, which leaves me with #dmcache, which is nicely built into #LVM.
One catch though: you need to create NVME cache volumes for each of the data volumes you create. This is all well and good if you're using a single volume for everything, and tools like #Longhorn work well in this kind of setup. Others, like the #OpenEBS LVM provisioner don't. There is a version in the making that just provisions raw disk images as files on whatever filesystem you have, but it's still lacking some features that are vital to me - primarily the ability to offer up StorageClasses that provision on different paths of the node, so I can pick a fast or slow filesystem to store it on.
Tips for other solutions?
-
I'm running my own #Kubernetes cluster on bare metal at #Hetzner. I'm going through different iterations of node configurations, and trying to figure out how to squeeze the most out of a hardware configuration with a 10TB+ HDD drive and two 512GB #NVME drives. #bcache has apparently been discontinued, which leaves me with #dmcache, which is nicely built into #LVM.
One catch though: you need to create NVME cache volumes for each of the data volumes you create. This is all well and good if you're using a single volume for everything, and tools like #Longhorn work well in this kind of setup. Others, like the #OpenEBS LVM provisioner don't. There is a version in the making that just provisions raw disk images as files on whatever filesystem you have, but it's still lacking some features that are vital to me - primarily the ability to offer up StorageClasses that provision on different paths of the node, so I can pick a fast or slow filesystem to store it on.
Tips for other solutions?
-
Having recently experienced a rather horrible #Kubernetes crash, I'm looking for #backup solutions. We're good with PostgreSQL since we're using #CNPG with remote transaction logs to an offsite #S3 bucket. I need something for volumes and maybe Kubernetes resources. #Longhorn offers S3 backups for it's own volumes, but for other #CSI like local #OpenEBS, maybe #Velero? Thoughts?
-
Having recently experienced a rather horrible #Kubernetes crash, I'm looking for #backup solutions. We're good with PostgreSQL since we're using #CNPG with remote transaction logs to an offsite #S3 bucket. I need something for volumes and maybe Kubernetes resources. #Longhorn offers S3 backups for it's own volumes, but for other #CSI like local #OpenEBS, maybe #Velero? Thoughts?
-
Having recently experienced a rather horrible #Kubernetes crash, I'm looking for #backup solutions. We're good with PostgreSQL since we're using #CNPG with remote transaction logs to an offsite #S3 bucket. I need something for volumes and maybe Kubernetes resources. #Longhorn offers S3 backups for it's own volumes, but for other #CSI like local #OpenEBS, maybe #Velero? Thoughts?
-
Having recently experienced a rather horrible #Kubernetes crash, I'm looking for #backup solutions. We're good with PostgreSQL since we're using #CNPG with remote transaction logs to an offsite #S3 bucket. I need something for volumes and maybe Kubernetes resources. #Longhorn offers S3 backups for it's own volumes, but for other #CSI like local #OpenEBS, maybe #Velero? Thoughts?
-
Having recently experienced a rather horrible #Kubernetes crash, I'm looking for #backup solutions. We're good with PostgreSQL since we're using #CNPG with remote transaction logs to an offsite #S3 bucket. I need something for volumes and maybe Kubernetes resources. #Longhorn offers S3 backups for it's own volumes, but for other #CSI like local #OpenEBS, maybe #Velero? Thoughts?
-
I've been doing things I shouldn't with #Kubernetes. We're using a replicated #MinIO cluster as the storage backend on #mstdndk, which requires a boat load of storage, especially if you forget to specify any kind of retention. So far, the quick workaround for a full disk, was just to expand the filesystem. Since we're replicating across nodes, we're using #OpenEBS #LVM for local storage. Poor partitioning means we're running out of storage on the volume group, but even worse - PVCs sizes were increased before checking if we had space for it. Kubernetes is now stuck in a most unfortunate situation - it can't grow the local filesystem, as the volume group is full and you're not allowed to decrease the size request. What then? Cue https://github.com/etcd-io/auger - a tools that allows you to edit #K8s resources directly in #etcd. Obviously you should never do this, but with steady hands and clinical precision, you can get yourself out of a pickle like mine. Size was reverted and PVCs were unstuck.
-
I've been doing things I shouldn't with #Kubernetes. We're using a replicated #MinIO cluster as the storage backend on #mstdndk, which requires a boat load of storage, especially if you forget to specify any kind of retention. So far, the quick workaround for a full disk, was just to expand the filesystem. Since we're replicating across nodes, we're using #OpenEBS #LVM for local storage. Poor partitioning means we're running out of storage on the volume group, but even worse - PVCs sizes were increased before checking if we had space for it. Kubernetes is now stuck in a most unfortunate situation - it can't grow the local filesystem, as the volume group is full and you're not allowed to decrease the size request. What then? Cue https://github.com/etcd-io/auger - a tools that allows you to edit #K8s resources directly in #etcd. Obviously you should never do this, but with steady hands and clinical precision, you can get yourself out of a pickle like mine. Size was reverted and PVCs were unstuck.
-
I've been doing things I shouldn't with #Kubernetes. We're using a replicated #MinIO cluster as the storage backend on #mstdndk, which requires a boat load of storage, especially if you forget to specify any kind of retention. So far, the quick workaround for a full disk, was just to expand the filesystem. Since we're replicating across nodes, we're using #OpenEBS #LVM for local storage. Poor partitioning means we're running out of storage on the volume group, but even worse - PVCs sizes were increased before checking if we had space for it. Kubernetes is now stuck in a most unfortunate situation - it can't grow the local filesystem, as the volume group is full and you're not allowed to decrease the size request. What then? Cue https://github.com/etcd-io/auger - a tools that allows you to edit #K8s resources directly in #etcd. Obviously you should never do this, but with steady hands and clinical precision, you can get yourself out of a pickle like mine. Size was reverted and PVCs were unstuck.
-
I've been doing things I shouldn't with #Kubernetes. We're using a replicated #MinIO cluster as the storage backend on #mstdndk, which requires a boat load of storage, especially if you forget to specify any kind of retention. So far, the quick workaround for a full disk, was just to expand the filesystem. Since we're replicating across nodes, we're using #OpenEBS #LVM for local storage. Poor partitioning means we're running out of storage on the volume group, but even worse - PVCs sizes were increased before checking if we had space for it. Kubernetes is now stuck in a most unfortunate situation - it can't grow the local filesystem, as the volume group is full and you're not allowed to decrease the size request. What then? Cue https://github.com/etcd-io/auger - a tools that allows you to edit #K8s resources directly in #etcd. Obviously you should never do this, but with steady hands and clinical precision, you can get yourself out of a pickle like mine. Size was reverted and PVCs were unstuck.
-
I've been doing things I shouldn't with #Kubernetes. We're using a replicated #MinIO cluster as the storage backend on #mstdndk, which requires a boat load of storage, especially if you forget to specify any kind of retention. So far, the quick workaround for a full disk, was just to expand the filesystem. Since we're replicating across nodes, we're using #OpenEBS #LVM for local storage. Poor partitioning means we're running out of storage on the volume group, but even worse - PVCs sizes were increased before checking if we had space for it. Kubernetes is now stuck in a most unfortunate situation - it can't grow the local filesystem, as the volume group is full and you're not allowed to decrease the size request. What then? Cue https://github.com/etcd-io/auger - a tools that allows you to edit #K8s resources directly in #etcd. Obviously you should never do this, but with steady hands and clinical precision, you can get yourself out of a pickle like mine. Size was reverted and PVCs were unstuck.
-
A unfortunate side affect of Longhorn experiencing I/O latency and saturation, is that volumes attached to pods become remounted read-only. This has very very unfortunate side effects on running databases, caches etc. Any tips on making #Longhorn behave would be greatly appreciated. I've looked briefly into OpenEBS' distributed, replicated storage but the requirements are currently not allowing it - specifically, replicated #OpenEBS needs an entire physical disk for itself.
-
A unfortunate side affect of Longhorn experiencing I/O latency and saturation, is that volumes attached to pods become remounted read-only. This has very very unfortunate side effects on running databases, caches etc. Any tips on making #Longhorn behave would be greatly appreciated. I've looked briefly into OpenEBS' distributed, replicated storage but the requirements are currently not allowing it - specifically, replicated #OpenEBS needs an entire physical disk for itself.
-
A unfortunate side affect of Longhorn experiencing I/O latency and saturation, is that volumes attached to pods become remounted read-only. This has very very unfortunate side effects on running databases, caches etc. Any tips on making #Longhorn behave would be greatly appreciated. I've looked briefly into OpenEBS' distributed, replicated storage but the requirements are currently not allowing it - specifically, replicated #OpenEBS needs an entire physical disk for itself.
-
A unfortunate side affect of Longhorn experiencing I/O latency and saturation, is that volumes attached to pods become remounted read-only. This has very very unfortunate side effects on running databases, caches etc. Any tips on making #Longhorn behave would be greatly appreciated. I've looked briefly into OpenEBS' distributed, replicated storage but the requirements are currently not allowing it - specifically, replicated #OpenEBS needs an entire physical disk for itself.
-
A unfortunate side affect of Longhorn experiencing I/O latency and saturation, is that volumes attached to pods become remounted read-only. This has very very unfortunate side effects on running databases, caches etc. Any tips on making #Longhorn behave would be greatly appreciated. I've looked briefly into OpenEBS' distributed, replicated storage but the requirements are currently not allowing it - specifically, replicated #OpenEBS needs an entire physical disk for itself.
-
#kubernetes question: our deployment stack is #skaffold plus #helm. We have #OpenEBS deployed, I'm trying to add their monitoring mixin to deployment. But they want to deploy whole kube-prometheus, which we already have. I have a script to generate their manifests and grab just those 4 files we're interested in, and have helm grab those. But here comes the question: how do I tie in the generation with the rest of the deployment?
[1/2] -
#kubernetes question: our deployment stack is #skaffold plus #helm. We have #OpenEBS deployed, I'm trying to add their monitoring mixin to deployment. But they want to deploy whole kube-prometheus, which we already have. I have a script to generate their manifests and grab just those 4 files we're interested in, and have helm grab those. But here comes the question: how do I tie in the generation with the rest of the deployment?
[1/2] -
#kubernetes question: our deployment stack is #skaffold plus #helm. We have #OpenEBS deployed, I'm trying to add their monitoring mixin to deployment. But they want to deploy whole kube-prometheus, which we already have. I have a script to generate their manifests and grab just those 4 files we're interested in, and have helm grab those. But here comes the question: how do I tie in the generation with the rest of the deployment?
[1/2] -
#kubernetes question: our deployment stack is #skaffold plus #helm. We have #OpenEBS deployed, I'm trying to add their monitoring mixin to deployment. But they want to deploy whole kube-prometheus, which we already have. I have a script to generate their manifests and grab just those 4 files we're interested in, and have helm grab those. But here comes the question: how do I tie in the generation with the rest of the deployment?
[1/2] -
#kubernetes question: our deployment stack is #skaffold plus #helm. We have #OpenEBS deployed, I'm trying to add their monitoring mixin to deployment. But they want to deploy whole kube-prometheus, which we already have. I have a script to generate their manifests and grab just those 4 files we're interested in, and have helm grab those. But here comes the question: how do I tie in the generation with the rest of the deployment?
[1/2] -
man, my #openebs jiva experience has been nightmare. it requires manual intervention like every day on my cluster, and the community is just a sea of ignored issues and slack messages. i don't know why the talos team recommends it.
am i the only one who feels this way?
-
man, my #openebs jiva experience has been nightmare. it requires manual intervention like every day on my cluster, and the community is just a sea of ignored issues and slack messages. i don't know why the talos team recommends it.
am i the only one who feels this way?
-
man, my #openebs jiva experience has been nightmare. it requires manual intervention like every day on my cluster, and the community is just a sea of ignored issues and slack messages. i don't know why the talos team recommends it.
am i the only one who feels this way?
