#upsscaler — Public Fediverse posts

Blog post now live

Creating a kubernetes autoscaling operator that responds to UPS events

https://www.beyondwatts.com/posts/creating-a-kubernetes-autoscaling-operator-that-responds-to-ups-events/

This has been a really interesting project, thanks for all the feedback!

#upsScaler #homelab #kubernetes

Some final thoughts on this thread now I have recovered from the shock of it actually working:

NUT reports an UPS run time of 24 mins. It really means 6 minutes, even though the load does not increase dramatically when scaling down.

The low battery warning /shutdown message from the UPS occurs too late so now we scale down as soon as the power goes.

The time to scale down has been reduced from 3min30sec to about a minute by doing more in parallel.

#upsScaler #homelab

Successful test today 😃

I have some tweaking to do with the scale down / up event timings but everything worked as planned. Probably time for a blog post…

#upsScaler #homelab

2. Scaling back up is configured to wait until the battery has recharged and reaches 15mins runtime. The UPS takes about 3 hours to recharge to this level.
Thinking about options for this one while I wait for the UPS to recharge.

We could scale back up as soon as the UPS comes back online or provide an override endpoint to trigger scaling back up <- this is how my nut-exporter mock works for testing

#upsScaler #homelab

Today's learnings...

1. The UPS is connected to a Synology NAS which exposes the status to the network with nut-exporter. When the UPS issues a low battery, the NAS goes into suspend mode before the operator can collect the status.
Improved approach: If nut-exporter as gone away and the last status was 'on battery' scale down the cluster

#upsScaler #homelab

Today should be the day the UPS gets unplugged as a final test. But there's a couple more tests to run first:

- full scale down without rook
- full scale down including rook
- switch off the UPS 😱

I've made assumptions about the states the UPS should present when the power goes out (ie On Battery, Shutdown or Forced Shutdown) and the code should be able to handle various scenarios but it will be interesting to see what actually happens..... 🤞

#upsScaler #homelab

Ok thanks to the suggestion from @nicr9 I have completely refactored the code to use a ups-scaler Custom Resource Definition instead of labels and annotations.

Most of the work was in refactoring the tests as in addition to new API calls, the order of the previous calls all changed...

Unbelievably, I just deployed to the cluster and successfully completed a scale down / scale up for five test workloads at the first attempt. Beer o'clock now 🍻

#upsScaler #homelab

Nope. Annotations disappear too.

All it takes is an operator restart 😢

Guess I’ll have to see if the labels can be added from the helm chart

#homelab #upsScaler

Fortunately there looks to be an easy option - use annotations instead of labels. I’m not using the label selector functionality so it doesn’t really make a difference.

But first I need to spin up a test cluster and recreate the issue. If it is the operator, a simple restart would trigger removing the labels…….

#homelab #upsScaler

One observation from today’s test that I need to figure out:

The rook operator removed custom labels from the ceph-exporter and csi-provisioner deployments when it was restarted. The annotations were untouched. Need to work out is this is by design or not…..

Would it matter if these #rook #ceph deployments are not scaled down?

#homelab #upsScaler

Success!!! Cluster scaled down including rook and then back up again 😎

I’m still amazed that the release yesterday worked first time. Wouldn’t have been possible without #nock. My end-to-end scale down/up tests have nearly 100 kubernetes API end point calls mocked up for testing

#homelab #upsScaler

OK it took a little work (not too much actually) but the scaling down order and the scaling up order can now be specified separately with different labels...

Successfully tested on the cluster with real deployments so now all I need to do is label up the rook deployments and test again.........

Fingers crossed.... (PS I might pause for 24 hours at this point and ponder whether it's really ready for this step)

#homelab #upsScaler

Rook-ceph successfully scaled to zero and back up… something else to implement though - ceph needs the Managers to be scaled down after the Monitors but the Monitors to be scaled back up before the Managers…

The current implementation assumed the up order was a reverse of the down order… need to think about this additional use case… 🤔

#homelab #upsScaler

Added functionality to suspend CronJobs. Took a bit longer than expected as they needed a different pattern to the other services but all working now.

Next stop: add rook-ceph into the scale down/up… but first I need to read up on the correct order to stop (and then start) the services

#homelab #upsScaler

One question still outstanding what to do with the nodes… if I shut them down they will turn back on when the power is restored but what if the power is returned before the UPS switches off… there is a use case where the nodes won’t turn back on… hmmm 🤔

#homelab #upsScaler #kubernetes

Successfully tested scaling down and back up workloads on the real cluster. Scaling down took about 3 minutes and back up about 20 - mainly waiting for databases to replicate. So far I’ve left rook-ceph running so the next round of testing will scale this down too…

Also migrated the UPS trigger away from Prometheus to Nut Exporter directly because , of course, once Prometheus has been scaled down, it cannot trigger a scale back up event!

#homelab #upsScaler #kubernetes

scaling back up now works too. I forgot about suspending CronJobs but that should be an easy add.

Now all I need to do is label some workloads - deployments , statefulsets, clusters etc… although I did get somewhat distracted tonight wondering if I can infer this from the Kubernetes API details. Ended up writing some code to draw network graphs with cytoscape and mermaid. Oops!

#homelab #upsScaler

Rather unbelievably running in the real cluster with some test workloads worked first time and everything scaled down as it should when the mock UPS triggered

Best write the scaleUp function now to recover these services

#homelab #upsScaler

Added down scaling for statefulsets into the mix. Found the CustomObject API endpoints when implementing CloudPG clusters and refactored the deployment/statefulset functions to call these. Simplifies everything somewhat.

Contemplating letting the #upsScaler loose in the real cluster on a test namespace… wonder what surprises that will uncover…

#homelab

Decided I should write some tests…

Now Prometheus and the Kubernetes API are mocked, I can simulate scaling workloads up and down without actually doing so 😃

#homelab #upsScaler

Not a bad first start... The idea is...

Set a down-priority using a label
Capture the original specReplicas with an annotation

When an UPS event is triggered, start scaling down in priority order.

Have a basic POC working for deployments.

Next stop CloudPG clusters which just need to be annotated to kick off an automatic hibernation!

#homelab #upsScaler