These daily backups from Proxmox's backup server may have saved my data.
When the mysql container is running, data is written to the filesystem. If I set this up properly, this would be on a persistent volume, so that it persists across restarts/shutdowns/etc.
In my case, because computers will do what you tell them to, the mysql data ends up being written locally to the k8s worker node. This lives in the dir /var/lib/containerd/io.containerd.grpc.v1.cri/containers/<random-id>/volumes/<lvm-label>/var/lib/mysql/.
Before I found that path, I figured the data had to live somewhere, and Linux can get pretty creative with mount paths, so I went looking. Based on binary text searches with xxd, and finding references to lomo soltado, I've found it. Other recipes also seem to be in there that I know I had stored.
Recovering the data from the files will be an exercise for tomorrow. But I feel a lot better about it now.
#homelab #pebkac #kubernetes #oops #selfhosting #bookstack #testyourbackups