home.social

#velero — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #velero, aggregated by home.social.

  1. What ended up working is using #Longhorn itself to backup my #Kubernetes volumes to my #S3 storage (#SeaweedFS), which works regardless of the volumes' access modes. This is fine, tho I still wish I could get #Velero working bcos in theory - it shd be better since it'd work with all storage rather than Longhorn alone... but yea honestly have no idea how to get it working with RWO volumes.

    RE: https://sakurajima.social/notes/alt7i6brp4

  2. What ended up working is using #Longhorn itself to backup my #Kubernetes volumes to my #S3 storage (#SeaweedFS), which works regardless of the volumes' access modes. This is fine, tho I still wish I could get #Velero working bcos in theory - it shd be better since it'd work with all storage rather than Longhorn alone... but yea honestly have no idea how to get it working with RWO volumes.

    RE: https://sakurajima.social/notes/alt7i6brp4

  3. What ended up working is using #Longhorn itself to backup my #Kubernetes volumes to my #S3 storage (#SeaweedFS), which works regardless of the volumes' access modes. This is fine, tho I still wish I could get #Velero working bcos in theory - it shd be better since it'd work with all storage rather than Longhorn alone... but yea honestly have no idea how to get it working with RWO volumes.

    RE: https://sakurajima.social/notes/alt7i6brp4

  4. What ended up working is using #Longhorn itself to backup my #Kubernetes volumes to my #S3 storage (#SeaweedFS), which works regardless of the volumes' access modes. This is fine, tho I still wish I could get #Velero working bcos in theory - it shd be better since it'd work with all storage rather than Longhorn alone... but yea honestly have no idea how to get it working with RWO volumes.

    RE: https://sakurajima.social/notes/alt7i6brp4

  5. What ended up working is using #Longhorn itself to backup my #Kubernetes volumes to my #S3 storage (#SeaweedFS), which works regardless of the volumes' access modes. This is fine, tho I still wish I could get #Velero working bcos in theory - it shd be better since it'd work with all storage rather than Longhorn alone... but yea honestly have no idea how to get it working with RWO volumes.

    RE: https://sakurajima.social/notes/alt7i6brp4

  6. I've set up #velero and #seaweedfs (as the #s3 storage) and yet backups of my pvcs are failing :( it seems that only the RWO vols are failing? but as far as im aware, velero/#kopia shdnt care/differentiate if it's RWO or not? any clue? underlying storage is #longhorn

    #kubernetes #homelab

  7. I've set up #velero and #seaweedfs (as the #s3 storage) and yet backups of my pvcs are failing :( it seems that only the RWO vols are failing? but as far as im aware, velero/#kopia shdnt care/differentiate if it's RWO or not? any clue? underlying storage is #longhorn

    #kubernetes #homelab

  8. I've set up #velero and #seaweedfs (as the #s3 storage) and yet backups of my pvcs are failing :( it seems that only the RWO vols are failing? but as far as im aware, velero/#kopia shdnt care/differentiate if it's RWO or not? any clue? underlying storage is #longhorn

    #kubernetes #homelab

  9. I've set up #velero and #seaweedfs (as the #s3 storage) and yet backups of my pvcs are failing :( it seems that only the RWO vols are failing? but as far as im aware, velero/#kopia shdnt care/differentiate if it's RWO or not? any clue? underlying storage is #longhorn

    #kubernetes #homelab

  10. I've set up #velero and #seaweedfs (as the #s3 storage) and yet backups of my pvcs are failing :( it seems that only the RWO vols are failing? but as far as im aware, velero/#kopia shdnt care/differentiate if it's RWO or not? any clue? underlying storage is #longhorn

    #kubernetes #homelab

  11. Very excited to have gotten through a pretty big lift + refactor in the #homelab over the last few weeks. For a while, I’ve had one foot in my old, disorganized, scattered Docker services, and one foot in a new, fully automated, #gitops pseudo-prod for home.

    Well, no more. I’ve cleaned up all of the old shit. Centralized config, linted, backed up, refactored, and smoke tested the core platform. It completely spins up and down, with one Playbook, data fully backed up, etc.

    Hardware:
    - 3x #amd mini PC’s w/32GB RAM
    - #synology DS1813+
    - 1x Intel miniPC for #homeassistant OS
    - TP-Link #Omada gateway, PoE switch, and AP’s

    My core stack is, going from hardware to workload lifecycle and access:
    - #proxmox
    - #fedora #fedora_coreos VM’s
    - #k3s #Rancher #kubernetes
    - #nfs subdir provisioner
    - #ansible
    - #argocd
    - #Tailscale
    - #1password Kubernetes Operator
    - #velero

    I just did a DR drill with my kids’ #minecraft servers, and it worked great, and I’m super pumped with #selfhosting #paperlessngx!

    Those are the first family-facing apps that I’ve deployed, and I’ve got a laundry list of more services on the way.

    It was far, far more work than it probably needed to be…but I also can fully trust it with our most important personal docs, which was priority one.

  12. Having recently experienced a rather horrible #Kubernetes crash, I'm looking for #backup solutions. We're good with PostgreSQL since we're using #CNPG with remote transaction logs to an offsite #S3 bucket. I need something for volumes and maybe Kubernetes resources. #Longhorn offers S3 backups for it's own volumes, but for other #CSI like local #OpenEBS, maybe #Velero? Thoughts?

    velero.io/

  13. Homelab project idea: Standby disaster recovery/Continuity of Operations node

    The parent's fileserver (PFS) will be running Proxmox, and will have plenty of headroom. The idea is to run a simple single-node cluster in a VM (likely Talos) and configure Velero to do a restore of critical applications to this single-node cluster.

    I'm not sure how to handle storage classes. The DR node won't have Ceph, I'm thinking maybe configuring a local storage provisioner and making fake storage classes (i.e. a "cephfs" storage class that lies and points to a local disk).

    Cilium/CNI would be configured in a much more basic manner as well maybe I could also do the fake thing and map the "cilium" ingress to something else, or just configure it in L2 mode. All I would need to do is switch some DNS records and I'd have only lost changes since the last backup.

    The first victory would be a proof of concept but it might be possible to have a daily task that overwrites the DR cluster with the most recent successful backup.

    Since the home automation and media streaming/acquisition apps won't be set to run on the DR node, a so goes reasonably sized VM should be able to handle things with no problem.

    This would also put me at ease that if needed I could move my self-hosted platform to an event smaller significantly more portable/lower power physical machine, maybe even a single odroid H4 with 2-4 disks



    #Kubernetes #Homelab #Velero #Backups #Selfhosted

  14. I was remembered, that behind velero backup Broadcom is. Time to ditch velero before Broadcom does Broadcom things and test Volsync for backups.

    #kubernetes #backup #velero #volsync

  15. The Velero backups are working except for the CephFS volumes ​:neocat_sob:​

    I have no idea why it just throws timeout errors but I can create a manual snapshot which doesn't take that long?

    I'm so close to getting past this step but I'm not exactly where to look next
    #Kubernetes #Velero #Ceph #Backups

  16. I'm very disappointed in you #Velero.

    You can't restore your own snapshots USING DEFAULT SETTINGS.

    Like data is in the bucket, you restore the CRs, but not the volumes?

    #YouHadOneJob

  17. the #velero adventure continued. my config in my cluster was good enough for things to run buuuut
    - this is necessary to work with backblaze b2 github.com/vmware-tanzu/velero
    - velero node agent is a thing which i hadn't needed to enable before (or went by a different value in a previous version and i'd removed it.. i think thats what happened) github.com/vmware-tanzu/helm-c

    apparently my backups haven't been working for a very, very long time! 😅

  18. #velero tweaks done!!!!!

    split out config into the relevant CRDs like the error suggested, and tweaked the values so the included resources arent stepping on my identically named bespoke resources. THAT WASNT SO BAD

    github.com/vmware-tanzu/helm-c

    note to self: things work when you slow down, read, get more slower, read again...

  19. #homelab housekeeping:
    - updated all my #helm charts (except #velero)
    - migrated #postgresql from previous major version to current major version
    - everything still works
    - feeling good about doing a hobby ive neglected for far too long

    #todo #wordpress layout adjustments, get matrix #synapse working again and REALLY RESEARCH proper synapse database housekeeping tasks so things dont get out of control like they always do

    happy friday yall 🎉

  20. Coming back from vacation and there is the next #Azure disaster 😭

    Changing the storageclasses of our #kubernetes #pvcs to the new #PremiumV2 one means that #pvc #snapshots are taking 10 min now, instead of 3 sec as with the old storageclass, resulting in #velero #backups to fail (global timeout).

    This even happens with #Azures managed backups which also use #Velero in the background 🤦

    As always #azure support is no help too :(

    #wtf #fml #k8s

  21. And the Operation Manual also received new content. It now describes how to backup and restore the complete namespace of #Univention #Nubus for Kubernetes using #Velero as example.

    See docs.software-univention.de/nu

  22. ⚠️ Heads up #velero users!

    When you're relying on velero using #prometheus metric-based alerts make sure to not only alert on
    `velero_backup_failure_total` but also on
    `velero_backup_partial_failure_total` 🧐

    After running velero reliably for years on #gke our backups suddenly started failing *partially*.

    Turns out #GCP must have changed something in AuthN, requiring an additional role to perform disk snapshots.
    As this resulted in partial failures only we almost missed it.

  23. @jenbanim @tek VPS? Don't you guys use K8s? I like DOKR.
    I've seen #DigitalOcean proposes SnapShooter too "Under the hood SnapShooter uses the open source project #Velero to manage the backup process of your clusters."

  24. Lily Cohen :firefish: (@lily) | Firefish Social

    "Didn’t you have backups?
    Yes and also apparently no. We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim data. While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data."

    firefish.social/notes/9iqefgi8

  25. @Rhoseigh mainly home lab stuff and some self-written apps:

    #Adguardhome
    #Docker-registry
    #Gitea
    #HomeAssistant
    #Hugo
    #MariaDB Galera
    #Minio
    #Mosquitto mqtt
    #Portainer
    #Trafefik
    #vaultwarden
    #Velero Cluster Backup
    #Wireguard VPN

    working on a blog posts with learnings and tips for getting these up and running….

  26. Today's self-hosting adventure: recovering from a stupid configuration mistake that resulted in lost data. First, audited the rest of my configuration to make sure nothing else was duplicating the same mistake!

    Then restored from Velero, which was a massive headache. Only was successful because I finally got exasperated and pulled the data out of the restic backup manually.

    I really want to like Velero, but it *really* wants you to just do full namespace restores. You try and tailor what you want to restore, and you quickly get off into the weeds.

    #kubernetes #velero #selfhosting

  27. What does everyone use for #backups these days? At the moment I am using: 1) for my #Macs: #TimeMachine for local backups to two different drives; #Arq to two offsite destinations (#idrive e2 and #backblaze b2); 2) "normal" servers: #restic to both iDrive e2 and Backblaze B2; 3) #kubernetes: #velero and #kasten. What about you?

  28. What does everyone use for #backups these days? At the moment I am using: 1) for my #Macs: #TimeMachine for local backups to two different drives; #Arq to two offsite destinations (#idrive e2 and #backblaze b2); 2) "normal" servers: #restic to both iDrive e2 and Backblaze B2; 3) #kubernetes: #velero and #kasten. What about you?

  29. What does everyone use for #backups these days? At the moment I am using: 1) for my #Macs: #TimeMachine for local backups to two different drives; #Arq to two offsite destinations (#idrive e2 and #backblaze b2); 2) "normal" servers: #restic to both iDrive e2 and Backblaze B2; 3) #kubernetes: #velero and #kasten. What about you?

  30. What does everyone use for #backups these days? At the moment I am using: 1) for my #Macs: #TimeMachine for local backups to two different drives; #Arq to two offsite destinations (#idrive e2 and #backblaze b2); 2) "normal" servers: #restic to both iDrive e2 and Backblaze B2; 3) #kubernetes: #velero and #kasten. What about you?

  31. What does everyone use for #backups these days? At the moment I am using: 1) for my #Macs: #TimeMachine for local backups to two different drives; #Arq to two offsite destinations (#idrive e2 and #backblaze b2); 2) "normal" servers: #restic to both iDrive e2 and Backblaze B2; 3) #kubernetes: #velero and #kasten. What about you?

  32. @tero I remember running into so many issues when I was looking at #openebs as my storage provider. My biggest thing was tying it with a backup solution #velero since openebs was creating a bunch of services for its volumes. Ended up being a nightmare. But the biggest thing was since I had single node instance, any power failure was devastating to the storage. Not all of the volumes would come back up.

    Since then I switched to #longhorn from #rancher and have had 0 issues.

  33. I really need to get on with writing some #Terraform to spiin up #Redis and #Memcache in Google #Cloud for work. I also really need to fix the permissions on my #Velero backups for my Mastodon cluster. But what I'm more fixated on is why this bloody smart light can't speak to the wifi!