#mergerfs — Public Fediverse posts

Anyone here used #mergerfs ?
https://github.com/trapexit/mergerfs
Sounds interesting

My experience with #OMV #NAS with #RaspberryPi, and using #SnapRAID + #MergerFS, is that while SnapRAID's really cool, it def could not beat real #RAID. My understanding is RAID works on a low/block level so it can achieve redundancy 'in real time' no matter what files you have, but since SnapRAID is in software, it's not only more involved but it also simply could not achieve full redundancy as you will def need to exclude a lot of files (which could render ur redundancy useless, depending on ur use case) and it's also more involved where you might need to e.g. stop a bunch of #Docker containers prior to it doing its thing.

I'm just not sure how to achieve 'real' RAID on an RPi 4 (of varying memory capacities). One benefit of SnapRAID (and MergerFS) is that you could easily implement it with USB-attached disks, which is the 'friendliest' choice for SBCs like the Pi which has no SATA connectivity built-in.

I'm setting up my fileserver again. Before I used MergerFS and 4x4tb NAS drivers, to aggregate the files on all the drives as one mountpoint. The nice thing about mergerfs is that the disks are completely separate (files are written whole to each disk - nice !). The data is not important at all to me and backed up so I don't need RAID etc. Any suggestions of an alternative approach? I hear ZFS pools are interesting, but never used zfs. #linux #mergerfs #zfs #datahoarding

Because I had a conveersation about #backup yesterday at the pool.

trapexit, author of mergerfs, has a collection of IMHO , good and new user friendly howtos over here.

https://github.com/trapexit/backup-and-recovery-howtos

#linix #btrfs #zfs #snapraid #mergerfs #backup #rsync

#Mergerfs is very cool.

#SelfHosted

I think I might RMA that one SATA SSD - that SSD is currently setup as 1 of 2 #SnapRAID data drives (there's another identical parity drive), that is then part of a #MergerFS pool.

There are a ton of video guides on how to replace a disk part of a #RAID array for failure/upgrade reasons on #TrueNAS, but not found one for this SnapRAID-MergerFS setup on #OpenMediaVault.

From what I can tell, there doesn't seem to be a graphical option on #OMV to do this easily/intuitively. Might need to look up some written guides/docs to do this safely, so I can send the SSD back, and recover my data on the replacement drive once it arrived.

---

Hopefully, this written guide should suffice:

🔗 https://wiki.omv-extras.org/doku.php?id=omv7:omv7_plugins:snapraid#recovery_operations

RE: https://sakurajima.social/notes/a8mrrt78bi

I've managed to get #OpenMediaVault working on my #RaspberryPi (running #Raspbian Lite) and the performance seems pretty impressive! Despite relying on USB storage for the SSDs.

This is my first time running a #NAS on the Pi, on #OMV, not using #ZFS or #RAID but rather an #Unraid like solution, 'cept, #FOSS called #SnapRAID in combination with #mergerfs (the drives themselves are simply #EXT4).

So far, honestly, so good. I got 2x 1TB SSDs for data, and another 1TB SSD for parity. Don't have a backup for the data themselves atm, but I do have a scheduled backup solution (#RaspiBackup) setup for the OS itself (SD card). It's also got #Timeshift for creating daily snapshots.

I'm not out of the woods yet though, cos after this comes the (somewhat) scary part - deploying #Immich on the Pi lol (using OMV's #Docker compose interface perhaps). I really could just deploy it in my #Proxmox #homelab, and I wouldn't have to worry about system resources or hardware transcoding, etc. but I really wanna experiment this 'everything hosted/contained in 1 Pi' concept.

Installing #OpenMediaVault on the #RaspberryPi so far... has been kind of a headache, despite how it always seemed 'simple' on other people's videos... Kinda wondering if I'm better off, just, setting up #SnapRAID, #mergerfs, and #Samba/#CIFS manually.

I don't think I need #OMV specifically, using the Pi as a simple #SMB share that I also happen to run some #Docker services on, I just thought it'd be neat to finally try out OMV, after having only used and been familiar with #TrueNAS all this while.

--

Well, while it was busy installing (which I initiated remotely, through SSH), for some fuckin' reason I got disconnected to it and I couldn't reconnect to it through SSH again. Why does this only happen when I'm a million miles away (exaggerating) and have no physical access to it but when I'm literally in the same building, doing the same type of thing thousands of times, everything just.. goes swell to the point that it deluded me into thinking #KVM 's a waste of money :')

Part 1 of a massive new video series I've been working really hard on behind the scenes for the last few weeks.

100% open source. 100% free.

Join me on the journey to build the Perfect Media Server.

https://youtu.be/Yt67zz9p0FU

#selfhosting #mediaserver #homeinfra #mergerfs #snapraid #docker

Switched my big 16 TB harddrive which served as a backup with the "smaller" 8 TB one which turned into the backup hdd.

Planning to just get another 8 TB one and using #mergerFS to expand the backup storage.

Anyone with a few experiences regarding that setup?

#selfhosting #linux #backup

True true! Perfect for the „poor man's Datacentre“ ;) I built a storage pool of my existing USB-Drives with various content, combined with some internal Drives, just by adding one or two Drives for RAID-Parity and writing a few lines of config and merged the content via #MergerFS (https://github.com/trapexit/mergerfs) to one big Storage. Boom, Cloud :awesome: :hacker:
@chpietsch

Well, added a storage tier this evening.

A couple 4 Tb drives which I had laying about, as #JBOD (#MergerFS), for those media files which are "replaceable" and don't really need to be on #ZFS mirrors (~70 & ~16 Tb).

I now feed into #Plex from the different tiers using #OverlayFS.

Pretty neat. I can move things around without Plex noticing, select the tier in #Sonarr / #Radarr, and if one of those 4 Tb drives goes down, NBD and I can see what is missing.

More iops don't hurt either.

Well, added a storage tier this evening.

A couple 4 Tb drives which I had laying about, as #JBOD (#MergerFS), for those media files which are "replaceable" and don't really need to be on #ZFS mirrors (~70 & ~16 Tb).

I now feed into #Plex from the different tiers using #OverlayFS.

Pretty neat. I can move things around without Plex noticing, select the tier in #Sonarr / #Radarr, and if one of those 4 Tb drives goes down, NBD and I can see what is missing.

More iops don't hurt either.

Well, added a storage tier this evening.

A couple 4 Tb drives which I had laying about, as #JBOD (#MergerFS), for those media files which are "replaceable" and don't really need to be on #ZFS mirrors (~70 & ~16 Tb).

I now feed into #Plex from the different tiers using #OverlayFS.

Pretty neat. I can move things around without Plex noticing, select the tier in #Sonarr / #Radarr, and if one of those 4 Tb drives goes down, NBD and I can see what is missing.

More iops don't hurt either.

Well, added a storage tier this evening.

A couple 4 Tb drives which I had laying about, as #JBOD (#MergerFS), for those media files which are "replaceable" and don't really need to be on #ZFS mirrors (~70 & ~16 Tb).

I now feed into #Plex from the different tiers using #OverlayFS.

Pretty neat. I can move things around without Plex noticing, select the tier in #Sonarr / #Radarr, and if one of those 4 Tb drives goes down, NBD and I can see what is missing.

More iops don't hurt either.

Well, added a storage tier this evening.

A couple 4 Tb drives which I had laying about, as #JBOD (#MergerFS), for those media files which are "replaceable" and don't really need to be on #ZFS mirrors (~70 & ~16 Tb).

I now feed into #Plex from the different tiers using #OverlayFS.

Pretty neat. I can move things around without Plex noticing, select the tier in #Sonarr / #Radarr, and if one of those 4 Tb drives goes down, NBD and I can see what is missing.

More iops don't hurt either.

Ich hab in meinem Desktop ja neben der SSD eine HDD für große Daten (Fotos, Videoschnitt) oder "das Datenarchiv" mit alten, selten genutzten Dingen. Bislang war es immer etwas fummelig zu verwalten, was wo abgelegt ist.
mergerfs https://github.com/trapexit/mergerfs sieht nach einem vielversprechenden Ansatz aus, eine gemeinsame Sicht in einem Verzeichnisbaum zu bekommen!
#linux #mergerfs

Doh, I forgot I needed to enable an extra setting for software that uses mmap when used on #mergerfs. https://github.com/trapexit/mergerfs#rtorrent-fails-with-enodev-no-such-device

@kirsten @wuffel RPi4 mit 4GB. Daran habe ich nen Fantec 4-bay Gehäuse mit den Platten. ~30TB Gesamt. Das ganze per USB3 angeschlossen. Leistung ist halt bei ca. 30MB/s im Netzwerk. Da ist die Miese Verbindung von USB und Raspi Schuld. AlsoOS nutze ich OMV. Die Platten sind gepoolt mit #mergerfs. Funktioniert absolut brauchbar. Mache meine ganzen Backups drüber. Und als Schmankerl läuft auch Docker mit paar Containern drauf. Hauptsächlich #selfhost Zeug.

After looking more into #TrueNAS Scale, I decided to just stick with #Snapraid + #MergerFS for my Plex Media. I like having the ability to mix and match drives and easily expand my storage.

I have never understood why RAID is so much more popular than SnapRaid.

Has anyone around the #Fediverse used a NAS with #MergerFS and #SnapRAID, rather than conventional RAID setups? I want to hear about its stability and usability in a real-world scenario.

The plan here is a file storage server with up to 14 disks of 16TB each. Other great ideas for this setup are welcome too, I have about 1 week left to orient on how I will configure it.

So I bought #Unraid and only then realized the booting-from-a-USB-stick thing isn't just for ease-of-use and optional, but actually an anti-piracy scheme.

I'm just not going to let my homemade NAS system be dependent on a flimsy USB stick. Guess I'll have to give #mergerfs + #snapraid another go, with #openMediaVault this time.

#linux #homeServer #nas #omv

https://www.ndr.de/kultur/film/Animationsfilm-aus-Hamburg-Maurice-der-Kater-beim-US-Festival-Sundance,mauricederkater102.html

Done. Now its time to send this #zfs mount to darkstar02.

Using #linux as desktop-workstation for #animation and #film has gotten harder the last years. All the awesome tools like #mergerfs , nfscached, zfs and kickstart are outweighed by gnomes lack of documentation, sneaking abandon of old linux/unix virtues like one tool for one job (systemd, NetworkManager), multi user. But still #linux makes things just possible.

Did you see Linux Update this week? Read it here and subscribe free today! https://mailchi.mp/linux-magazine.com/linux-update-cloning-a-debian-system-with-apt-clone #Debian #aptclone #Nitrux #GNOME #scripts #OpenBSD #MergerFS #RaspberryPi #jobs #events #FOSSback #GBFest2023 #OpenSource #Linux #FOSS

Do you know that MergerFS is integrated in CasaOS since version 0.3.8?

https://blog.casaos.io/blog/13.html

#mergerfs #nas #docker #raid

Ah, I was wondering why #MergerFS seemed like a permission nightmare for me, from digging I realized only mmap was failing, and it turns out enabling direct_io or cache.files=off causes mmap to fail on MergerFS. TIL!

https://www.reddit.com/r/jellyfin/comments/eoo18p/comment/feeqj8v/?utm_source=share&utm_medium=web2x&context=3

CasaOS has just released its v0.3.7, which includes a lot of fixes in storage management and beta support of MergerFS (feature toggled)

https://github.com/IceWhaleTech/CasaOS

#foss #nas #mergerfs #casaos #selfhost

@tobomir I run two nearly similar systems just because I got them for free apart from the HDDs. They are Fujitsu Esprimo P5731 & P5730 PCs. I run an additional old self-built PC with an EE AMD Athlon. None of the systems runs close to 24/7. All run #OpenMediaVault & use #mergerfs to pool the drives.

Soooo ... if you have #mergerfs (which is really nice btw) and try to copy to it with rsync you might find that you have not enough space even tho there is still plenty.
Issue is that rsync creates all folders first (that can be created on almost full drive) and try to copy to them. In case no more space is in the folder on phys drive your copy will fail.
My solution was to create all the folders with rsync on proper drive and then switch to pool->pool rsync