#cephfs — Public Fediverse posts

Latest experience with Ceph: Never enable `rbytes` on the CephFS client mount unless absolutely necessary.
This places an extremely heavy metadata load on the MDS and the OSDs.
So we’re not talking about a 20–30% increase, but rather, depending on the size and depth of the filesystem, an increase of several dozen to hundreds of times.

#ceph #cephfs #osd #mds

Latest experience with Ceph: Never enable `rbytes` on the CephFS client mount unless absolutely necessary.
This places an extremely heavy metadata load on the MDS and the OSDs.
So we’re not talking about a 20–30% increase, but rather, depending on the size and depth of the filesystem, an increase of several dozen to hundreds of times.

#ceph #cephfs #osd #mds

Latest experience with Ceph: Never enable `rbytes` on the CephFS client mount unless absolutely necessary.
This places an extremely heavy metadata load on the MDS and the OSDs.
So we’re not talking about a 20–30% increase, but rather, depending on the size and depth of the filesystem, an increase of several dozen to hundreds of times.

#ceph #cephfs #osd #mds

Latest experience with Ceph: Never enable `rbytes` on the CephFS client mount unless absolutely necessary.
This places an extremely heavy metadata load on the MDS and the OSDs.
So we’re not talking about a 20–30% increase, but rather, depending on the size and depth of the filesystem, an increase of several dozen to hundreds of times.

#ceph #cephfs #osd #mds

Latest experience with Ceph: Never enable `rbytes` on the CephFS client mount unless absolutely necessary.
This places an extremely heavy metadata load on the MDS and the OSDs.
So we’re not talking about a 20–30% increase, but rather, depending on the size and depth of the filesystem, an increase of several dozen to hundreds of times.

#ceph #cephfs #osd #mds

The ultimate #NAS-setup is a single-node #Ceph, is that correct?
#ZFS can't rebalance data (unless using dangerous rewriting)
#BTRFS shouldn't RAID5.
#LVM can't rebalance data.

#CephFS is able to mix and match any number of striping, mirroring and erasure coding on the same set of hard drives.

Windows Storage Spaces is LVM in better, but CephFS makes all of this available on Linux. Have I missed any FS that is as flexible as either, Storage Spaces or CephFS?

#Linux #HomeLab #SelfHost

Wheee, after some database upgrades and a whole pile of other things, lots of things updated today.

Big find of the day: kernel-6.17.11-200.fc42.x86_64 has a kernel: BUG: kernel NULL pointer dereference, address with #CephFS #Ceph, backing down to kernel-6.16.7-200.fc42.x86_64 got me back up on that machine, but that was weird seeing ls get killed trying to look what was in a directory. Caused a whole MESS of debug issues with what was trying to use that too.

Fun times.

Second #server rack installed in the #homelab. Most important takeway is that there is pretty RGB in it. For the nerds, I've decommissioned a lot of old 11th and 12th gen Dell Servers. Currently running a SuperMicro H11 Epyc Server in a CSE-826 chassis, a SuperMicro 4028GR-TRT2, and still a Dell R720 (last to be replaced) in a #Proxmox HA cluster. Currently moving all my media storage from #ZFS to #CephFS with 2+1 Erasure Coding and using replicaset 3 for more important data. Eventually would like to run five nodes specifically for #Ceph so I can run 3+2 EC to be more comfortable but with mostly read-only media storage, some downtime when I need to reboot a node doesn't bother me too much

RE: https://transfem.social/notes/a1ceiuaamq9700l3

#Proxmox VE - C'est une question qui revient souvent 😉
Non, je n'ai pas (encore) de stockage distribué #CephFS, et ce, peu importe le cluster❗
En attendant, je m'en passe bien 👍
↪️ https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster

#Proxmox VE - C'est une question qui revient souvent 😉
Non, je n'ai pas (encore) de stockage distribué #CephFS, et ce, peu importe le cluster❗
En attendant, je m'en passe bien 👍
↪️ https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster

Now that my #ceph cluster is up and running, I am trying to create a new user for mounting #cephfs on Ubuntu. I have the mount working with the admin user but am confused setting up a new user...

Do I need to create a keyring file and a secret file?

Does anyone have a blog post that might be useful?

#homelab

Дата-центр ЦЕРН на 1 эксабайт: как хранят данные

Большая наука невозможна без больших вычислений. По крайней мере, это утверждение справедливо в ядерной физике. Мы видим, что львиная доля самых мощных суперкомпьютеров установлена именно в научных учреждениях, в том числе университетах. Сразу несколько отраслей современной науки напрямую зависят от компьютерных расчётов и анализа больших данных, собранных по результатам наблюдений. Например, в Европейском центре ядерных исследований (ЦЕРН) работает один из крупнейших дата-центров в мире. Без этого вычислительного кластера мы бы искали бозон Хиггса ещё очень долго, а Стандартная модель так бы и осталась незавершённой.

https://habr.com/ru/companies/ruvds/articles/822681/

#ruvds_статьи #поиск_лекарств #CERN #ЦЕРН #БАК #Большой_взрыв #зептосекунда #JBOD #Ultrastar_Data102 #CMS #ALICE #ATLAS #LHCb #бозон_Хиггса #European_Grid_Infrastrucrure #OpenStack #CephFS #CASTOR #XRootD #Swift #RADOS_block_devices #RBD #сверхтекучий_гелий

Дата-центр ЦЕРН на 1 эксабайт: как хранят данные

Большая наука невозможна без больших вычислений. По крайней мере, это утверждение справедливо в ядерной физике. Мы видим, что львиная доля самых мощных суперкомпьютеров установлена именно в научных учреждениях, в том числе университетах. Сразу несколько отраслей современной науки напрямую зависят от компьютерных расчётов и анализа больших данных, собранных по результатам наблюдений. Например, в Европейском центре ядерных исследований (ЦЕРН) работает один из крупнейших дата-центров в мире. Без этого вычислительного кластера мы бы искали бозон Хиггса ещё очень долго, а Стандартная модель так бы и осталась незавершённой.

https://habr.com/ru/companies/ruvds/articles/822681/

#ruvds_статьи #поиск_лекарств #CERN #ЦЕРН #БАК #Большой_взрыв #зептосекунда #JBOD #Ultrastar_Data102 #CMS #ALICE #ATLAS #LHCb #бозон_Хиггса #European_Grid_Infrastrucrure #OpenStack #CephFS #CASTOR #XRootD #Swift #RADOS_block_devices #RBD #сверхтекучий_гелий

Дата-центр ЦЕРН на 1 эксабайт: как хранят данные

Большая наука невозможна без больших вычислений. По крайней мере, это утверждение справедливо в ядерной физике. Мы видим, что львиная доля самых мощных суперкомпьютеров установлена именно в научных учреждениях, в том числе университетах. Сразу несколько отраслей современной науки напрямую зависят от компьютерных расчётов и анализа больших данных, собранных по результатам наблюдений. Например, в Европейском центре ядерных исследований (ЦЕРН) работает один из крупнейших дата-центров в мире. Без этого вычислительного кластера мы бы искали бозон Хиггса ещё очень долго, а Стандартная модель так бы и осталась незавершённой.

https://habr.com/ru/companies/ruvds/articles/822681/

#ruvds_статьи #поиск_лекарств #CERN #ЦЕРН #БАК #Большой_взрыв #зептосекунда #JBOD #Ultrastar_Data102 #CMS #ALICE #ATLAS #LHCb #бозон_Хиггса #European_Grid_Infrastrucrure #OpenStack #CephFS #CASTOR #XRootD #Swift #RADOS_block_devices #RBD #сверхтекучий_гелий

Do I know anyone who's successfully gotten cephfs exported from a proxmox cephfs cluster into a VM?

I can get it mounted, all good. It can see contents there, yay! When it goes to write it gets a "Operation not permitted" error but creates an empty file with the right name.

What weird permissions bit do I have not set right?

#Ceph #CephFS #Proxmox

Fun and games trying to get CephFS working today… decided I'd use that to keep a copy of some of my archives on the cluster. (Might as well use those 10 2TB SSDs for something.)

Well, turns out the stock instructions for setting up a user for CephFS do not work. You will be allowed to mount a filesystem, you'll be able to create directories, and *empty* files, but you'll not be able to put any *data* in those files.

My file system (unimaginatively named `cephfs`), uses pools named `data` and `metadata` for storage.

This wound up being the user permissions that worked:

```
caps mds = "allow rw"
caps mon = "allow r"
caps osd = "allow rw tag cephfs data=metadata, allow rw pool=data"
```

#Ceph #CephFS

"But #cephfs is slow.." #ceph

Anyone here familiar with #cephfs recovery? I ran out of space because it isn't reclaiming free space. Any ideas what I need to do?

Adding space actually reduced the available space causing further issues.

#ceph #rook #kubernetee

Reboosts appreciated

TIL: "CephFS has a configurable maximum file size, and it's 1TB by default."

Guess how I found out ...
But where do I store my 9TB #PostgreSQL database copy (not in production, temporary) now?

#Ceph #CephFS

Hmmm, die Mastodon Datenbank finde ich nun auch nicht gerade klein mit 59 GB.
Die fast 400 GB Medien stoeren mich nun mit dem #CephFS zum Glück nicht mehr. Insbesondere nicht, wenn ich die VM dann mal auf einen anderen Host verschieben muss.

This is early work - I'll need to work through the same logic using cloud, and once we have that, we need to optimize. It's just exciting after an arduous weekend of failure with #rook and #cephfs. It's OK if things are hard sometimes, but that kind of complexity is a red flag. 🟥

irgendwie verstehe ich #CephFS noch nicht so recht....
- Für ein CephFS brauch ich OSDs
- außerdem braucht ein CephFS auch eigene Metadata und Data Pools
- und es braucht offenbar pro CephFS einen eigenen MDS

Wenn ich nun auf einem 3-node Ceph Cluster mehrere getrennte Filesysteme haben will, kann ich maximal 3 haben, weil MDS hier beschränkt.
Oder ist die Idee dann, dass man mit Volumes arbeitet?

Also irgendwie komme ich mit #CephFS noch nicht so ganz klar:

Ich hab einen Pool, ein CephFS, ein Subvolume und die Anleitung:

https://docs.ceph.com/en/quincy/cephfs/client-auth/

Aber wo bekomme ich die client.<id> her? Hmmm...

A #Proxmox question again:

How can I change the default mount path for #CephFS? I don't want it to be under /mnt/pve/<STORAGE_ID>/ as described in https://pve.proxmox.com/wiki/Storage:_CephFS, but maybe on /srv/pve/<STORAGE_ID> instead.

Editing the path after creation in storage.conf feels wrong to me...

Hmmm, I think installing this helm chart of the #CephFS CSIDriver on my #Kubernetes cluster has ruined it: https://artifacthub.io/packages/helm/ceph-csi/ceph-csi-cephfs

Every node is pegged at 100% CPU usage, only able to respond to a small fraction of pings, and not responding to `kubectl` at all

There's a warning about this at https://www.talos.dev/docs/v0.14/guides/storage/#storage-clusters :
> Also, if your cluster is small, just running Ceph may eat up a significant amount of the resources you have available.

I just wasn't expecting 100% of my resources to be consumed, and just by the CSIDriver

This isn't even the actual CephFS cluster that I'd need to be able to store files, this is just the driver to connect to such a cluster, and it is all-consuming!

I'm going to leave this for a few hours and see if this converges on a working state by itself, otherwise I'm going to have to wipe and start with a fresh cluster :S

So kids, don't do CephFS