#hdf5 — Public Fediverse posts

@AtomAntWirr Bislang lesen die mit `delay(500);` (also „fast“ zweimal pro Sekunde) ihre Sensoren aus und schicken das per UDP an (ehemals Raspberry Pi, jetzt) den zentralen HomeServerLaptop 🙈.

Da wird dann empfangen, gepuffert, abgespeichert, archiviert (als #HDF5!) etc.

Die Boards haben ein kleines USB-Netzteil; Verbrauch nicht quantitativ messbar.

Im Badezimmer ist die gesamte Platine deutlich größer, der BME weiter weg vom Wemos, isst daher konsequent niedrigere Temperaturen.

2/?

DIe Inititative der #IUCr, Rohdaten #FAIR verfügbar zu machen, erfordert ein kompatibles Datenformat. Dieses wird in den Communitys als CIF/CBF bzw. NeXuS in #HDF5-Containern definiert.

Dies führt unter anderem dazu, dass Industrie entsprechende Konverter aus ihren proprietären Dateiformaten bereitstellt 🥳

📢Blosc2 3.10.2 Released! 📢

Building on the functions introduced in version 3.9, we have extended lazy evaluation to general expressions with ALL blosc2 functions in version 3.10.

Blosc2's compute engine ingests many array formats, as well as blosc2 arrays, with impressive results - see the graphic below!
Details on the comparisons in our blog: https://www.blosc.org/posts/tensordot-pure-persistent/

Blosc2's compute engine also powers Cat2Cloud (visit the demo site here https://cat2.cloud/demo/)!

#Zarr
#HDF5
#Python

"I’m experiencing significant slow-downs when converting data from #LMDB to HDF5 format."

Solution: don't do that. #HDF5 is a pig.

https://stackoverflow.com/questions/79706522/hdf5-write-performance-degrades-over-time-when-converting-from-lmdb-3-7m-entri

I've surprised myself how fun it is to write this pure #haskell #hdf5 parser. It's, in a way, gruesome, fiddling with raw bytes, guessing some details, getting new files with features still unsupported, and I have no clue how exciting it will be to implement all the common filters. But the language is delightful and I'm making constant glacial progress. That accounts for a lot I suppose...

🚀 Excited to share more about Caterva2, your ultimate gateway to Blosc2/HDF5 repositories! 🚀

Caterva2 is designed to redefine how you interact with large datasets.

Want to see it in action? 🤔 We've just released a new introductory video showcasing Caterva2's main functionalities! 🎬

👉 https://ironarray.io/caterva2

#Caterva2 #Blosc2 #HDF5 #BigData #DataManagement #FreeSoftware #Python #DataScience #Tech

Now we have THomas from #EuXFEL with “down to the bytes: can we simplify alternative access to #HDF5?“

– h5glance

#HDF5 User Group meeting, preparing for day two …

Zdenek from MAX IV #Synchrotron in Lund shows their “innovative data acquisition solutions with HSDS“.

Now it's Gerd from the #HDF5 group showing the history and future of the “most versatile container for sharing scientific and engineering data”.

Now Elena from the #HDF5 group / lifeboat is revisiting SWMR (single writer, multiple readers) and file versioning features.

MIchael from PureStorage continues on #HDF5 scalability and consistency for #HPC High Performance Computing-Storage.

Now it's @FrancescAlted to introduce the #Blosc2 #compression algorithm to reduce #HDF5 file size.

https://indico.desy.de/event/48471/timetable/#20250526

Moin from the European #HDF5 User Group Meeting @DESY in Hamburg!

We just got an introduction by Anton Barty into science and data storage at DESY, including particle physics, astroparticles, and photon science. Peta Bytes within few days, we'd like to write and read in parallel, with high bandwidth and low latency. Let's see where the problems and solutions are ;)

#HDF5 jest super. W skrócie:

1. Oryginalnie, projekt używał systemu budowania autotools. Instalował binarkę h5cc, która — obok bycia nakładką na kompilator — miała dodatkowe opcje do uzyskiwania informacji o instalacji HDF5.
2. Później dodano alternatywny system budowania #CMake. W ramach tego systemu budowania instalowana jest uproszczona binarka h5cc, bez tych dodatkowych funkcji.
3. Każdy, kto próbował budować przez CMake, szybko odkrywał, że ta nowa binarka psuje większość paczek używających HDF5, więc wracano do autotools i zgłoszono problem do HDF5.
4. Autorzy zamknęli zgłoszenie, stwierdzając (tłum. moje): "Zmiany w h5cc przy użyciu CMake zostały udokumentowane w Release.txt, kiedy ich dokonano - kopia archiwalna powinna być dostępna w plikach z historią."
5. Autorzy ogłosili zamiar usunięcia wsparcia autotools.

Co stawia nas w następującej sytuacji:

1. Praktycznie wszyscy (przynajmniej #Arch, #Conda-forge, #Debian, #Fedora, #Gentoo) używa autotools, bo budowanie przy pomocy CMake psuje zbyt wiele.
2. Oryginalnie uznano to za problem w HDF5, więc nie zgłaszano problemu innym paczkom. Podejrzewam, że wiele dystrybucji nawet nie wie, że HDF5 odrzuciło zgłoszenie.
3. Paczki nadal są "zepsute", i zgaduję, że ich autorzy nawet nie wiedzą o problemie, bo — cóż, jak wspominałem — praktycznie wszystkie dystrybucje nadal używają autotools, a przy testowaniu budowania CMake nikt nie zgłaszał problemów do innych paczek.
4. Nawet nie mam pewności, czy ten problem da się "dobrze" naprawić. Nie znam tej paczki, ale wygląda to, jakby funkcjonalność usunięto bez alternatywy, i tym samym ludzie mogą co najwyżej samemu zacząć używać CMake (wzdych) — tym samym oczywiście psując swoje paczki na wszystkich dystrybucjach, które budują HDF5 przez autotools, o ile nie dodadzą dodatkowo kodu dla wsparcia tego drugiego wariantu.
5. Wszystko wskazuje na to, że HDF5 jest biblioteką, której autorów nie obchodzą ich własni użytkownicy.

https://github.com/HDFGroup/hdf5/issues/1814

#HDF5 is doing great. So basically:

1. Originally, upstream used autotools. The build system installed a h5cc wrapper which — besides being a compiler wrapper — had a few config-tool style options.
2. Then, upstream added #CMake build system as an alternative. It installed a different h5cc wrapper that did not have the config-tool style options anymore.
3. Downstreams that tried CMake quickly discovered that the new wrapper broke a lot of packages, so they reverted to autotools and reported a bug.
4. Upstream closed the bug, handwaving it as "CMake h5cc changes have been noted in the Release.txt at the time of change - archived copy should exist in the history files."
5. Upstream announced the plans to remove autotools support.

So, to summarize the current situation:

1. Pretty much everyone (at least #Arch, #Conda-forge, #Debian, #Fedora, #Gentoo) is building using autotools, because CMake builds cause too much breakage.
2. Downstreams originally judged this to be a HDF5 issue, so they didn't report bugs to affected packages. Not sure if they're even aware that HDF5 upstream rejected the report.
3. All packages remain "broken", and I'm guessing their authors may not even be aware of the problem, because, well, as I pointed out, everyone is still using autotools, and nobody reported the issues during initial CMake testing.
4. I'm not even sure if there is a good "fix" here. I honestly don't know the package, but it really sounds like the config-tool was removed with no replacement, so the only way forward might be for people to switch over to CMake (sigh) — which would of course break the packages almost everywhere, unless people also add fallbacks for compatibility with autotools builds.
5. The upstream's attitude suggests that HDF5 is pretty much a project unto itself, and doesn't care about its actual users.

https://github.com/HDFGroup/hdf5/issues/1814

Three Ways of Storing and Accessing Lots of Images in #Python
https://realpython.com/storing-images-in-python/

Using plain files, #LMDB, and #HDF5. It's too bad there's an explicit serialization step for the LMDB case. In C we'd just splat the memory in and out of the DB as-is, with no ser/deser overhead.

Also they use two separate tables for image and metadata in HDF5, but only one table in LMDB (with metadata concat'd to image). I don't see why they didn't just use two tables there as well.

New software descriptor published on ing.grid! "h5RDMtoolbox - A Python Toolbox for FAIR Data Management around HDF5" by Matthias Probst and Balazs Pritz https://www.inggrid.org/article/id/4028/ #RDM #HDF5 #python #FAIR #kit

NeXus ist ein in unseren ommunities weitverbreitetes #Datenformat. Es bietet einen Standard, welche Parameter gespeichert und wie sie im #HDF5-Datei strukturiert werden sollen, um #Metadaten zusammen mit den Daten in einer hochstrukturierten Weise zu integrieren.

Darauf aufbauend gibt es eine maschinenlesbare #Ontologie; diese definiert eindeutige Bezeichner und schafft ein kontrolliertes Vokabular für die Namen aller experimentellen Parameter und gemessenen Variablen im Experiment.

2/3

Gestern haben Rolf und Heike beim #NFDI-Netzwerktreffen Berlin-Brandenburg unser Daphne-Konsortium vorgestellt. Speziell sind sie auf die Terminologien und Datenformate eingegangen, welche für #Synchrotron- und #Neutronen-Experimente genutzt werden.

1/3

Die Folien gibt's bei Zonodo:
https://zenodo.org/records/12728050

#RDM #Ontologien #NeXus #HDF5

#HDF5 is still a popular file format in scientific computing, but #LMDB is always superior, especially in machine learning. https://www.reddit.com/r/MachineLearning/comments/1ad6j2i/d_how_to_make_my_training_faster/

New preprint available: h5RDMtoolbox - A Python Toolbox for FAIR Data Management around HDF5.

"This paper presents an open-source package, called h5RDMtoolbox, written in Python helping to quickly implement and maintain FAIR research data management along the entire data lifecycle using HDF5 as the core file format"

https://preprints.inggrid.org/repository/view/23/

#datamanagement #metadata #HDF5 #datalifecycle #python

HELPMI:
Develop a user-driven #NeXus extension proposal for laser-plasma (#laser #plasma) experiments, by #Helmholtz #Metadata Collaboration

Basically, having nice and standardised metadata (e.g. in #HDF5 files) for #pewpew experiments, making data #FAIR for others – and also more easy to access for yourself ;)

Now we're learning about h5wasm, especially viewing compressed #hdf5 data files with web assembly #wasm.

And after … a couple of days, here the #bugfix for my #memoryLeak
It turns out that using #HDF5 needs some cleanup routines – that's totally okay, but I have plenty of #Terabytes that worked without 🤔

I'm at ESTEC (ESA's European Space Research and Technology Centre) in the Netherlands this week for my first #PLATO science working team meeting as community scientist. One interesting discussion has been on astronomical file formats (honestly)... Future NASA missions look like they might move from #fits to #ASDF, while other astronomical surveys have started using #HDF5. But it's tough to try to predict exactly where the community will be in 5-10 years time. Input welcome!