#ahalodeck — Public Fediverse posts

https://cloud.arkthis.com/index.php/s/AHAlodeck2026?dir=/videos

A 50 minute show-and-tell, about #AHAlodeck, and how #xattrs on #ZFS can be used as #DataCentric catalog/environment.

Enjoy!

Perfect for #longterm preservation #dltp - and any other digital data collections.

#AHAlodeck R&D news:

- Testdataset: 386.5 GB
- 120.000 filesystem objects (files and folders)
- 7616839 key/value informations gathered/extracted as #xattrs.

Result: A 55 MB .tar.bz2 #holotar.

Fully search-and-filter-able by any of the 7 Mio xattrs.

Extraction step required only once (6h)
Afterwards:
Indexing done in 60 SECONDS! 🤩 🥇

#AHAlodeck R&D news:

- Testdataset: 386.5 GB
- 120.000 filesystem objects (files and folders)
- 7616839 key/value informations gathered/extracted as #xattrs.

Result: A 55 MB .tar.bz2 #holotar.

Fully search-and-filter-able by any of the 7 Mio xattrs.

Extraction step required only once (6h)
Afterwards:
Indexing done in 60 SECONDS! 🤩 🥇

#AHAlodeck R&D news:

- Testdataset: 386.5 GB
- 120.000 filesystem objects (files and folders)
- 7616839 key/value informations gathered/extracted as #xattrs.

Result: A 55 MB .tar.bz2 #holotar.

Fully search-and-filter-able by any of the 7 Mio xattrs.

Extraction step required only once (6h)
Afterwards:
Indexing done in 60 SECONDS! 🤩 🥇

#AHAlodeck R&D news:

- Testdataset: 386.5 GB
- 120.000 filesystem objects (files and folders)
- 7616839 key/value informations gathered/extracted as #xattrs.

Result: A 55 MB .tar.bz2 #holotar.

Fully search-and-filter-able by any of the 7 Mio xattrs.

Extraction step required only once (6h)
Afterwards:
Indexing done in 60 SECONDS! 🤩 🥇

Same data (key/value pairs) viewed in different application:

`apt install eiciel`

That's a #DataCentric paradigm in action! You can literally shoot any mixed (annotated) data at that.

It really puts fun back into using data daily.

#AHAlodeck

Handed in my presentation proposal for #nttw10 #notimetowait

Presenting "#DataGardens" 🌻️🌻️🌻️
Powered by #AHAlodeck to host any arbitrary data, offload key/value to filesystem - and transform datas into annotated related object graphs.

Pure "#ScienceFaction"!
#WeAllHaveBigDataNow

in #ZFS on #Linux, #Proxmox, #Debian, #Ubuntu #FOSS #FinF #ObjectStorage

Handed in my presentation proposal for #nttw10 #notimetowait

Presenting "#DataGardens" 🌻️🌻️🌻️
Powered by #AHAlodeck to host any arbitrary data, offload key/value to filesystem - and transform datas into annotated related object graphs.

Pure "#ScienceFaction"!
#WeAllHaveBigDataNow

in #ZFS on #Linux, #Proxmox, #Debian, #Ubuntu #FOSS #FinF #ObjectStorage

Handed in my presentation proposal for #nttw10 #notimetowait

Presenting "#DataGardens" 🌻️🌻️🌻️
Powered by #AHAlodeck to host any arbitrary data, offload key/value to filesystem - and transform datas into annotated related object graphs.

Pure "#ScienceFaction"!
#WeAllHaveBigDataNow

in #ZFS on #Linux, #Proxmox, #Debian, #Ubuntu #FOSS #FinF #ObjectStorage

Handed in my presentation proposal for #nttw10 #notimetowait

Presenting "#DataGardens" 🌻️🌻️🌻️
Powered by #AHAlodeck to host any arbitrary data, offload key/value to filesystem - and transform datas into annotated related object graphs.

Pure "#ScienceFaction"!
#WeAllHaveBigDataNow

in #ZFS on #Linux, #Proxmox, #Debian, #Ubuntu #FOSS #FinF #ObjectStorage

Download #wikidata dump for testing #ahalodeck and #xattrs on #zfs long-term storage... 😎

Found this while doing so:

https://www.wikidata.org/wiki/Wikidata:Lists/lexemes/en

It's a statistical breakdown of the #English #language. #Interesting.

#AHAlodeck R&D news:

Created holotar-copy of a real-world mixed audio collection (recording production and digitized tracks archive): ~120.000 filesystem objects (files/folders)

(donated by recording engineers)

filesystem metadata-only = 175 MB (.tar)
+exiftool metadata as xattrs = 252 MB (.tar)

...compressed (.tar.bz2): **6,3 MB** ( 🤯 😎 🤓 💾 ❗)

**This is AMAZING!**

I can browse, order, catalog and manifest any object in this collection using standard #GNU #Linux tools.
@beet_keeper

#AHAlodeck: My best friend wrote a mini-basic fulltext indexer for #xattrs in the shell.

Took ~1.5 seconds (!) to index ~14.000 files with "de-embedded" music tags.

🥳 🎉
I love xattrs.
Imagine doing this with embedded metadata tags? 😉

https://git-annex.branchable.com/git-annex-metadata/

git seems to have embedded key-value #metadata annotation capabilities... 😻 #AHAlodeck ahoi!

https://github.com/ArkThis/AHAlodeck/blob/main/res/0byte-db_entry.png

#xattr #linux #ext4 #zfs #samba #smb

Right-click-edit-metadata happily in-use over 1 year now: Amazing!

#AHAlodeck #Recoll #SENlabs

so **each** key/value-pair of #metadata can be about 64kB.

This is sufficient to hold most annotations stored as #EmbeddedMetadata currently present in most file formats.
(eg #mp3, #mp4, #iptc, #tiff, #jpg, etc)

But from then on, the file-format becomes irrelevant for this task!

#ahalodeck #digipres #audiovisual #cataloging

so **each** key/value-pair of #metadata can be about 64kB.

This is sufficient to hold most annotations stored as #EmbeddedMetadata currently present in most file formats.
(eg #mp3, #mp4, #iptc, #tiff, #jpg, etc)

But from then on, the file-format becomes irrelevant for this task!

#ahalodeck #digipres #audiovisual #cataloging

Very helpful insights on "how much actual bytes" can be stored in #xattrs on `btrfs`:

https://github.com/kdave/btrfs-progs/issues/917

Current limit = about the nodesize (it was formatted with).

nodesize - sizeof(struct btrfs_header) - sizeof(struct btrfs_item) - sizeof(btrfs_dir_item)

Current default = 16K - 101 - 25 - 30 , which is 16228 bytes

Maximum (for now) is ~64kB, if formatted like this:

`$ mkfs.btrfs -n 64K /dev/sdX`

#ahalodeck #metadata #digipres

https://vectorvfs.readthedocs.io/en/latest/index.html

#VectorFS: "transforms your Linux filesystem into a vector database by leveraging the native VFS (Virtual File System) extended attributes."

This proofs my design to offload /many/ #database functionality onto the filesystems is a working way to go.

*nice* 😌 🛋️ #AHAlodeck

https://sen-labs.org/

"Link local files, reference data seamlessly, and manage your personal knowledge with just your file browser – all open-source and integrated into your filesystem."

Sounds familiar?

No this is not my #AHAlodeck, but I'm very happy someone has the same ideas - and even happier that we're now in contact with each other! 🥳

"Personal Knowledge Graphs: why, what, and where to?"
(Jul 4, 2021)

Ivo Velitchkov

https://youtu.be/7ik0JNnoXBc

Very interesting. It confirms my observations and research on #AHAlodeck:

It's time we allow ourselves to do-digital-data differently. #GLAM #DLTP #Archiving

Oh, I'm loving this!

> "The Data-Centric revolution puts data at the center of the enterprise. Applications are optional visitors to the data."

https://datacentricmanifesto.org/

#AHAlodeck #RDF #GraphQL #xattrs

https://personalknowledgegraphs.com/#/page/extending%20the%20desktop%20into%20a%20personal%20knowledge%20graph%20with%20sen

I'm not alone! 🥳

Yes! It's time to offload meta+data issues to the filesystem. A common, standard ground - and scale with hardware - not #EternalMigration neverending human-hours...

#Haiku #BeOS #AHAlodeck

Me engaging on #Smbd ("Samba") mailing lists about support for "larger" (eg 64kB) key/value attributes on filesystem level (#xattrs):

https://lists.samba.org/archive/samba-technical/2025-February/139325.html

Good news! 🥳

> "in 2021 we added smbd max xattr size with a default of 64KB."

#AHAlodeck #DLTP

Reading a "History of Graph Databases" article on #redis:

https://redis.io/glossary/graph-database/

Having worked with mostly SQL and table data, I feel like entering a new era of data handling!

(changing data paradigms, changes everything - 😎 🌈 🦄 ) #AHAlodeck

I truly believe it's time to take "thinking data differently" more serious and read up on Wikidata-related data-modeling stuff:

https://www.mediawiki.org/wiki/Wikibase/DataModel/Primer

#DigiPres #Metadata #Cataloging #AHAlodeck 🌈 🤯 💾