#lmdb — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #lmdb, aggregated by home.social.
-
Can Persistent Tries Beat #LMDB ? Leaves Database Benchmarked
https://hackernoon.com/can-persistent-tries-beat-lmdb-leaves-database-benchmarked
Trie wins easily with fixed-size keys and point lookups. LMDB still wins for sequential reads, range lookups.
The writeup fails to report resource usage of each tested system, which is also a significant consideration; LevelDB and RocksDB are resource hogs. No idea what the Trie footprint is.
-
@cararemixed ISTR that JFS is pretty good with large numbers of small files. It's a question that comes up a lot. https://superuser.com/questions/1352833/what-filesystem-can-i-use-for-a-large-amount-of-small-sized-data
LLM training also involves millions of small files, but most of those systems now use #LMDB instead, storing image files as DB records. This is a workload that LMDB handles better than any filesystem.
-
Your Executable is an SQLite Database
https://www.reddit.com/r/sqlite/comments/1vwsyja/your_executable_is_a_sqlite_database/
Defines an SQL schema for mapping an ELF file to a database and provides a binfmt_misc loader for running an executable SQLite file.
Reminds me of my scheme to store executable slapd modules as #LMDB records. His approach would be more efficient using LMDB instead of SQLite's B+tree code.
-
SDNS
A high-performance, recursive DNS resolver server with DNSSEC support, focused on preserving privacy.
Benchmarks faster than PowerDNS Recursor, Unbound, and KnotDNS Resolver
https://sdns.dev/BENCHMARKS.html
KnotDNS uses #LMDB as a cache but still has the slowest results. A fast data store won't help you if everything running above it is slow.
-
dpdata
Manipulating data formats of DeePMD-kit, VASP, QE, PWmat, and LAMMPS, etc.
Turn atomistic simulation outputs into interoperable, machine-learning-ready datasets.
Export DeePMD NumPy layouts or a single #LMDB database; LMDB can store frames with different compositions and atom counts.
-
Awesome League
A curated list of tools, libraries, and resources for working with League of Legends files, assets, and mods
Hash → path tables for resolving the game's hashed file and field names.
* League Hashes by CDragon - Actively maintained collection of hashes used by League
* lmdb-hashes - Packs the CDragon hash tables into #LMDB databases published as zstd-compressed releases -
I've just come back to a project I started about a month ago, and in the meantime #LMDB has released version 1.0!
I'm super excited because encrypting the database is a requirement of mine, and in the 0.9 versions the encryption support was in an unofficial branch. Now in 1.0 it's included by default! :flan_hurrah:
-
Stability Engineering for Large-Scale Distributed Training:
The optimization techniques described in this article—distributed manual GC, CPU core binding and NUMA optimization, #LMDB prefetching, —have been engineered and implemented using HyperAcc. This has resulted in an average 50% increase in overall training throughput, These techniques have already been successfully deployed in autonomous driving mass production model manufacturing tasks.
-
Algolia vs Typesense vs Meilisearch: Engine Choice (2026)
* Evaluating Algolia vs Typesense vs Meilisearch centres on commercial merchandising features versus infrastructure cost control.
*Typesense runs entirely in RAM, Meilisearch uses disk-backed #LMDB storage, and Algolia operates a managed cluster with a 100 KB record limit.
https://no7software.co.uk/blog/algolia-vs-typesense-vs-meilisearch-ecommerce
-
ArcticDB is a high-performance, serverless DataFrame database designed specifically for the Python data science ecosystem. It lets you work with Pandas DataFrames and NumPy arrays while storing the underlying data in storage systems such as Amazon S3 or #LMDB .
-
Want to boost your job prospects? Get some experience with #LMDB https://jobcannon.io/skills/lmdb-embedded-database
"Mastery takes 3-4 weeks. Practitioners earn 15-25% premium because they optimize applications 10-100x faster. The 5% who design LMDB schemas for high-throughput are valuable in fintech, real-time systems."
-
HS5 is a high-performance, self-hosted object storage service designed to scale up on a single node.
The software provides an Amazon S3-compatible API alongside a web interface for managing buckets, users and access permissions. It separates object metadata from the stored data, allowing metadata to reside on fast storage while object contents are kept on higher-capacity disks.
https://www.linuxlinks.com/hs5-self-hosted-object-storage-service/
Built on #LMDB
-
If you use #LMDB you're already getting the most efficient possible use of your RAM, so news like this won't hit you as hard as it otherwise might. https://graphics.social/@metin/117117572588422759
-
RDF4j 6.0.0 released with improved #LMDB support
-
When the dataset does not fit in RAM
Parquet, WebDataset and #LMDB solve the same problem by different routes: making it possible not to read. Anatomy of the columnar format, batch reading with constant memory, and a benchmark measured cold over 30 million rows and 40,000 samples — with the numbers, the code and the primary sources.
https://manpla.net/en/posts/when-the-dataset-does-not-fit-in-ram/
-
A high-performance #LMDB backend for konserve using Project Panama FFI (Java 22+).
* Zero-copy reads via LMDB's memory-mapped architecture
* Projection: read one field out of a range of values without decoding them (below) — 49x PostgreSQL JSONB on the case it is built for
https://github.com/replikativ/konserve-lmdb
These guys get it: store records in a format that requires no deserialization, then you can get direct access to subfields of large complex records instantly.
-
Another key difference between #LMDB and #SQLite's B+tree code: while both are single writer and both support multiprocess access, SQLite uses shared/exclusive reader/writer locks. That means while any process has write access to the DB, all other processes are blocked. In LMDB, writers don't block readers, readers never block.
That blocking behavior in SQLite makes even small deployments feel sluggish, making people turn to solutions that are way overkill. https://www.xda-developers.com/replaced-home-assistants-built-in-database-postgresql-smart-home-snappy/
-
Transition metal complex homogenization catalyst ethylene refueling reaction DFT calculation data set
The dataset is more than 4.5GB in size, covering 402 different catalytic reaction systems, with a total of 628,073 single-point energy data points. The data set mainly contains the three-dimensional structure information of the catalyst molecule, the energy of each state point on the reaction path, the geometric configuration, ...
https://www.nbsdc.cn/general/dataDetail?id=6a78a419f175603b25c449bb&type=1
In #LMDB format
-
@live-change/db-store-lmdb
Database with observable data for live querieshttps://npmmirror.com/package/%40live-change/db-store-lmdb?version=0.1.23
Built on #LMDB
-
Axis
A fault-tolerant, strongly consistent distributed key-value store designed for coordination, distributed locking, and metadata management.
https://rafeemohamed.com/projects/axis/
Written in java, built on #LMDB
-
TIL Autodesk uses #LMDB https://www.dllme.com/dll/files/lmdb
-
References for the book _Designing_Data_Intensive_Applications_
https://github.com/ept/ddia-references/blob/master/chapter-03-refs.md
Seems like #LMDB is the only Copy-on-Write B+tree here. Lots of LSM references though.
-
To recap: jd.com is the largest retailer in China. Their web servers host billions of product listings, using a customized redis interface over #LMDB . Their testing found LevelDB performance to be too erratic, RocksDB performance good in pure write or pure read, but erratic in mixed read/write workloads. They found LMDB was the only storage engine that gave consistent performance and met their needs.
-
Billion-level commodity details page evolution technology decryption | High Availability Architecture Series
We 618 PV hundreds of millions of dollars on the day, the server-side TOP99 response time is less than 38ms
Data storage, we use JIMDB, Redis plus persistent storage engine, can store more than N times the amount of data in memory, some of our current systems are Redis+ #LMDB engine storage
-
OwnTracks Recorder stores and serves the location data published by the OwnTracks apps for iOS and Android. It subscribes to an MQTT broker, keeps the history in a local #LMDB store, and provides a web interface plus an HTTP API for maps, tracks and friend positions - a self-hosted alternative to commercial location sharing.
-
CoreDB is an embedded bitemporal graph database designed around graph evolution as a first-class concept.
The progression I’m exploring is:
G → G(t) → 𝒢[t₀,t₁]Not just:
“What did the graph look like?”
but:
“What changed?”
“What persisted?”
“What appeared or disappeared?”
“What did the system know at the time?”It runs as an embedded library on #LMDB , so there’s no server or network dependency.