#rocksdb — Public Fediverse posts

Снапшоты дедуплицирующей файловой системы на LSM-дереве: опыт TATLIN.BACKUP

Привет, Хабр! Меня зовут Александр Черепанов, я руковожу группой в команде Data Path TATLIN.BACKUP. Мы уже писали в блоге про наше хранилище. Сначала про то, как с нуля делали дедуплицирующую файловую систему: чанки, FastCDC, протокол T-BOOST. Потом про то, что такое снапшоты, зачем они нужны и как мы их устроили на уровне архитектуры: экстенты, файл как список их идентификаторов, выбор Mark-and-Sweep вместо подсчета ссылок для сборки мусора. С тех пор снапшоты уже почти год в проде, и, судя по отзывам, работают неплохо. Эта статья принципиально другая. Мы достанем отвертку и залезем внутрь: разберемся, на чем технически строятся снапшоты, как мы их сделали поверх RocksDB, при чем тут LSM-дерево и Bloom-фильтр размером с системный диск ноутбука.

https://habr.com/ru/companies/yadro/articles/1067560/

#схд #снапшоты #lsmдерево #rocksdb

How MVCC and Transactions Work in #RocksDB

https://artem.krylysov.com/blog/2026/07/23/how-mvcc-and-transactions-work-in-rocksdb/

Quite complex, not fully documented, not widely used, easily misused.

That's the problem with retrofitting transactions after the fact...

Disk-native Yjs

#LMDB instead of #RocksDB
It's good for read-heavy operations and has very predictable speed in that regard. I never managed to create a benchmark when RocksDB would actually win here for any realistic scenario.

https://www.bartoszsypytkowski.com/ysr-architecture/

@davidculley so what? Actually, many contributions from #meta #facebook to #opensource world including #MySQL. #RocksDB born there. And don't forget #zstd.

Funny, nothing like this ever happened before Cloudflare switched from #LMDB to #RocksDB. Just sayin...

https://blog.cloudflare.com/introducing-quicksilver-configuration-distribution-at-internet-scale/

https://www.archynewsy.com/cloudflare-quicksilver-multi-level-caching-billions-of-requests/

cl-rocksdb should now work on Arch Linux. This is because the rocksdb package on Arch Linux no longer depends on jemalloc.

https://github.com/veer66/cl-rocksdb/issues/1

#CommonLisp #rocksdb

"What happened – Switching from #RocksDB to #LMDB and tweaking pool allocation cut RSS by 52% in Pinterest’s GraphQL edge layer.

Why it matters – A reminder that “just buy more RAM” is not a strategy when you’re at a Pinterest scale"

https://www.linkedin.com/pulse/new-record-ddos-managing-450m-images-calling-uber-bart%C5%82omiej-%C5%BCyli%C5%84ski-lrncf

RocksDB is a resource hog...

Oh wow, another #database that’s a bajillion times faster than SQLite when you’re doing who-knows-what with a tractor engine! 🚜📈 Is it a bird? Is it a plane? No, it’s just #AnuDB riding on the coattails of #RocksDB while trying to sound impressive with big numbers and fancy GitHub buzzwords. 💨🔧
https://github.com/hash-anu/AnuDB #Performance #TechInnovation #HackerNews #ngated

AnuDB– Backed on RocksDB, 279x Faster Than SQLite in Parallel Workloads

https://github.com/hash-anu/AnuDB

#HackerNews #AnuDB #RocksDB #SQLite #Performance #Database #Technology

Low-Level Memory Management in Scalable Distributed Architectures:
Approaches to Improving Reliability and Performance of Digital
Services
Terletska Khrystyna

Lviv Polytechnic National University, 79013, Ukraine, Lviv

The study includes
practical examples from #RocksDB and #LMDB
https://ijrpr.com/uploads/V6ISSUE4/IJRPR42112.pdf

Apache Flink: тестирование собственного сериализатора состояния

Привет, Хабр! На связи Александр Бобряков, техлид команды МТС Аналитика. Это мой одиннадцатый пост про Apache Flink. В предыдущей части мы рассмотрели сериализацию данных во Flink, написали сериализатор, поддерживающий эволюцию схемы для Flink-состояния в операторе на основе Jackson. В этой части мы научимся писать тесты на эволюцию схемы состояния при использовании своего сериализатора. Весь разбираемый исходный код можно найти в репозитории AlexanderBobryakov/flink-spring . В master-ветке представлен итоговый проект по всей серии. Этот материал соответствует релизной ветке с названием release/10_test_JacksonStateSerializer .

https://habr.com/ru/companies/ru_mts/articles/873436/

#сериализация_данных #jacksonstateserializer #эволюция_схемы #Тестирование_Flink #Statefulоператоры #Flink_MiniCluster #RocksDB #Java_ClassLoader #автоматизация_тестирования #java

SAE paper comparing #LMDB, #RocksDB, and #MongoDB. If automotive engineers are evaluating RocksDB or MongoDB, you should be afraid. Very afraid. https://saemobilus.sae.org/papers/comparative-analysis-rocksdb-lmdb-mongodb-a-performance-evaluation-2024-26-0331

@almibe they did a pretty good benchmark report too https://github.com/lmdbjava/benchmarks/blob/master/results/20160710/README.md

#lmdb #rocksdb #java

And this one, OMFG.

https://x.com/baotiao/status/1853889937063760316

"RocksDB has a more simple API. LMDB doesn't seem designed specifically as a KV storage engine"

This guy has absolutely no friggin idea what he's talking about. #LMDB is the most fundamentally distilled essence of a transactional KV storage engine. #RocksDB is a Rube Goldbergian Frankenstein's monster that even its authors admit they don't know how to tune or optimize correctly.

Starting a series of blog posts with ten facts about YugabyteDB

https://dev.to/franckpachot/10-facts-about-yugabytedb-misconceptions-debunked-4ee4
#postgresql #distributed #lock #mvcc #raft #rocksdb #pusdowns #sync #batch

Relationale Datenbanken wie MySQL sind bewährt, aber Key-Value-Stores gewinnen in der Cloud an Bedeutung. Martin Loschwitz beleuchtet ihre Vorteile wie Etcd in Kubernetes, die weniger komplex sind und für bestimmte Anwendungen effizienter arbeiten. Erfahren Sie, wie Consul und RocksDB die Leistung und Skalierbarkeit verbessern, und warum Key-Value-Stores in modernen IT-Setups oft die bessere Wahl sind.

🔗 https://www.linux-magazin.de/ausgaben/2024/08/key-value-stores/

#KeyValueStore #LinuxMagazin #Consul #RocksDB

🚀 Die August-Ausgabe vom Linux-Magazin ist da! 🎉

Wir tauchen tief in die Welt der NoSQL-Datenbanken ein und zeigen Ihnen, wo ihre Einsatzgebiete liegen und welche Vorteile sie bieten. 🌐💡

🔍 Highlights:

- CouchDB und MongoDB: Dokumentenorientierte NoSQL-Datenbanken
- Qdrant: KI-fähige, vektororientierte Datenbank
- Key-Value-Stores: Einblicke in Etcd (Kubernetes) und RocksDB (Ceph)

📘 Zum Heft https://www.linux-magazin.de/magazine/2024/08/

#LinuxMagazin #NoSQL #Datenbanken #CouchDB #MongoDB #Qdrant #Etcd #RocksDB

Due to its high performance and adaptability, #RocksDB is an excellent choice for modern databases. Let's explore its use in #YugabyteDB and how it has been customized to serve a Distributed SQL database
https://www.linkedin.com/pulse/rocksdb-excellent-choice-modern-sql-databases-lsm-tree-franck-pachot-wze0f

How can I swap out data to disk, to save memory in #rust?

I'm storing lots of data in a big dict/hashtable. Keys are 64bit ints, values about 20-100 bytes. But I have 600-800M items. I'm currently using a `BTreeMap<i64, _>`.

Solution must be Rust, open source. Ideally nothing that needs another service running (ie j want something "embedded")

Is #RocksDB a sensible choice?
:boost_requested:

cl-rocks is a RocksDB binding for Common Lisp. I use it for creating apps and manipulating my databases interactively via a Lisp REPL.

https://github.com/veer66/cl-rocksdb

#commonlisp #rocksdb

An amusing part - their benchmark results show that #LMDB trounces #RocksDB and #MongoDB in both read *and write* performance, yet they still write in their conclusion "RocksDB is optimized for high-performance operations, especially for write-intensive workloads, making it suitable for applications requiring fast data ingestion and low latency."

Completely ignoring their own data.

Have to wonder if an AI wrote this paper.

How paranoid of #Meta should I be? I am down this rabbit hole of trying to install #Firefish on a Pi4 and have reached the point where i need to recompile sonic, which seems to use #RocksDB which is a meta open source product... maybe I should cut my losses and run at this point? Why can't everyone just use Redis?

If you're using #RocksDB instead of #LMDB in your project, you're probably Doing It Wrong.

Higher reliability, greater efficiency, lower write amplification with LMDB https://lobste.rs/s/vgfvfi/what_are_you_doing_this_week#c_d5d5sp

Yes @Yugabyte stores shards as RocksDB, which is popular for small, embedded, write intensive, databases. Then what's the magic using it for large Distributed SQL?
- sharding large tables to small tablets
- optimizing the read path
---
RT @Yugabyte
Does #RocksDB scaling limitation have an impact on #YugabyteDB? 🤔

Find out by reading our new tips & tricks blog!⬇️
https://hubs.la/Q01N3vHr0
https://twitter.com/Yugabyte/status/1652328401212960769

How #RocksDB works https://artem.krylysov.com/blog/2023/04/19/how-rocksdb-works/

How #RocksDB works - Artem Krylysov https://artem.krylysov.com/blog/2023/04/19/how-rocksdb-works/