#bzip2 — Public Fediverse posts

Holy crap! It's got #Unicos too! I shall resurrect my project from 1999 to get #bzip2 to support #Cray!

(the bug is that bzip2 insists (insisted?) that `short` be 16 bits)

Holy crap! It's got #Unicos too! I shall resurrect my project from 1999 to get #bzip2 to support #Cray!

(the bug is that bzip2 insists (insisted?) that `short` be 16 bits)

@BRicker @Boston_PM my favourite C portability screw up is #bzip2 which insists that `short` be 16 bits.

Getting that to work on a platform where short, int and long were all 64 bits, and where you only had C89, no fancy C99 data types like `int16_t`, was "fun".

@BRicker @Boston_PM my favourite C portability screw up is #bzip2 which insists that `short` be 16 bits.

Getting that to work on a platform where short, int and long were all 64 bits, and where you only had C89, no fancy C99 data types like `int16_t`, was "fun".

Archivierung und Kompression mit tar: Grundlagen, Optionen und Beispiele

https://andreas-moor.de/archivierung-und-kompression-mit-tar-grundlagen-optionen-und-beispiele/

Kompression mit zip: Grundlagen, Optionen und Beispiele

https://andreas-moor.de/kompression-mit-zip-grundlagen-optionen-und-beispiele/

Kompression mit bzip2 Linux: Grundlagen, Optionen und Beispiele

https://andreas-moor.de/kompression-mit-bzip2-linux-grundlagen-optionen-und-beispiele/

Kompression mit gzip: Grundlagen, Optionen und Beispiele

https://andreas-moor.de/kompression-mit-gzip-grundlagen-optionen-und-beispiele/

Vergleich von Archivierung und Kompression unter Linux: tar/star, gzip, bzip2, xz und zip

https://andreas-moor.de/vergleich-von-archivierung-und-kompression-unter-linux-tar-star-gzip-bzip2-xz-und-zip/

WARNING

TIL that bunzip2 in Linux / Debian / MX Linux deletes, I repeat, DELETES the original archive when you run it in vanilla form

`bunzip2 dfly-x86_64-6.4.2_REL.iso.bz2`

resulted in the iso being unpacked with deletion of the original WITHOUT WARNING

Since I'm on expensive LTE+ 4G internet that is significant.

I have not used bunzip2 in years, but should have remembered this hostile default. It was not that way IIRC
Do I need to read the manpages of commands I have not used in years in Linux now? Why was the default changed

Luckily I have copies of the bzip2 iso on multiple partitions of HDD and SSD

#bzip2 #bunzip2 #sh #bash #warning #TIL #Linux #OpenSource #POSIX

WARNING

TIL that bunzip2 in Linux / Debian / MX Linux deletes, I repeat, DELETES the original archive when you run it in vanilla form

`bunzip2 dfly-x86_64-6.4.2_REL.iso.bz2`

resulted in the iso being unpacked with deletion of the original WITHOUT WARNING

Since I'm on expensive LTE+ 4G internet that is significant.

I have not used bunzip2 in years, but should have remembered this hostile default. It was not that way IIRC
Do I need to read the manpages of commands I have not used in years in Linux now? Why was the default changed

Luckily I have copies of the bzip2 iso on multiple partitions of HDD and SSD

#bzip2 #bunzip2 #sh #bash #warning #TIL #Linux #OpenSource #POSIX

Ah yes, because what the world desperately needs is yet another #BZip2 encoder, but this time dressed up in #Ada and sprinkled with the ✨ magic ✨ of AIMachineLearning™. It's the classic tale: boy meets algorithm, algorithm meets Ada, and everyone lives happily ever after in a world of compressed bits nobody asked for. 🤷‍♂️💾
https://gautiersblog.blogspot.com/2025/09/writing-competitive-bzip2-encoder-in.html #AIMachineLearning #Compression #TechHumor #HackerNews #ngated

Ah yes, because what the world desperately needs is yet another #BZip2 encoder, but this time dressed up in #Ada and sprinkled with the ✨ magic ✨ of AIMachineLearning™. It's the classic tale: boy meets algorithm, algorithm meets Ada, and everyone lives happily ever after in a world of compressed bits nobody asked for. 🤷‍♂️💾
https://gautiersblog.blogspot.com/2025/09/writing-competitive-bzip2-encoder-in.html #AIMachineLearning #Compression #TechHumor #HackerNews #ngated

Writing a competitive BZip2 encoder in Ada from scratch in a few days – part 3

https://gautiersblog.blogspot.com/2025/09/writing-competitive-bzip2-encoder-in.html

#HackerNews #Writing #BZip2 #Ada #Encoder #Competitive #Coding #Part3

Writing a competitive BZip2 encoder in Ada from scratch in a few days – part 3

https://gautiersblog.blogspot.com/2025/09/writing-competitive-bzip2-encoder-in.html

#HackerNews #Writing #BZip2 #Ada #Encoder #Competitive #Coding #Part3

Xz format inadequate for general use
https://www.nongnu.org/lzip/xz_inadequate.html
#ycombinator #lzip #LZMA #bzip2 #gzip #data_compression #long_term_archiving

Xz format inadequate for general use
https://www.nongnu.org/lzip/xz_inadequate.html
#ycombinator #lzip #LZMA #bzip2 #gzip #data_compression #long_term_archiving

🎉 Behold, the groundbreaking revelation: #Xz is not the Holy Grail of data formats! 🚀 Apparently, using xz for digital preservation is like using a sieve as a bucket—bound to fail. Who knew? 🤦‍♂️ Stick to #bzip2, #gzip, or #lzip if you want actual functionality and avoid sinking your data into the abyss of inadequacy. 🔍💾
https://www.nongnu.org/lzip/xz_inadequate.html #dataformats #digitalpreservation #HackerNews #ngated

🎉 Behold, the groundbreaking revelation: #Xz is not the Holy Grail of data formats! 🚀 Apparently, using xz for digital preservation is like using a sieve as a bucket—bound to fail. Who knew? 🤦‍♂️ Stick to #bzip2, #gzip, or #lzip if you want actual functionality and avoid sinking your data into the abyss of inadequacy. 🔍💾
https://www.nongnu.org/lzip/xz_inadequate.html #dataformats #digitalpreservation #HackerNews #ngated

🚀 Oh, the riveting #saga continues! Witness as #Ada, the language nobody asked for, takes on yet another #unnecessary feat: building a #BZip2 #encoder that absolutely nobody needed – in #record time! 🤯 Part 2, because once wasn't enough! 🤡
https://gautiersblog.blogspot.com/2025/07/writing-bzip2-encoder-in-ada-from.html #Feat #Time #Part2 #HackerNews #ngated

🚀 Oh, the riveting #saga continues! Witness as #Ada, the language nobody asked for, takes on yet another #unnecessary feat: building a #BZip2 #encoder that absolutely nobody needed – in #record time! 🤯 Part 2, because once wasn't enough! 🤡
https://gautiersblog.blogspot.com/2025/07/writing-bzip2-encoder-in-ada-from.html #Feat #Time #Part2 #HackerNews #ngated

Writing a competitive BZip2 encoder in Ada from scratch in a few days – part 2

https://gautiersblog.blogspot.com/2025/07/writing-bzip2-encoder-in-ada-from.html

#HackerNews #Writing #BZip2 #Encoder #Ada #Programming #Competitive #Coding #Part2

Writing a competitive BZip2 encoder in Ada from scratch in a few days – part 2

https://gautiersblog.blogspot.com/2025/07/writing-bzip2-encoder-in-ada-from.html

#HackerNews #Writing #BZip2 #Encoder #Ada #Programming #Competitive #Coding #Part2

Как написать bzip2-архиватор на Python: разбираем преобразование Барроуза-Уилера

Привет! Я Рома, бэкендер-питонист в KTS . Это вторая статья в моем цикле об алгоритме архивации bzip2 . Первую можно прочитать здесь , но для понимания сегодняшней темы она необязательна. Ниже я разберу преобразование Барроуза-Уилера — ключевой этап сжатия bzip2.

https://habr.com/ru/companies/kts/articles/937554/

#архиваторы #архивация #сжатие_данных #алгоритмы #bzip2архиватор #bzip2 #bwt

@ermo

I'm very slowly creeping towards having checksum files auto-built. There's a missing part that needs to be done.

But I'm at least over one initial hurdle of switching from pax -z to pax -j. Not that that helps in the #FreeBSD 10 case because FreeBSD 10's pax does not have -j.

(Make an archive with -z and it isn't idempotent, because #gzip has a timestamp.)

So there's still the installing #GhostBSD mountain to climb, and seeing whether that has pax -j yet. (-:

#bzip2 #pax

@ermo

I'm very slowly creeping towards having checksum files auto-built. There's a missing part that needs to be done.

But I'm at least over one initial hurdle of switching from pax -z to pax -j. Not that that helps in the #FreeBSD 10 case because FreeBSD 10's pax does not have -j.

(Make an archive with -z and it isn't idempotent, because #gzip has a timestamp.)

So there's still the installing #GhostBSD mountain to climb, and seeing whether that has pax -j yet. (-:

#bzip2 #pax

You'll find this benchmarking adventure in its own blog post "Performance lessons of implementing lbzcat in Rust" https://anisse.astier.eu/lbzip2-rs.html

#RustLang #lbzip2 #bzip2 #benchmarking #performance

You'll find this benchmarking adventure in its own blog post "Performance lessons of implementing lbzcat in Rust" https://anisse.astier.eu/lbzip2-rs.html

#RustLang #lbzip2 #bzip2 #benchmarking #performance

lbzip2 internally implements a full task-scheduling runtime, and splits tasks at a much smaller increments; it supports bit-aligned blocks (that are standard in bzip2 format), while my Rust implementation purposefully doesn't: I wanted to rely on the bzip2 crate that only supports byte-aligned buffers, and keep code simple (which I failed IMHO). FIN 15/15

#lbzip2 #bzip2

lbzip2 internally implements a full task-scheduling runtime, and splits tasks at a much smaller increments; it supports bit-aligned blocks (that are standard in bzip2 format), while my Rust implementation purposefully doesn't: I wanted to rely on the bzip2 crate that only supports byte-aligned buffers, and keep code simple (which I failed IMHO). FIN 15/15

#lbzip2 #bzip2

That's it for the benchmarking! You can find my implementation at http://github.com/anisse/lbzip2-rs/ ; it's very much PoC-quality code, so use at our own risks! I chose to manually spawn threads instead of using rayon or an async runtime; there are other things I'm not proud of, like busy-waiting instead of condvar for example. 14/N

#lbzip2 #bzip2 #RustLang #async #rayon

That's it for the benchmarking! You can find my implementation at http://github.com/anisse/lbzip2-rs/ ; it's very much PoC-quality code, so use at our own risks! I chose to manually spawn threads instead of using rayon or an async runtime; there are other things I'm not proud of, like busy-waiting instead of condvar for example. 14/N

#lbzip2 #bzip2 #RustLang #async #rayon

We've been running benchmarks on single CPU cores since the start. What if we unleash the parallel mode? Here are the results: lbzip2 is still much faster on the 8 cores; my implementation holds up fine, but is only 80% faster than bzip2, while running on 8 cores. On bigger files though, it starts to pay off, with up to 6.3x faster, while lbzip2 can go to 7.7x. 13/N

#lbzip2 #bzip2

We've been running benchmarks on single CPU cores since the start. What if we unleash the parallel mode? Here are the results: lbzip2 is still much faster on the 8 cores; my implementation holds up fine, but is only 80% faster than bzip2, while running on 8 cores. On bigger files though, it starts to pay off, with up to 6.3x faster, while lbzip2 can go to 7.7x. 13/N

#lbzip2 #bzip2

Overall, my Rust implementation (using the bzip2-rs crate) is (much) slower than lbzip2, and faster than bzip2. For some reasons, it also sees huge performance boost on performance cores, most likely due to better IPC and branch prediction. 12/N

#lbzip2 #bzip2

Overall, my Rust implementation (using the bzip2-rs crate) is (much) slower than lbzip2, and faster than bzip2. For some reasons, it also sees huge performance boost on performance cores, most likely due to better IPC and branch prediction. 12/N

#lbzip2 #bzip2

What about my implementation of lbzcat? It was designed to work with files generated by lbzip2: it does not work on some files compressed by bzip2, and silently produces incorrect output (!). So we'll limit benchmarking to files produced by lbzip2. 11/N

#lbzip2 #bzip2 #RustLang

What about my implementation of lbzcat? It was designed to work with files generated by lbzip2: it does not work on some files compressed by bzip2, and silently produces incorrect output (!). So we'll limit benchmarking to files produced by lbzip2. 11/N

#lbzip2 #bzip2 #RustLang

But wait, there is more. lbzip2 also does compression, and does so in a way that optimizes decompression. If we use a file compressed by lbzip2, it can be even faster, even on a single thread: up to 125% faster 10/N

#lbzip2 #bzip2

But wait, there is more. lbzip2 also does compression, and does so in a way that optimizes decompression. If we use a file compressed by lbzip2, it can be even faster, even on a single thread: up to 125% faster 10/N

#lbzip2 #bzip2

While running this, I discovered something. lbzip2's detection of the cores it can run on is... lacking: it uses the number of globally online cpu cores instead the syscall sched_getaffinity for the current process, so let's manually limit the number of processing threads: lbzip2 is now always faster, between 4% and 8%. 9/N

#lbzip2 #bzip2

While running this, I discovered something. lbzip2's detection of the cores it can run on is... lacking: it uses the number of globally online cpu cores instead the syscall sched_getaffinity for the current process, so let's manually limit the number of processing threads: lbzip2 is now always faster, between 4% and 8%. 9/N

#lbzip2 #bzip2

Let's have a look! Indeed, bzip2 is faster than lbzip2. Between 18% and 95% faster! 8/N

#lbzip2 #bzip2

Let's have a look! Indeed, bzip2 is faster than lbzip2. Between 18% and 95% faster! 8/N

#lbzip2 #bzip2

That gives us our baseline: bzip2 (in C) vs bzip2 (in Rust). But is it a fair enough comparison? I mentioned initially that I was implementing an lbzip2 "clone" (mostly a PoC for the decompression part). lbzip2 is an other program (a C binary, without a library), that can compress and decompress bzip2 files in parallel. Surely it should be slower than bzip2 since it has the parallel management overhead? 7/N

#lbzip2 #bzip2 #RustLang #perf

But why? Let's see what perf stat has to say: the Rust version has less instructions, but with much less IPC (Instruction-per-clock); the Rust version also has less branches and misses in general. On the efficiency cores, we see that worse IPC and branch prediction of the Rust version give the advantage to the C version. 6/N

#bzip2 #RustLang #perf

What about the efficiency cores? (0-3 on my system); well, everything breaks down on those CPUs. The C implementation is now faster by a small margin! 5/N

Edit: it's probably not significant enough, see answers.

#bzip2

So, let's start with the most obvious, the performance cores; they make the most sense for a CPU-intensive task; on my systems those are CPU 4-7. We can see here that the promise of the Trifecta Foundation holds, as the rust implementation is about 4% faster. 4/N

#bzip2 #RustLang #AsahiLinux

I'll do benchmarks using my own bzcat and comparing that with bzip2's original bzcat. All running on an M1 laptop running Linux. The M1 CPU has both performance and efficiency cores. We'll run the benchmark on both, using hyperfine to do the comparisons. 3/N

#bzip2 #AsahiLinux #hyperfine

First, let's look at the baseline. The @trifectatech foundation recently claimed that bzip2-rs had faster decompression than libbzip2's original C implementation. Is this true? Well, it depends... 2/N

#bzip2 #RustLang

For fun I implemented lbzip2's lbzcat (parallel bzip2 decompression) clone in rust, using the bzip2 crate. I found a lot of different interesting things. 1/N

#lbzip2 #bzip2 #RustLang

Ah, the riveting saga of squeezing a #BZip2 encoder into the confines of #Ada continues! 🤔 Because nothing screams "cutting-edge competition" like dusting off a 1980s language for a #compression algorithm that takes just "a few days" (and a few decades too many) to write! 🚀🔧
https://gautiersblog.blogspot.com/2025/07/writing-bzip2-encoder-in-ada-from.html #CuttingEdge #Tech #RetroProgramming #HackerNews #ngated

Writing a competitive BZip2 encoder in Ada from scratch in a few days – part 2

https://gautiersblog.blogspot.com/2025/07/writing-bzip2-encoder-in-ada-from.html

#HackerNews #Writing #BZip2 #Encoder #Ada #Competitive #Coding #Part2

The `bzip2` crate switches from C to 100% Rust, https://trifectatech.org/blog/bzip2-crate-switches-from-c-to-rust/.

Use less memory. Faster to run. Simpler cross-compilation. That’s it.

The crate has been audited too, neat!

#RustLang #c #bzip2 #compression