#csvdiff — Public Fediverse posts

RE: https://floss.social/@janriemer/114760556247092176

New version of #CSVDiff is out! :awesome:

https://crates.io/crates/csv-diff

Thanks to it now using `extract_if` instead of a hacky drain-then-filter impl, you can now e.g. diff your x-mas wishlist against your "actual-gifts-received list" _25% faster_ (and be happy, if no diff is reported)! 🎅 🚀

⚠️ The new version has an MSRV of 1.88 (in order to use `extract_if`)!

See the complete changelog for details:
https://gitlab.com/janriemer/csv-diff/-/blob/main/CHANGELOG.md#012-23-december-2025

Happy X-Mas y'all! 🎄 🎁

#Rust #RustLang #CSV #Crate #Release

I can't wait to use `extract_if` in #CSVDiff! :awesome:

csv-diff makes use of a "manual" (aka hacky) implementation of it using `drain` (to remove equal csv records) in combination with an "intermediate" HashMap to restore the not-to-be-removed csv records (the ones that are different):

https://gitlab.com/janriemer/csv-diff/-/blob/main/src/diff_result.rs?ref_type=heads#L599

We can _probably_ remove this hacky implementation and replace it with `extract_if`! Very exciting!

2/2

#CSVDiff

#Fuzzing along in #CSVDiff :awesome:

In the second screenshot I've highlighted some interesting parts:

Key field indices are 2 and 3, so when diffing the records, where key fields are highlighted, they'll be compared as `Modify`, because:
- key fields are equal between left and right record
- other fields are unequal between left and right record

The other two records on the right have no corresponding left record - so those are `Add`ed records

#Rust #FuzzTesting #RustLang #PropertyTesting

Huh, seems like I really have been living on the bleeding edge (of #FormalVerification):

https://github.com/creusot-rs/creusot/discussions/1477#discussioncomment-12991148

The verification in the prev toot is currently not possible in #Creusot due to missing specs for the `Hash` trait and HashMap more broadly. 😔

Oh well, seems like (at least currently!) I won't be able to fully verify the diffing algorithm of #CSVDiff.🥺

Options I have now are:
- Only verify parts of the algorithm (that don't depend on HashMap ops)
or
- Use fuzzing/property testing

Just published a new version of csv-diff (v0.1.1) 🚀

https://lib.rs/crates/csv-diff

This fixes a nasty bug regarding sort order of modified csv records. 😖

Details in the MR/PR:
https://gitlab.com/janriemer/csv-diff/-/merge_requests/31

Also, two new incoming PRs for #qsv, the #CSV toolkit:

The first updates to the latest csv-diff, fixing aforementioned bug:
https://github.com/dathere/qsv/pull/2456

The second fixes a bug regarding conversion from column names to indices:
https://github.com/dathere/qsv/pull/2457

#Rust #RustLang #OpenSource #CSVDiff

Ouch, there is another bug and this time it is actually _in #CSVDiff itself_!

It happens with sorting the results of modified rows (urgh, I'm also not happy with the sorting code).😨

Thankfully, datatraveller1 already has found a reproducible example - thank you so much! ❤️

Bug:
https://github.com/dathere/qsv/issues/2443#issuecomment-2598987465

I think I already found a solution, but needs rigorous testing first!

Potential solution:
https://github.com/dathere/qsv/issues/2443#issuecomment-2599681431

#qsv #Bug #csv

Nice, I think I found the bug! 🐛

See all the explanation and possible solution here:

=> https://github.com/dathere/qsv/issues/2443#issuecomment-2597097311

Workaround is also present and explained, so should be no blocker for people.

Will prob provide a fix on the weekend. 🤞

#CSVDiff #qsv #Bug #Fix #Bugfix

Uh ohhhh, someone reported a bug in qsv's `diff` command.😮 🙈

https://github.com/dathere/qsv/issues/2443

Hopefully, we can resolve this soon! 🤞🥺

I have a strong suspicion, but let's see... I need more info first from the OP.

#Bug #Issue #CSVDiff #Diff #CLI #qsv

@shuttle I consequently use #TDD, where possible.

Yes, sure, #Rust prevents a lot of bugs at compile time already, but not logic bugs.

For example in #CSVDiff we have ~70 unit tests and ~12 integration tests. The only "bug report" we have ever gotten was due to a corrupted CSV file (being mistaken with a bug in diff):

See here (qsv):
https://github.com/jqnatividad/qsv/issues/1258#issuecomment-1712924932

csv-diff:
https://gitlab.com/janriemer/csv-diff

In the future I'd like to add property and mutation testing as well 🤓

#RustLang #Testing #UnitTest

#CsvDiff has finally reached v0.1.0, it's first ever non-alpha/-beta release! 🎉

New features like getting at the headers from the diffresult have been needed for the following PR in qsv (which is in final review):
https://github.com/jqnatividad/qsv/pull/1395

When merged, you'll be able to decide, whether the diffresult should output headers or not (see examples in the PR). :awesome:

Check out csv-diff's Changelog for the full details:
https://gitlab.com/janriemer/csv-diff/-/blob/main/CHANGELOG.md?ref_type=heads#010-30-october-2023

#CSV #qsv #CLI #DataScience #DataEngineering

And this is why #UnitTests and #TDD are awesome/necessary (even in #Rust/ #RustLang):

The original requirement:
figure out how many columns the _result_ of diffing two CSVs in #CsvDiff have.

Do you see the error-pattern?

It's
- when we have no diff
&&
- at least one CSV has headers

which makes sense, because I've implemented the feature in the diffing logic, but at that point header information is already lost (in some other thread).

Isn't that beautiful!?🥰

#SoftwareEngineering #UnitTest

I can't reproduce the bug. ¯\_(ツ)_/¯

Neither in #CsvDiff ...
https://gitlab.com/janriemer/csv-diff/-/commit/0ac6ba724656dbcc620eefda3f8ec5d8064134d6

...nor in #qsv
https://github.com/jqnatividad/qsv/pull/1297

My assumption is that they have forgotten to specify the option --right-delimiter (or --left-delimiter, respectively), when executing `qsv diff`:
https://github.com/jqnatividad/qsv/issues/1258#issuecomment-1712842883

Anyway, we now have additional tests in csv-diff and qsv, so definitely a win, regardless of the outcome! 🎉

@floriann FYI

#Rust #RustLang #OpenSource #NotReproducible

Oh noes, apparently I haven't considered different delimiters for the left and right #CSV in #CsvDiff.😱

Someone reported a bug in `qsv diff` (which uses csv-diff) with this scenario.

https://github.com/jqnatividad/qsv/issues/1258

I'll have a look at it tomorrow.

Glad, csv-diff is actively used! ❤️

#Rust #RustLang #OpenSource #Bug #Bugs

If you want to know, how to provide a large resource (such as an owned String) to a criterion benchmark, you can use the `iter_batched` method:

https://docs.rs/criterion/latest/criterion/struct.Bencher.html#method.iter_batched

See an example of this in #CsvDiff

https://gitlab.com/janriemer/csv-diff/-/blob/main/benches/bench_csv_diff.rs#L67

3/3

#Rust #RustLang #Performance #Benchmark #AIIsNotIntelligent

Not sure where this will lead to, but it sounds fun and exciting, so let's try! :awesome: :rust: :ferris:

#CSVDiff

Yay! Sorting the #csv diff result by columns has just been merged into #qsv! 🥳

https://github.com/jqnatividad/qsv/pull/827

#CSVDiff #DataScience #CLI #Data

A new version of csv-diff is out (v0.1.0-beta.2) 🎉

https://lib.rs/crates/csv-diff

This version adds a method, which allows you to sort your diff result by columns (it was already possible to sort by lines).

See the changelog for an example:
https://gitlab.com/janriemer/csv-diff/-/blob/8642a8a7ba14e22d076cee8c3f690c17f41d7528/CHANGELOG.md#010-beta2-19-february-2023

Sorting by columns will soon be integrated into qsv, the #CSV toolkit:
https://github.com/jqnatividad/qsv/issues/714

Thank you @jqnatividad for the idea of this feature! 💚

#Rust #RustLang #CSVDiff #DataScience #qsv #CLI

@hyde Also check out `qsv`. 🙂

It's an actively maintained fork of xsv (xsv is not maintained anymore).

qsv is _very active_ in development.

And shameless plug in the end 😁
Just a few days ago, `csv-diff` got merged:
https://github.com/jqnatividad/qsv/pull/711

csv-diff is a crate for comparing CSVs with ludicrous speed:
https://gitlab.com/janriemer/csv-diff

So the new command `qsv diff` is now the fastest #CSV differ in the world! 🚀

#BlazinglyFast #CSV #CSVDiff #Data #CLI #Rust #RustLang

Announcement 🎉 🥳

csv-diff will be integrated into qsv, the CSV toolkit soon! 🎉 :ferris:

PR:
https://github.com/jqnatividad/qsv/pull/711

Comparing the majestic million dataset with 1,000,000 rows x 12 columns takes less than 800ms and only about 150mb of RAM!
With this, it is the fastest #CSV differ in the world!🚀

See the following svg recording for a demo:

https://gist.githubusercontent.com/janriemer/0e86a236a7174af6faed1b981815a5ea/raw/2ab235a2bdd5798f2348d42da48fdda276fb5f0f/qsv_diff_recording.svg

csv-diff:
https://gitlab.com/janriemer/csv-diff

#Rust #RustLang #Data #Diff #CsvDiff #Difference #Performance #DataScience #Oxidization

🥳 A new version of csv-diff has just been released! 🚀

https://docs.rs/csv-diff/latest/csv_diff/

csv-diff is the fastest CSV-diffing library in the world - written in #Rust

It can compare two 1,000,000 rows x 9 columns CSVs in < 600ms!

Note that this is still a beta release and the library itself is still very young.

#RustLang #Release #CSV #CSVDiff #Performance #DataScience #Data #Diff #Difference #OpenSource #Crate