home.social

#parquet — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #parquet, aggregated by home.social.

  1. Some design tests for 🟥 🟩 diff'ing files hosted on Hugging Face.

    With the new storage backend (Xet), similar files share many data "chunks".

    As HF provides an API to get the list of these chunks, you can now compute the diff and display a chart in the browser without downloading the files. Very fast.

    Test here: observablehq.com/@severo/parqu

    Details about the three charts in the responses.

    #parquet #huggingface #diff #dataviz #deduplication