#blosc2 — Public Fediverse posts

🚀 Our #PyDataGlobal 2025 tutorial recording on modern #Blosc2 & #Caterva2 features is out!
We show how compression is more than just a space saver, boosting performance for large in-memory & out-of-memory arrays via auto-chunking & parallelism.

We also cover: 🌐 Serving Blosc2/#HDF5 data online with Caterva2 ☁️ Computing directly in the cloud (no downloads needed!)

Watch here: 👉 https://www.youtube.com/watch?v=tUvSI3EpTBQ&list=PLGVZCDnMOq0qmerwB1eITnr5AfYRGm0DF&index=80

#Python #DataScience #BigData #HPC #DataHandling

Interact with your vasts remote datasets right in your phone! 📱

I've built a demo Jupyter notebook that connects to a Cat2Cloud server from an Android phone and slices into an 8 TB dataset, downloading a 1 MB chunk in under 100 milliseconds. ⚡

The 8 TB dataset is from the Gaia DR3 catalogue. As it turns out, there are ~1000 stars in a cube of 100 light-years in our vicinity; the space is mostly empty.🔭 🌌

Try this out by visiting: https://cat2.cloud/demo/roots/@public/large/slice-gaia-3d.ipynb

#Cat2Cloud #Blosc2 #BigData

💊IronPill 2💊
In the second of our series of short videos ("ironPills") showcasing ironArray's work, we see how Blosc2 can be used to power heavy-duty linear algebra (100GB!) workflows
⚡1.5-2x faster than PyTorch + h5py!
🧱 automated chunking optimised for your machine's cache hierarchy
🐍 simple one-line syntax 𝚋𝚕𝚘𝚜𝚌𝟸.𝚖𝚊𝚝𝚖𝚞𝚕(𝙰, 𝙱, 𝚞𝚛𝚕𝚙𝚊𝚝𝚑='𝚘𝚞𝚝.𝚋𝟸𝚗𝚍')

See blog here: https://ironarray.io/blog/la-blosc

#Blosc2
#Data
#SignalProcessing
#LinearAlgebra

💊 IronPill 1💊
In the first of a series of short videos ("ironPills") showcasing ironArray's work, we see how Blosc2 can be used to calculate Fourier approximations:
⚡5x faster than NumPy
👣 fraction of the memory footprint
🐍 pythonic one-line syntax 𝚜𝚞𝚖(𝚊 * 𝚌𝚘𝚜(𝚝) + 𝚋 * 𝚜𝚒𝚗(𝚝), 𝚊𝚡𝚒𝚜=𝟷)

See full notebook here: https://github.com/Blosc/python-blosc2/blob/main/examples/ndarray/ironpill1.ipynb

(inspired by this blog post: https://towardsdatascience.com/numexpr-the-faster-than-numpy-library-that-no-ones-heard-of/)

#Blosc2
#Data
#SignalProcessing

🗣️ Announcing Python-Blosc2 3.8.0 🚀

A step closer to compliance with the array-api standard: data-apis.org/array-api!
This is an effort across all array-based libraries so that your code works (e.g. for both blosc2 and NumPy) by simply changing the import statement below!

Highlights:
✅ C-Blosc2 updated to latest 2.21.2
✅ Incorporate isnan, isfinite, isinf
✅ Better indexing coverage
✅ linspace and arange functions more numerically stable
✅ Improved array-api compliance

#arrayapi #Blosc2

#Blosc2 now runs directly in your browser! Leveraging the power of #WASM, #Pyodide, and #JupyterLite, you can harness efficient, adaptable compression through the web's universal interface. Experience the future of large-scale data processing without leaving your browser window.

Compress Better, Compute Bigger, Share Faster

#WebAssembly #DataCompression #WebDevelopment #DataScience

📢 We are pleased to announce the integration of a new stack feature in #Blosc2 🚀, which allows for stacking large arrays along a new axis.

Performance benchmarks show that while aligned chunks yield the best results, #Blosc2 with unaligned chunks can still outperform #NumPy—a welcome discovery! 🎉

Many thanks to Luke Shaw for his excellent work on this new functionality. 🙏

We've updated our recent blog post:
Check it out! 🔗 https://www.blosc.org/posts/blosc2-new-concatenate/#stacking-arrays

#Python #DataScience #Performance #OpenSource

🚀 Excited to share more about Caterva2, your ultimate gateway to Blosc2/HDF5 repositories! 🚀

Caterva2 is designed to redefine how you interact with large datasets.

Want to see it in action? 🤔 We've just released a new introductory video showcasing Caterva2's main functionalities! 🎬

👉 https://ironarray.io/caterva2

#Caterva2 #Blosc2 #HDF5 #BigData #DataManagement #FreeSoftware #Python #DataScience #Tech

Now it's @FrancescAlted to introduce the #Blosc2 #compression algorithm to reduce #HDF5 file size.

Big news! #Caterva2 enters advanced beta stage 🎉 🎉

Caterva2 is a FOSS distributed system written in Python meant for sharing Blosc2 datasets (either native or converted on-the-fly from HDF5) among different hosts.

It follows the pub-sub paradigm, so it can publish data once and allow multiple subscribers to access it, saving time and resources. It comes with a Python API and a Web interface for easy browsing.

Learn more in https://ironarray.io/caterva2

Make Compression Better 😀
#blosc2 #ironarray

Happy after showing how real-time exploration of the Milky Way stars in an array cube of 7.3 TB can be done in a laptop with 8 GB RAM and 15 GB of free disk space thanks to #ESAGaia data and the magic of #compression via #Blosc2. Also shown how #Btune helps immensely in determining the best combination of filters and codecs for achieving best #performance or cratio.
Thanks to the attendees; it has been a great experience! #SciPy2023

Slides available at: https://www.blosc.org/docs/Exploring-MilkyWay-SciPy2023.pdf

Excited to travel to #SciPy2023 to present our approach on how to explore efficiently the Milky Way by using the #gaia dataset and leveraging #compression. Also, I'll introduce #Btune (btune.blosc.org), an AI tool for improving the compression process without the headaches of trying the many combinations of filters and codecs that are possible in #Blosc2. Looking forward to see you if you are coming too!