#snakemakehackathon2026 — Public Fediverse posts

New blog post: https://blogs.fediscience.org/rupture-de-catenaire/2026/08/09/the-2nd-snakemake-hackathon/

My reflections on the latest #SnakemakeHackathon2026 . A little late, but nevertheless.

If you want to comment, please do so under the article. Only there comments will be persistent.

#Reproduciblecomputing #HPC #Hackthon

RE: https://fediscience.org/@snakemake/116571962095785816

This little bit "performance improvements" lowered the number of file system access events for considerably! #Snakemake triggers many such events for keeping track of metadata. Which is important, but may cause some delays due to file system overhead - particularly on parallel and/or network file systems. The feature to outsource parts of this to sqlite was implemented during the #SnakemakeHackathon2026 . I hope, I can test the improvements next Monday!

#HPC #ReproducibleComputing

RE: https://fediscience.org/@biohackrxiv/116477505622903440

Today, we published the summary (as a #BioHackrXiv preprint) of the #SnakemakeHackathon2026 . All the accomplishments which contribute once more to improve the #Snakemake "ecosystem" for #reproducibleResearch & #Dataanalysis .

Many thanks to @egonw from BioHackrXiv for helping us!

"Snakemake Hackathon 2026" https://doi.org/10.37044/osf.io/h6zqj_v1

"Nonetheless, the platform’s continued evolution faces several open challenges: improving core performance on heterogeneous high-performance-computing (HPC) resources, extending the plugin architecture for domain-specific extensions, and lowering the entry barrier for novice users while preserving full reproducibility. Here we report on the Snakemake Hackathon 2026, convened in Munich, Germany (9–13 March 2026) with more than 40 participants representing academia, industry, and national-level research infrastructure." https://index.biohackrxiv.org/2026/04/27/h6zqj.html

#SnakemakeHackathon2026 #biohackathon #snakemake

The #SnakemakeHackathon2026 has ended, we are still preparing our preprint release. But, our host has prepared a note on their homepage: https://go.tum.de/946236 🥳

#Snakemake #ReproducibleComputing

RE: https://fediscience.org/@snakemake/116295568336688286

This is a big step forward: The SLURM plugin for Snakemake now supports so-called job arrays. These are cluster jobs, with ~ equal resource requirements in terms of memory and compute resources.

The change in itself was big: The purpose of a workflow system is to make use of the vast resources of an HPC cluster. Hence, jobs are submitted to run concurrently. However, for a job array, we have to "wait" for all eligible jobs to be ready. And then we submit.

To preserve concurrent execution of other jobs which are ready to be executed, a thread pool has been introduced. In itself, I do not see job arrays as such a big feature: The LSF system profited much more from arrays than the rather lean SLURM implementation does.

BUT: the new code base will ease further development to pooling many shared memory tasks (applications which support no parallel execution or are confined to one computer by "only" supporting threading). Until then, there is more work to do.

#HPC #SLURM #Snakemake #SnakemakeHackathon2026 #ReproducibleComputing #OpenScience

Did you know? We started to gather cluster profiles for Snakemake users in a little repo: https://github.com/snakemake/snakemake-cluster-profiles

They should serve as a template for others.

#HPC #Snakemake #ReproducibleComputing #SnakemakeHackathon2026

2/3 accomplished ✅

Now, for a release a dependency has to be released with my two fixes, which have already been merged. Then, the world will see a new #nanopub application.

A #SnakemakeHackathon2026 result.

RE: https://fediscience.org/@snakemake/116245074392817916

Hm, I thought(!) that I fixed some issues with this bot. Appears not to be the case. Moreover, the `**docs** should be bold, not ignored. 🤦

Any way, this release is yet another leftover from the #SnakemakeHackathon2026 !

Personally, the week in Munich at the #SnakemakeHackathon2026 was really neat. I met friends and acquaintances, took the time to meet an old friend of mine not working in academia any more and the wonderful @FrankSonntag from our #FediScience association.

And now, tired, on my way back. Thanks to the railway service in Germany, I enjoy some boredom and the opportunity to do something else. Even reading a disc world novel (which is not "novel" any more) does not help.

#lifeis2short

I learned, that I am the first to write a reporter plugin, which is part of the #Snakemake organization.

That will change. @fbartusch is working on an #ROCrate plugin. Yours truly is working on a #nanopub plugin. Both will ease publishing workflow analysis metadata and making our computing a bit more transparent. Our motivation? Well, did you ever read a data analysis paper (e.g. from an #Bioinformatis group) recently? See?

#SnakemakeHackathon2026

Other things worth mentioning? My list is biased, but I liked the demo for
- a new beautiful logging interface, which might become the default at some time (incl. colour schemes which will help colour-blind colleagues).
- an improved way to plot workflow DAGs.

For both visually appealing changes, I hope very much to post some screenshot, soon.

#SnakemakeHackathon2026

Which new features did I like the most?

Well, there are so many, it merits a preprint for which @egonw already lend tremendous support.

Anyway, here are my favourites:
- #Snakemake tracks all #metadata during the workflow execution. This caused many(!) file access requrests. Now, we have a SQLite-DB for that purpose, thereby lifting quite some overhead.
- containerizing workflows to a Dockerfile was possible for long time. With "--containerize apptainer" there is direct support for #apptainer
- when a workflow is aborted abruptly it cannot delete its lockfile. Now running `--unlock` will not require calculating the DAG any more.

Oh, there is much, much more. But the Changelog is already linked.

#SnakemakeHackathon2026

As for the little executor plugin for the #SLURM batch system (for which I promised a release supporting array job support) ... Well, only a little bug fix release could be accomplished: https://github.com/snakemake/snakemake-executor-plugin-slurm/releases/tag/v2.5.4

Unfortunately, I wanted to use the common #Snakemake logo without the letters "#HPC" and missed one entry. So our announcement bot did not work.

Anyway, a faulty file system connection kept me from debugging the new feature. Stay tuned. It is almost ready.

#SnakemakeHackathon2026

We were a colourful and “scientifically diverse” group of bioinformaticians,, physicists, engineers ...

It is my hope, particularly, that with the introduction of more and more features the requirements of the #ATLAS experiment of #CERN will be met and deployment of our workflows will ease computation on #HPC clusters around the world

#SnakemakeHackathon2026

RE: https://fediscience.org/@snakemake/116222696140712833

What a week at the #SnakemakeHackathon2026 !

What a wonderful week with wonderful people!

We were pretty productive and this #Snakemake release is just the peak of it. The list of features, bug fixes, performance improvement and additional documentation is so long — our little announcement robot cannot display it all. Even here on FediiScience with its 1500-character limit!

#ReproducibleComputing #OpenScience

Finally, some personal progress: Thanks to @fbartusch a bug of the #SLURM executor plugin for Snakemake was fixed (dealing with nested quoting). A release is upcoming.

And: I generated my first (still faulty) test #nanopub from Snakemake 🥳

#SnakemakeHackathon2026

good luck to the hackers at #SnakemakeHackathon2026 !

The #SnakemakeHackathon2026 just started! Lots of interesting topics. If you are curious: #SnakemakHackathon or #SnakemakeHackathon2026 are your hashtags to follow. 😉

#reproducibleComputing #OpenScience #Datananalysis

This cannot be:

I am trying to compile a few stats for the #Snakemake executor plugin for #SLURM on #HPC systems. Preparing for a lighting talk at the #SnakemakeHackathon2026

PyPi: 20,000 downloads last month
BioConda: > 60,000 total (aggregated over all versions)

Impressive as it might be, this is contradictory. PyPi would exceed BioConda by a huge margin.

Does anyone know how to get all-time statistics from either platform? #BioConda or #PyPi?

What do you see here? This is an example knowledge graph describing a #Snakemake analysis workflow. You see the workflow description, a linked data set and a linked report.

All work done to boost #HPC user support for those conducting their workflows on HPC systems (you can run Snakemake on other platforms, too).

My to-do list:
- an assertion template for workflows: ✅
- another for reports: ✅ (simple datasets are already in the #nanopub verse)
- a plugin to gather software metadata and publish as a nanopub ❌ (half done: #SnakemakeHackathon2026 )

Kudos to @nanopub / @tkuhn and @johanneskoester - without them this pursuit would (have been) futile! And my feeling is that @fbartusch will play an important role in any further development ...

#OpenScience #ReproducibleComputing

The #Snakemake plugin for #SLURM on #HPC clusters will support JobArrays, soon:

1057691_1 2dcf44cc-+ rule_map_reads_wild+ 32 COMPLETED 0:0
1057691_2 2dcf44cc-+ 32 RUNNING 0:0
1057691_3 2dcf44cc-+ 32 RUNNING 0:0
1057691_4 2dcf44cc-+ 32 RUNNING 0:0
1057691_5 2dcf44cc-+ 32 RUNNING 0:0
1057691_6 2dcf44cc-+ 32 RUNNING 0:0

Hope to do more during next week's #SnakemakeHackathon2026 / #SnakemakeHackathon

All in all, the last year has been eventful. Not so much new functionality, but accounting for LOTS of exceptions on various clusters.

It is time to move forward at the #SnakemakeHackathon2026 !

Well, I did a release today:

https://fediscience.org/@snakemake/116097371050966872

And had a wonderful, productive discussion with co-developers. Looking eagerly forward to the #SnakemakeHackathon2026!