#snakemake — Public Fediverse posts

Today? Among other things:

- wrote a looong mail for our new "PR" colleague. She asked wonderful questions about #HPC and the local research landscape. Right to the point. I hope, I helped her.
- almost finished my talk about supporting #Snakemake for #HPC admins. Below is a little motivation slide.

Today-I-have-done post:

- signed up for the NHR conference in Paderborn, Germany (September), will be there for a day or two. Hope to see some lovely colleagues.
- was notified about the closure of a GitHub issue. Only then I followed up, announcing the SLURM job array support by #Snakemake's #SLURM executor plugin (from v2.6.0 on which has been released in March). There apparently is need for better information channels. We have this announcement bot @snakemake - but I do realize, there is a) work to be done to make it more versatile for the Snakemake ecosystem (which requires "spare" time, so perhaps next time, when being bored whilst recovering from some sickness ...). However, communicating features to issue writers is not scalable!
- made a doodle to gather colleagues for planning our courses (more on this, soon). Of course, the doodle was useless. One of us is going on holiday, so we will have the meeting before that. Not considering preferences. 🤷‍♂️
- worked quite a bit on teaching material. Different one than the last time. Also, no release, yet. Need to figure out how to do releases with codeberg ...

Oof: https://pepy.tech/search?q=snakemake-executor-plugin-slurm

That would be > 500 k via #PyPI and > 100 k via #Conda (got the number with `condastats`)

That, of course, includes all downloads over all versions and including CI usage.

However, there are ~5,000 downloads per relevant version update for Conda alone. This fills me with pride and gives some ammunition for the "for admins" part of my #Snakemake #HPC tutorial. 😉

#SLURM

I feel that I have to be missing something about #SnakeMake or maybe I'm just on too old a version for compatibility with our cluster. But it really seems like the wildcard_constraints *cannot* have a negative match. Only positive matches.

Normal #Python regex can do this for sure. It just doesn't seem to work with rule wildcard_constraints.

Today-I-have-done:

- first login to the cluster I should work and teach on
- installed #Snakemake and the #SLURM plugin, both using #pixi Noticed that I need to amend the docs for this.
- installed the #Nanopub reporter plugin, too.
- merged a couple of dependency PRs (those are auto-generated) for the Nanopub plugin.
- Finished the onboarding part for the course which I intend to give. There still is no release.
- saw a notice from the Snakemake announcement bot account (https://fediscience.org/@snakemake/117037142309030490 ) and remembered, that I wanted to get rid of the serif font. Needs some tinkering. So, not this fix will not be there too soon, I'm afraid. @johanneskoester this is not forgotten!

Something, I have not done, but Johannes, is to set up the Fediwall for Snakemake (see https://snakemake.github.io/fediwall/ ) - for which I need to include the announcement robot, too.

(This series of posts is not a thread of posts - otherwise all Snakemake related stuff would not show up on the Fediwall.)

3/n

As of today, I will be starting a today-I-have-done series. Already tried before. Possibly with breaks, as I will not always find the time to post. Sometimes, I might sprinkle in a blog post or event announcement.

Next Monday I will start on a new position. And I have to commute. Possibly no time to take a break for posting on social media during working hours. Hence, the idea of posting little partially research-related summary posts is appealing to me.

Let me start:
- I tinkered with a #Fediwall for #Snakemake (https://cmeesters.github.io/Snakemake-Fediwall/). It will not be the final FediWall page as this is merely an experiment. But my idea is that as science communicators in the Fediverse we have to provide information proactively and not wait for demand to magically develop. So, this is a first step. (We cannot expect everyone to have a fable for social media - this will work for anyone without an account, too.)
- I did a few updates to the #HPC teaching material for Snakemake. We had a little "alliance" of like-minded people developing this material. A few drop-outs and a pretty LaTeX-oriented and obfuscated development environment and it was basically me. But I know there is some interest and I hope to be able to revive this group. Also, essentially new courses are on the horizon already.
- Last not least, some work on the teaching material for Snakemake #nanopub reporter plugin.

There are no releases, yet.

1/n

@rupdecat @yokofakun I’m not disagreeing, I’ve experimented with LLMs and #Snakemake quite a bit. Meh. For me though we have to accept we’ll see this more & more, and then I guess the most important things become persuading AIs to use best practice layouts, with good annotation and clarity so that humans can understand and quality control. QC is absolutely necessary. I wonder if it’s a documentation issue?

@yokofakun my preliminary answer: No! The last user I visited in my old function (two weeks ago) vibe coded 3 #Snakemake workflows instead of 1 which could do the work (with different parameters).

So, without background knowledge starting from scratch apparently #LLMs aren't great at figuring out a reliable approach. Also, it appeared to me that, possibly due to the training cycle, the workflows looked pretty old (as in: not considering any fairly recent feature).

(Only stumbled over your post by chance.)

#biostars Forum:AI for bioinformatics pipeline development. Is it a good idea? https://www.biostars.org/p/9621061/

#ai #bioinformatics #workflow #pipeline #snakemake #nextflow

It's tough sometimes trying to build an all-in-one #rstats #docker image so I can run pipelines with #snakemake and #singularity without relying on any local pkgs installed.

Run excruciatingly long build. Test a file. "Just kidding! You need this H5 reading package even though it isn't a dependency."

Repeat. Repeat.

I just gave up on updating a pipeline because it involves an entire external rust toolchain outside of R and calling it from R. That would have been nice to know an hour ago. 🙄

ReproHackathon à #COMPAS finit !

ça s'est bien passé : les artefacts avaient tous des problèmes ! 🙂 (même ceux qui utilisaient #Nix 😱)

Principalement du manque de doc et des problèmes de packaging

Les auteurs d'artefacts semblent contents des retours qu'ils ont eut donc parfait 🙂

Bon, les petits retours pour améliorer les artefacts :
- utiliser des gestionnaires de paquets fonctionnels ❄️ 🐮 (#nix #guix )
- utiliser des outils de workflow au lieu d'un master script bash obscure (#snakemake)
- utiliser des testbeds comme Grid'5000/SLICES-FR
- mettre son code sur Zenodo/Software Heritage

Je vais essayer d'écrire un petit rapport pour résumer tout ça (quand j'aurai du temps...)

RE: https://mas.to/@nanopub/116714551434180700

One talk will be from me about #Snakemake and creating #nanopub reports. See you!

Today I've done:

- accepted a review request (difficult to even see them these days in the wave of other mails). It is from some colleagues at a place where I have some acquaintances. None of the persons I know, so I am lucky for otherwise I would have declined. As the review is anonymous, and I know many people in the field, mentioning this here on Mastodon will not become an issue.
- followed an online meeting. Actually one of the non-boring ones.
- re-installed an environment I involuntarily screwed up yesterday when testing things for a user
- had a nice lunch with @KrawallHamster , @moschlar and others
- written a number of mails
- debugged code, debugged code, debugged code - will postpone a new #Snakemake plugin release for #SLURM to next week, when I can think straight. At least the CI pipeline is fine for this PR I was working on. But I always do live tests on an actual cluster if the change is not trivial.
- actually finished the review task (first round)

And I 🚴 , up- and downhill, through the May heat (this is a thing these days!!!). Lesson learned: Next time, I will take a break, sit on a bench, read and drink to have a rest for the last leg. The afternoon heat is no fun!

RE: https://mas.to/@nanopub/116619026431272841

A presentation about #Snakemake and software provenance with #nanopub by Yours Truthfully.

Of interest for those in the Snakemake workflow management system and citing workflow metadata in academic journals. Related to https://fediscience.org/@snakemake/116523521544819091 - oh, and I just realized a typo in the announcement bot message 😉

Today:

- written a blurb for my presentation at the upcoming #nanopub session
- release the #SLURM executor plugin for #Snakemake v2.7.0 - see https://fediscience.org/@snakemake/116617420491776431
- tried to mitigate the issue that TMOUT on an HPC login brings: sending SIGHUB to all detached multiplexers (so far no remedy and I tried a lot(!), don't send me tips).
- futile further debugging attempts. In the end it worked. Might result in a new release next week.

#academicchatter

... and two more, to get

- Snakemake workflow registrations: https://w3id.org/np/RApMWbZ0ixGj88tPB4cEamKJQuQTZ5Dxpjh1a5BWteTdE
- or Snakemake report registrations: https://w3id.org/np/RA8rPcBcreaZF01eA6Xeh_wwqQcM-Nz7xQyKbLhRHkCXA

just to make a start. Pretty crude, but a start towards better software provenance.

#nanopub #Snakemake

Just published a nanopublication to list all other nanopubs mentioning the Snakemake Workflow Management System: https://w3id.org/np/RAzeQbv1gXqKIuLic4xTP1gYRnL4YhU_5oRl62nbNl2Xk

Not yet sophisticated, but I am starting to like #SparQl 😉

#nanopub #Snakemake

RE: https://fediscience.org/@snakemake/116571962095785816

This little bit "performance improvements" lowered the number of file system access events for considerably! #Snakemake triggers many such events for keeping track of metadata. Which is important, but may cause some delays due to file system overhead - particularly on parallel and/or network file systems. The feature to outsource parts of this to sqlite was implemented during the #SnakemakeHackathon2026 . I hope, I can test the improvements next Monday!

#HPC #ReproducibleComputing

RE: https://fediscience.org/@snakemake/116523521544819091

Software provenance with #Snakemake: Using the reporter plugin for nanopublications, we can now get slightly improved nanopublications like this one: https://w3id.org/np/RAmgzfta63xx0wWc_zzQVm7kwOc4tsEOA0JJJCfsiLL1g (press on the little blue arrow on the right to see the full details). Automatically captured for this workflow: https://w3id.org/np/RAjHDlPDghZzc9ZvQ3uJQNJ9Jd_KAYzZt7dk5PXKgjRyE - again expressed a nanopub declaration. 😉

It now supports to capture the "classic" software support for #Conda and Snakemake wrappers.

There is more work to do. Let's see when and if I get to it.

#reproducibleComputing #softwareprovenance #nanopub

RE: https://fediscience.org/@rupdecat/116477850614860143

The #SnakemakeHacksthon2026 was a great experience. Learned so many things and also worked a bit on improving the #Snakemake ecosystem.
The Hackathon's summary preprint is just out, thanks to @rupdecat and all the others.
Looking forward to the 2027 edition.

RE: https://fediscience.org/@biohackrxiv/116477505622903440

Today, we published the summary (as a #BioHackrXiv preprint) of the #SnakemakeHackathon2026 . All the accomplishments which contribute once more to improve the #Snakemake "ecosystem" for #reproducibleResearch & #Dataanalysis .

Many thanks to @egonw from BioHackrXiv for helping us!