#snakemake — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #snakemake, aggregated by home.social.
-
RE: https://mas.to/@nanopub/116619026431272841
A presentation about #Snakemake and software provenance with #nanopub by Yours Truthfully.
Of interest for those in the Snakemake workflow management system and citing workflow metadata in academic journals. Related to https://fediscience.org/@snakemake/116523521544819091 - oh, and I just realized a typo in the announcement bot message 😉
-
Today:
- written a blurb for my presentation at the upcoming #nanopub session
- release the #SLURM executor plugin for #Snakemake v2.7.0 - see https://fediscience.org/@snakemake/116617420491776431
- tried to mitigate the issue that TMOUT on an HPC login brings: sending SIGHUB to all detached multiplexers (so far no remedy and I tried a lot(!), don't send me tips).
- futile further debugging attempts. In the end it worked. Might result in a new release next week. -
Today:
- written a blurb for my presentation at the upcoming #nanopub session
- release the #SLURM executor plugin for #Snakemake v2.7.0 - see https://fediscience.org/@snakemake/116617420491776431
- tried to mitigate the issue that TMOUT on an HPC login brings: sending SIGHUB to all detached multiplexers (so far no remedy and I tried a lot(!), don't send me tips).
- futile further debugging attempts. In the end it worked. Might result in a new release next week. -
Today:
- written a blurb for my presentation at the upcoming #nanopub session
- release the #SLURM executor plugin for #Snakemake v2.7.0 - see https://fediscience.org/@snakemake/116617420491776431
- tried to mitigate the issue that TMOUT on an HPC login brings: sending SIGHUB to all detached multiplexers (so far no remedy and I tried a lot(!), don't send me tips).
- futile further debugging attempts. In the end it worked. Might result in a new release next week. -
Today:
- written a blurb for my presentation at the upcoming #nanopub session
- release the #SLURM executor plugin for #Snakemake v2.7.0 - see https://fediscience.org/@snakemake/116617420491776431
- tried to mitigate the issue that TMOUT on an HPC login brings: sending SIGHUB to all detached multiplexers (so far no remedy and I tried a lot(!), don't send me tips).
- futile further debugging attempts. In the end it worked. Might result in a new release next week. -
Today:
- written a blurb for my presentation at the upcoming #nanopub session
- release the #SLURM executor plugin for #Snakemake v2.7.0 - see https://fediscience.org/@snakemake/116617420491776431
- tried to mitigate the issue that TMOUT on an HPC login brings: sending SIGHUB to all detached multiplexers (so far no remedy and I tried a lot(!), don't send me tips).
- futile further debugging attempts. In the end it worked. Might result in a new release next week. -
... and two more, to get
- Snakemake workflow registrations: https://w3id.org/np/RApMWbZ0ixGj88tPB4cEamKJQuQTZ5Dxpjh1a5BWteTdE
- or Snakemake report registrations: https://w3id.org/np/RA8rPcBcreaZF01eA6Xeh_wwqQcM-Nz7xQyKbLhRHkCXAjust to make a start. Pretty crude, but a start towards better software provenance.
-
Just published a nanopublication to list all other nanopubs mentioning the Snakemake Workflow Management System: https://w3id.org/np/RAzeQbv1gXqKIuLic4xTP1gYRnL4YhU_5oRl62nbNl2Xk
Not yet sophisticated, but I am starting to like #SparQl 😉
-
RE: https://fediscience.org/@snakemake/116571962095785816
This little bit "performance improvements" lowered the number of file system access events for considerably! #Snakemake triggers many such events for keeping track of metadata. Which is important, but may cause some delays due to file system overhead - particularly on parallel and/or network file systems. The feature to outsource parts of this to sqlite was implemented during the #SnakemakeHackathon2026 . I hope, I can test the improvements next Monday!
-
RE: https://fediscience.org/@snakemake/116571962095785816
This little bit "performance improvements" lowered the number of file system access events for considerably! #Snakemake triggers many such events for keeping track of metadata. Which is important, but may cause some delays due to file system overhead - particularly on parallel and/or network file systems. The feature to outsource parts of this to sqlite was implemented during the #SnakemakeHackathon2026 . I hope, I can test the improvements next Monday!
-
RE: https://fediscience.org/@snakemake/116571962095785816
This little bit "performance improvements" lowered the number of file system access events for considerably! #Snakemake triggers many such events for keeping track of metadata. Which is important, but may cause some delays due to file system overhead - particularly on parallel and/or network file systems. The feature to outsource parts of this to sqlite was implemented during the #SnakemakeHackathon2026 . I hope, I can test the improvements next Monday!
-
RE: https://fediscience.org/@snakemake/116571962095785816
This little bit "performance improvements" lowered the number of file system access events for considerably! #Snakemake triggers many such events for keeping track of metadata. Which is important, but may cause some delays due to file system overhead - particularly on parallel and/or network file systems. The feature to outsource parts of this to sqlite was implemented during the #SnakemakeHackathon2026 . I hope, I can test the improvements next Monday!
-
RE: https://fediscience.org/@snakemake/116571962095785816
This little bit "performance improvements" lowered the number of file system access events for considerably! #Snakemake triggers many such events for keeping track of metadata. Which is important, but may cause some delays due to file system overhead - particularly on parallel and/or network file systems. The feature to outsource parts of this to sqlite was implemented during the #SnakemakeHackathon2026 . I hope, I can test the improvements next Monday!
-
RE: https://fediscience.org/@snakemake/116523521544819091
Software provenance with #Snakemake: Using the reporter plugin for nanopublications, we can now get slightly improved nanopublications like this one: https://w3id.org/np/RAmgzfta63xx0wWc_zzQVm7kwOc4tsEOA0JJJCfsiLL1g (press on the little blue arrow on the right to see the full details). Automatically captured for this workflow: https://w3id.org/np/RAjHDlPDghZzc9ZvQ3uJQNJ9Jd_KAYzZt7dk5PXKgjRyE - again expressed a nanopub declaration. 😉
It now supports to capture the "classic" software support for #Conda and Snakemake wrappers.
There is more work to do. Let's see when and if I get to it.
-
RE: https://fediscience.org/@snakemake/116523521544819091
Software provenance with #Snakemake: Using the reporter plugin for nanopublications, we can now get slightly improved nanopublications like this one: https://w3id.org/np/RAmgzfta63xx0wWc_zzQVm7kwOc4tsEOA0JJJCfsiLL1g (press on the little blue arrow on the right to see the full details). Automatically captured for this workflow: https://w3id.org/np/RAjHDlPDghZzc9ZvQ3uJQNJ9Jd_KAYzZt7dk5PXKgjRyE - again expressed a nanopub declaration. 😉
It now supports to capture the "classic" software support for #Conda and Snakemake wrappers.
There is more work to do. Let's see when and if I get to it.
-
RE: https://fediscience.org/@snakemake/116523521544819091
Software provenance with #Snakemake: Using the reporter plugin for nanopublications, we can now get slightly improved nanopublications like this one: https://w3id.org/np/RAmgzfta63xx0wWc_zzQVm7kwOc4tsEOA0JJJCfsiLL1g (press on the little blue arrow on the right to see the full details). Automatically captured for this workflow: https://w3id.org/np/RAjHDlPDghZzc9ZvQ3uJQNJ9Jd_KAYzZt7dk5PXKgjRyE - again expressed a nanopub declaration. 😉
It now supports to capture the "classic" software support for #Conda and Snakemake wrappers.
There is more work to do. Let's see when and if I get to it.
-
RE: https://fediscience.org/@snakemake/116523521544819091
Software provenance with #Snakemake: Using the reporter plugin for nanopublications, we can now get slightly improved nanopublications like this one: https://w3id.org/np/RAmgzfta63xx0wWc_zzQVm7kwOc4tsEOA0JJJCfsiLL1g (press on the little blue arrow on the right to see the full details). Automatically captured for this workflow: https://w3id.org/np/RAjHDlPDghZzc9ZvQ3uJQNJ9Jd_KAYzZt7dk5PXKgjRyE - again expressed a nanopub declaration. 😉
It now supports to capture the "classic" software support for #Conda and Snakemake wrappers.
There is more work to do. Let's see when and if I get to it.
-
RE: https://fediscience.org/@snakemake/116523521544819091
Software provenance with #Snakemake: Using the reporter plugin for nanopublications, we can now get slightly improved nanopublications like this one: https://w3id.org/np/RAmgzfta63xx0wWc_zzQVm7kwOc4tsEOA0JJJCfsiLL1g (press on the little blue arrow on the right to see the full details). Automatically captured for this workflow: https://w3id.org/np/RAjHDlPDghZzc9ZvQ3uJQNJ9Jd_KAYzZt7dk5PXKgjRyE - again expressed a nanopub declaration. 😉
It now supports to capture the "classic" software support for #Conda and Snakemake wrappers.
There is more work to do. Let's see when and if I get to it.
-
RE: https://fediscience.org/@rupdecat/116477850614860143
The #SnakemakeHacksthon2026 was a great experience. Learned so many things and also worked a bit on improving the #Snakemake ecosystem.
The Hackathon's summary preprint is just out, thanks to @rupdecat and all the others.
Looking forward to the 2027 edition. -
RE: https://fediscience.org/@rupdecat/116477850614860143
The #SnakemakeHacksthon2026 was a great experience. Learned so many things and also worked a bit on improving the #Snakemake ecosystem.
The Hackathon's summary preprint is just out, thanks to @rupdecat and all the others.
Looking forward to the 2027 edition. -
RE: https://fediscience.org/@rupdecat/116477850614860143
The #SnakemakeHacksthon2026 was a great experience. Learned so many things and also worked a bit on improving the #Snakemake ecosystem.
The Hackathon's summary preprint is just out, thanks to @rupdecat and all the others.
Looking forward to the 2027 edition. -
RE: https://fediscience.org/@rupdecat/116477850614860143
The #SnakemakeHacksthon2026 was a great experience. Learned so many things and also worked a bit on improving the #Snakemake ecosystem.
The Hackathon's summary preprint is just out, thanks to @rupdecat and all the others.
Looking forward to the 2027 edition. -
RE: https://fediscience.org/@rupdecat/116477850614860143
The #SnakemakeHacksthon2026 was a great experience. Learned so many things and also worked a bit on improving the #Snakemake ecosystem.
The Hackathon's summary preprint is just out, thanks to @rupdecat and all the others.
Looking forward to the 2027 edition. -
RE: https://fediscience.org/@biohackrxiv/116477505622903440
Today, we published the summary (as a #BioHackrXiv preprint) of the #SnakemakeHackathon2026 . All the accomplishments which contribute once more to improve the #Snakemake "ecosystem" for #reproducibleResearch & #Dataanalysis .
Many thanks to @egonw from BioHackrXiv for helping us!
-
RE: https://fediscience.org/@biohackrxiv/116477505622903440
Today, we published the summary (as a #BioHackrXiv preprint) of the #SnakemakeHackathon2026 . All the accomplishments which contribute once more to improve the #Snakemake "ecosystem" for #reproducibleResearch & #Dataanalysis .
Many thanks to @egonw from BioHackrXiv for helping us!
-
RE: https://fediscience.org/@biohackrxiv/116477505622903440
Today, we published the summary (as a #BioHackrXiv preprint) of the #SnakemakeHackathon2026 . All the accomplishments which contribute once more to improve the #Snakemake "ecosystem" for #reproducibleResearch & #Dataanalysis .
Many thanks to @egonw from BioHackrXiv for helping us!
-
RE: https://fediscience.org/@biohackrxiv/116477505622903440
Today, we published the summary (as a #BioHackrXiv preprint) of the #SnakemakeHackathon2026 . All the accomplishments which contribute once more to improve the #Snakemake "ecosystem" for #reproducibleResearch & #Dataanalysis .
Many thanks to @egonw from BioHackrXiv for helping us!
-
RE: https://fediscience.org/@biohackrxiv/116477505622903440
Today, we published the summary (as a #BioHackrXiv preprint) of the #SnakemakeHackathon2026 . All the accomplishments which contribute once more to improve the #Snakemake "ecosystem" for #reproducibleResearch & #Dataanalysis .
Many thanks to @egonw from BioHackrXiv for helping us!
-
"Snakemake Hackathon 2026" https://doi.org/10.37044/osf.io/h6zqj_v1
"Nonetheless, the platform’s continued evolution faces several open challenges: improving core performance on heterogeneous high-performance-computing (HPC) resources, extending the plugin architecture for domain-specific extensions, and lowering the entry barrier for novice users while preserving full reproducibility. Here we report on the Snakemake Hackathon 2026, convened in Munich, Germany (9–13 March 2026) with more than 40 participants representing academia, industry, and national-level research infrastructure." https://index.biohackrxiv.org/2026/04/27/h6zqj.html
-
"Snakemake Hackathon 2026" https://doi.org/10.37044/osf.io/h6zqj_v1
"Nonetheless, the platform’s continued evolution faces several open challenges: improving core performance on heterogeneous high-performance-computing (HPC) resources, extending the plugin architecture for domain-specific extensions, and lowering the entry barrier for novice users while preserving full reproducibility. Here we report on the Snakemake Hackathon 2026, convened in Munich, Germany (9–13 March 2026) with more than 40 participants representing academia, industry, and national-level research infrastructure." https://index.biohackrxiv.org/2026/04/27/h6zqj.html
-
"Snakemake Hackathon 2026" https://doi.org/10.37044/osf.io/h6zqj_v1
"Nonetheless, the platform’s continued evolution faces several open challenges: improving core performance on heterogeneous high-performance-computing (HPC) resources, extending the plugin architecture for domain-specific extensions, and lowering the entry barrier for novice users while preserving full reproducibility. Here we report on the Snakemake Hackathon 2026, convened in Munich, Germany (9–13 March 2026) with more than 40 participants representing academia, industry, and national-level research infrastructure." https://index.biohackrxiv.org/2026/04/27/h6zqj.html
-
"Snakemake Hackathon 2026" https://doi.org/10.37044/osf.io/h6zqj_v1
"Nonetheless, the platform’s continued evolution faces several open challenges: improving core performance on heterogeneous high-performance-computing (HPC) resources, extending the plugin architecture for domain-specific extensions, and lowering the entry barrier for novice users while preserving full reproducibility. Here we report on the Snakemake Hackathon 2026, convened in Munich, Germany (9–13 March 2026) with more than 40 participants representing academia, industry, and national-level research infrastructure." https://index.biohackrxiv.org/2026/04/27/h6zqj.html
-
"Snakemake Hackathon 2026" https://doi.org/10.37044/osf.io/h6zqj_v1
"Nonetheless, the platform’s continued evolution faces several open challenges: improving core performance on heterogeneous high-performance-computing (HPC) resources, extending the plugin architecture for domain-specific extensions, and lowering the entry barrier for novice users while preserving full reproducibility. Here we report on the Snakemake Hackathon 2026, convened in Munich, Germany (9–13 March 2026) with more than 40 participants representing academia, industry, and national-level research infrastructure." https://index.biohackrxiv.org/2026/04/27/h6zqj.html
-
The #SnakemakeHackathon2026 has ended, we are still preparing our preprint release. But, our host has prepared a note on their homepage: https://go.tum.de/946236 🥳
-
The #SnakemakeHackathon2026 has ended, we are still preparing our preprint release. But, our host has prepared a note on their homepage: https://go.tum.de/946236 🥳
-
The #SnakemakeHackathon2026 has ended, we are still preparing our preprint release. But, our host has prepared a note on their homepage: https://go.tum.de/946236 🥳
-
The #SnakemakeHackathon2026 has ended, we are still preparing our preprint release. But, our host has prepared a note on their homepage: https://go.tum.de/946236 🥳
-
The #SnakemakeHackathon2026 has ended, we are still preparing our preprint release. But, our host has prepared a note on their homepage: https://go.tum.de/946236 🥳
-
RE: https://fediscience.org/@snakemake/116295568336688286
This is a big step forward: The SLURM plugin for Snakemake now supports so-called job arrays. These are cluster jobs, with ~ equal resource requirements in terms of memory and compute resources.
The change in itself was big: The purpose of a workflow system is to make use of the vast resources of an HPC cluster. Hence, jobs are submitted to run concurrently. However, for a job array, we have to "wait" for all eligible jobs to be ready. And then we submit.
To preserve concurrent execution of other jobs which are ready to be executed, a thread pool has been introduced. In itself, I do not see job arrays as such a big feature: The LSF system profited much more from arrays than the rather lean SLURM implementation does.
BUT: the new code base will ease further development to pooling many shared memory tasks (applications which support no parallel execution or are confined to one computer by "only" supporting threading). Until then, there is more work to do.
#HPC #SLURM #Snakemake #SnakemakeHackathon2026 #ReproducibleComputing #OpenScience
-
RE: https://fediscience.org/@snakemake/116295568336688286
This is a big step forward: The SLURM plugin for Snakemake now supports so-called job arrays. These are cluster jobs, with ~ equal resource requirements in terms of memory and compute resources.
The change in itself was big: The purpose of a workflow system is to make use of the vast resources of an HPC cluster. Hence, jobs are submitted to run concurrently. However, for a job array, we have to "wait" for all eligible jobs to be ready. And then we submit.
To preserve concurrent execution of other jobs which are ready to be executed, a thread pool has been introduced. In itself, I do not see job arrays as such a big feature: The LSF system profited much more from arrays than the rather lean SLURM implementation does.
BUT: the new code base will ease further development to pooling many shared memory tasks (applications which support no parallel execution or are confined to one computer by "only" supporting threading). Until then, there is more work to do.
#HPC #SLURM #Snakemake #SnakemakeHackathon2026 #ReproducibleComputing #OpenScience
-
RE: https://fediscience.org/@snakemake/116295568336688286
This is a big step forward: The SLURM plugin for Snakemake now supports so-called job arrays. These are cluster jobs, with ~ equal resource requirements in terms of memory and compute resources.
The change in itself was big: The purpose of a workflow system is to make use of the vast resources of an HPC cluster. Hence, jobs are submitted to run concurrently. However, for a job array, we have to "wait" for all eligible jobs to be ready. And then we submit.
To preserve concurrent execution of other jobs which are ready to be executed, a thread pool has been introduced. In itself, I do not see job arrays as such a big feature: The LSF system profited much more from arrays than the rather lean SLURM implementation does.
BUT: the new code base will ease further development to pooling many shared memory tasks (applications which support no parallel execution or are confined to one computer by "only" supporting threading). Until then, there is more work to do.
#HPC #SLURM #Snakemake #SnakemakeHackathon2026 #ReproducibleComputing #OpenScience
-
RE: https://fediscience.org/@snakemake/116295568336688286
This is a big step forward: The SLURM plugin for Snakemake now supports so-called job arrays. These are cluster jobs, with ~ equal resource requirements in terms of memory and compute resources.
The change in itself was big: The purpose of a workflow system is to make use of the vast resources of an HPC cluster. Hence, jobs are submitted to run concurrently. However, for a job array, we have to "wait" for all eligible jobs to be ready. And then we submit.
To preserve concurrent execution of other jobs which are ready to be executed, a thread pool has been introduced. In itself, I do not see job arrays as such a big feature: The LSF system profited much more from arrays than the rather lean SLURM implementation does.
BUT: the new code base will ease further development to pooling many shared memory tasks (applications which support no parallel execution or are confined to one computer by "only" supporting threading). Until then, there is more work to do.
#HPC #SLURM #Snakemake #SnakemakeHackathon2026 #ReproducibleComputing #OpenScience
-
RE: https://fediscience.org/@snakemake/116295568336688286
This is a big step forward: The SLURM plugin for Snakemake now supports so-called job arrays. These are cluster jobs, with ~ equal resource requirements in terms of memory and compute resources.
The change in itself was big: The purpose of a workflow system is to make use of the vast resources of an HPC cluster. Hence, jobs are submitted to run concurrently. However, for a job array, we have to "wait" for all eligible jobs to be ready. And then we submit.
To preserve concurrent execution of other jobs which are ready to be executed, a thread pool has been introduced. In itself, I do not see job arrays as such a big feature: The LSF system profited much more from arrays than the rather lean SLURM implementation does.
BUT: the new code base will ease further development to pooling many shared memory tasks (applications which support no parallel execution or are confined to one computer by "only" supporting threading). Until then, there is more work to do.
#HPC #SLURM #Snakemake #SnakemakeHackathon2026 #ReproducibleComputing #OpenScience
-
Did you know? We started to gather cluster profiles for Snakemake users in a little repo: https://github.com/snakemake/snakemake-cluster-profiles
They should serve as a template for others.
#HPC #Snakemake #ReproducibleComputing #SnakemakeHackathon2026
-
Did you know? We started to gather cluster profiles for Snakemake users in a little repo: https://github.com/snakemake/snakemake-cluster-profiles
They should serve as a template for others.
#HPC #Snakemake #ReproducibleComputing #SnakemakeHackathon2026
-
Did you know? We started to gather cluster profiles for Snakemake users in a little repo: https://github.com/snakemake/snakemake-cluster-profiles
They should serve as a template for others.
#HPC #Snakemake #ReproducibleComputing #SnakemakeHackathon2026
-
Did you know? We started to gather cluster profiles for Snakemake users in a little repo: https://github.com/snakemake/snakemake-cluster-profiles
They should serve as a template for others.
#HPC #Snakemake #ReproducibleComputing #SnakemakeHackathon2026
-
Did you know? We started to gather cluster profiles for Snakemake users in a little repo: https://github.com/snakemake/snakemake-cluster-profiles
They should serve as a template for others.
#HPC #Snakemake #ReproducibleComputing #SnakemakeHackathon2026
-
Geez I hate #LLM #AI showing up at the top of my search results. I was looking up the #SnakeMake command page to remind myself of the syntax for latency waiting for files.
I search. The first result is #AI generated and claims there is no latency command in #SnakeMake. The first real result was the command syntax page. Which detailed the --latency-wait command. Which has existed in the command list for a *LONG* time. 🙄
-
Geez I hate #LLM #AI showing up at the top of my search results. I was looking up the #SnakeMake command page to remind myself of the syntax for latency waiting for files.
I search. The first result is #AI generated and claims there is no latency command in #SnakeMake. The first real result was the command syntax page. Which detailed the --latency-wait command. Which has existed in the command list for a *LONG* time. 🙄