#parallelization — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #parallelization, aggregated by home.social.
-
This was extremely frustrating because parallelizing step A would bring runtime down from 10+ seconds to under 1 second (yes, it's a beefy machine), but it was definitely not worth it to do that if step C would then be some 10 times (if not more) slower, with low CPU usage to boot.
I have not investigated in detail the reasons for this massive slowdown. I suspect step C in general was slow because of (parallel and) dynamic memory allocation to build the per-G-element lists plus the need for each element to run over the whole L list to find the relevant elements.
I still don't know why it would become slower when the L was generated in parallel, but I actually found a solution that makes step C much simpler: sorting L by the second index in the tuples (the j) makes sure that the list can be trivially split (each section assigned to the proper j) simply by cutting it up at the boundaries when the j changes.
Sorting itself is extremely fast even on a list with millions of elements, and with this change step C takes less than a second *without parallelization*. It's not even worth parallelizing it anymore (in contrast to step A that does significantly benefit from it).
2/2
-
I had a weird experience with #Julia and #parallelization.
I had to collect all the pairs of items in two sets (S and G) that satisfied a particular relation function R. For Reasons™, it's easier to find these pairs looking at each element of S and find which elements of G are in relation with it, but this information is better used in reverse (i.e. on use we want, for any given element of G, find which elements of S are in relation with it.
So my idea was to build a set of tuples (i, j, r) where i is the index of an element of S, j the index of an element of G, and r the result of applying R to these two elements (which gives a value I need “on use” of this information I'm collecting).
To build this set of tuples, I went with (step A) a function mapping each index i to the list of tuples where the relation was possible, and then (step B) merge the lists together into a single list L.
Then (step C) for each index j of G, the approach was to find_all elements in L in which the second index was j, and thus have for each element of G the elements of S it was in relation with, including the relation function value, just as I wanted.
My original implementation was SLOW. Step A would take some 20 seconds, which I was able to bring down to 10 with optimizations, but step C, which would take between 30 and 130 second *using parallelization*.
The worst of it? I could trivially parallelize step A, but in this case step C would take FOREVER (stopped waiting after 15 minutes).
1/n
-
Ah, another #GitHub wonder 🥱: #Forkrun claims to be the turbocharged, NUMA-aware, bash-native parallelizer we've all been waiting for 🎉... because dealing with threading complexities wasn't hard enough already, right? 🤦♂️ Just what we needed—another inscrutable tool promising to revolutionize workflows, while managing to bewilder mere mortals. 🚀
https://github.com/jkool702/forkrun #NUMA #parallelization #workflow #tools #tech #news #HackerNews #ngated -
#fura-utils
added #parallelization to the #opus conversion #bash script, and created a #flac conversion one,
you can find them as `fura-2opus` and `fura-2flac`, have fun mass converting! 😉
https://github.com/FraYoshi/fura-utils/commit/403e13e9fc7684f2552e7e0c2b059970f11b4e2coh, don't forget to install `parallel`, it is now a requirement for this script to work!
-
There was something interesting going on on one of my systems:
If a certain function in #Python was called as a separate process with the #multiprocessing library, then the sort_values function of #Pandas would just hang (and therefore the process would never produce the output I was waiting for). Called from the main process was OK.
The solution was to change the sorting algorithm by the `kind="stable"` parameter. Weird.
-
Ah yes, because the world was desperately incomplete without a way to hash a 25-byte string in merely 68 clock cycles. 😴🔧 Meanwhile, the rest of us are still waiting for the riveting sequel where we parallelize the #parallelization of parallelizing. 🚀💼
https://www.controlpaths.com/2025/06/29/parallelizing_sha256-calculation-fpga/ #hashing #innovation #tech #humor #developer #life #HackerNews #ngated -
Ah, yes, because nothing says "cutting-edge tech" like juggling Git worktrees and #Tmux while your AI coding agent goes "brrr" 🙄. Truly groundbreaking stuff: discovering #parallelization in 2024 like it's a rare species. 🚀🔧
https://www.skeptrune.com/posts/git-worktrees-agents-and-tmux/ #cuttingEdgeTech #GitWorktrees #AICodingAgent #HackerNews #ngated -
LLM Codegen go Brrr – Parallelization with Git Worktrees and Tmux
https://www.skeptrune.com/posts/git-worktrees-agents-and-tmux/
#HackerNews #LLM #Codegen #Tmux #Git #Worktrees #Parallelization
-
🐲 Oh, look! Someone spent their free time rendering 27,000 #dragons and 10,000 lights on a GPU—because that's the absolute pinnacle of #productivity, right? 🤖 Just what the world needed: another article about high-performance #parallelization strategies that only 0.000001% of the population will pretend to understand! 😂
https://logdahl.net/p/gpu-driven #GPU #techhumor #HackerNews #ngated -
I'm excited to be heading back to an in-person conference! QA or the Highway is a week from today. There's still time to join us. Come hear me talk about #parallelization, #TestData, and #risk with our #SoftwareTesting #Automation and #DevOps. If you attend, please come say hi!
-
One billion rows in Go: Nice explication of optimization and profiling techniques
https://benhoyt.com/writings/go-1brc/
#parallelization #optimization #programming #hashtables #golang #go #+ -
my technical projects (underway) or interests, with hashtags, so findable:
book & blog series on software performance & scalability
#software #programming #performance #scalability #scaling #optimization #tuning #engineering #guides #HPC #BrendanGregg
my tech interests and favored tools:
#Go #Golang #Python #C #Linux #git #SQL #webdev #cloud #Internet #DistributedSystems #threading #concurrency #parallelization #EDA #queues #MessagePassing #CLI #TUI #curses #Terminals #consulting #Systems
-
'Let's Make Block Coordinate Descent Converge Faster: Faster Greedy Rules, Message-Passing, Active-Set Complexity, and Superlinear Convergence', by Julie Nutini, Issam Laradji, Mark Schmidt.
http://jmlr.org/papers/v23/18-045.html
#parallelization #sparse #optimization -
Good morning ☕️☕️
Can’t wait these build jobs to finish …
#Parallelization doesn’t always make everything faster -
This year's programming revelation for me was the #multiprocessing library #ray. Now that I've discovered how easily you can adapt existing code to make it scale up to any number of nodes, I can never go back to Python's built-in multiprocessing, especially considering all the limitations that have suddenly evaporated in a ray :mind_blown: Also, the interface is so dead simple, I can't believe it. #python #programming #DataScience #bigdata #scalability #parallelization #interfaces #ux
-
@hex
I've never had twins, but I'm sure they are. My sister's wife gave birth to twins a couple months before my sister gave birth to a single child. And it certainly seems like a lot of work from what I've seen.
#parallelization -
Speedy Scripts: Sandra Henry-Stocker shows you how the open source pa.sh tool can speed up complex scripts with parallelization https://www.fosslife.org/speed-linux-scripts-pash-tool #tools #ParallelProcessing #code #programming #performance #Linux #OpenSource #FOSS #parallelization
-
Sources for using OpenBlas multi-threading numerical libraries in CRAN R:
[1] https://csantill.github.io/RPerformanceWBLAS/ (including Benchmarks and HowTo for Linux)
[2] http://brettklamer.com/diversions/statistical/faster-blas-in-r/
[3] https://www.r-bloggers.com/why-is-r-slow-some-explanations-and-mklopenblas-setup-to-try-to-fix-this/
[4] https://github.com/xianyi/OpenBLAS (binary packages for Windows)
#rstats #cran #multicore #multithreading #parallelization #linux #windows #OpenBlas #BLAS #LAPACK
-
@rstats I would like to ask you about your experience with multi-threading and multi-core processors in R:
What is the current state of official CRAN R in using multi-threaded libraries to fully utilize multi-core processors? Were there any recent developments (on Linux and Windows)?
Now in 2020, how is the performance of CRAN R compared to Microsoft/Revolution R?#rstats #cran #multicore #multithreading #parallelization #linux #windows
-
I'm excited to be heading back to an in-person conference! QA or the Highway is a week from today. There's still time to join us. Come hear me talk about #parallelization, #TestData, and #risk with our #SoftwareTesting #Automation and #DevOps. If you attend, please come say hi!
-
I'm excited to be heading back to an in-person conference! QA or the Highway is a week from today. There's still time to join us. Come hear me talk about #parallelization, #TestData, and #risk with our #SoftwareTesting #Automation and #DevOps. If you attend, please come say hi!
-
I'm excited to be heading back to an in-person conference! QA or the Highway is a week from today. There's still time to join us. Come hear me talk about #parallelization, #TestData, and #risk with our #SoftwareTesting #Automation and #DevOps. If you attend, please come say hi!
-
I'm excited to be heading back to an in-person conference! QA or the Highway is a week from today. There's still time to join us. Come hear me talk about #parallelization, #TestData, and #risk with our #SoftwareTesting #Automation and #DevOps. If you attend, please come say hi!
-
my technical projects (underway) or interests, with hashtags, so findable:
book & blog series on software performance & scalability
#software #programming #performance #scalability #scaling #optimization #tuning #engineering #guides #HPC #BrendanGregg
my tech interests and favored tools:
#Go #Golang #Python #C #Linux #git #SQL #webdev #cloud #Internet #DistributedSystems #threading #concurrency #parallelization #EDA #queues #MessagePassing #CLI #TUI #curses #Terminals #consulting #Systems
-
my technical projects (underway) or interests, with hashtags, so findable:
book & blog series on software performance & scalability
#software #programming #performance #scalability #scaling #optimization #tuning #engineering #guides #HPC #BrendanGregg
my tech interests and favored tools:
#Go #Golang #Python #C #Linux #git #SQL #webdev #cloud #Internet #DistributedSystems #threading #concurrency #parallelization #EDA #queues #MessagePassing #CLI #TUI #curses #Terminals #consulting #Systems
-
my technical projects (underway) or interests, with hashtags, so findable:
book & blog series on software performance & scalability
#software #programming #performance #scalability #scaling #optimization #tuning #engineering #guides #HPC #BrendanGregg
my tech interests and favored tools:
#Go #Golang #Python #C #Linux #git #SQL #webdev #cloud #Internet #DistributedSystems #threading #concurrency #parallelization #EDA #queues #MessagePassing #CLI #TUI #curses #Terminals #consulting #Systems
-
#Introduction
I’m a #father, a #filmmaker, dabbler in #engineering of many types.
Technologically, my interests lie in the bleeding edge of possibility wherever I see it.
#nixos #nixflakes #foss
#haskell #GHC #dependentlytyped #dependenttypes #functionalprogramming
#plutus #cardano #smartcontracts #formalmethods #formalverification
#idris #swift #arduino #riscv #Rust
#decentralization #parallelization #biomimicry #buddhism #leftist
#cinelighting #cinematography
#anarchosyndicalism #science