#multiprocessing — Public Fediverse posts

If you upgraded to Ubuntu 26.04, have an old Python script that uses multiprocessing (with no threading), and "ps -ef | grep python" suddenly looks like a war and peace manuscript. In 2 minutes Claude will tell you the secret one line to make Python 3.14 behave like Python 3.12 did for you up until now.

Spoiler: multiprocessing.set_start_method('fork')

How does Claude know all these things?

#Python3 #Multiprocessing #Ubuntu26_04 #Claude

Нейросети, генетика и десктоп: как я построил микрофреймворк для обучения AI-агентов с неблокирующим GUI

Микрофреймворк для параллельного обучения AI-агентов в средах Gymnasium с графическим интерфейсом на wxPython. Решает классическую проблему «зависшего GUI» при длительном обучении нейросетей: вычисления вынесены в отдельные процессы-сервисы, а интерфейс остаётся полностью отзывчивым. Поддерживает плагинную систему для добавления новых сред, визуализацию прогресса (графики Matplotlib), генетический алгоритм обучения (нейроэволюцию через DEAP) и сборку в один .exe через PyInstaller с автоматическим CI/CD.

https://habr.com/ru/articles/1030208/

#python #нейросети #генетический_алгоритм #нейроэволюция #pytorch #wxpython #multiprocessing #микрофреймворк #desktop_приложение #gui

Разбор threading vs multiprocessing vs asyncio в Python

При работе с Python да и другими языками программирования часто возникает необходимость ускорения выполнения кода, масштабирования обработки данных или работы с большим количеством сетевых запросов. Именно в Python для решения этих задач существуют три базовых метода. Это: threading, multiprocessing и asyncio. На первый взгляд – механизмы схожие. Но при детальном разборе ясно, что они решают принципиально разные задачи, опираются на разные модели исполнения и обладают своими ограничениями. В статье расскажу об особенностях каждого метода – будет интересно и познавательно.

https://habr.com/ru/articles/991478/

#threading #multiprocessing #asyncio #Python #параллельность #конкурентность #CPUbound #event_loop #многопоточность

Very interesting read about how ~1970‘s C-based synchronous timeslice hardware architecture doesn‘t fit +2000’s networked GUI #multiprocessing anymore.

It‘s true, even if we live the paradigm change in #programming we need to bolt it on the previous stuff using async or dispatcher libs, message handlers, etc. and then we say „look mum, I made it event-driven and distributed“ 😅

It is even difficult to imagine an AS/400‘s memory model if you lived in a C landscape.

https://programmingsimplicity.substack.com/p/hardware-stockholm-syndrome

Very interesting read about how ~1970‘s C-based synchronous timeslice hardware architecture doesn‘t fit +2000’s networked GUI #multiprocessing anymore.

It‘s true, even if we live the paradigm change in #programming we need to bolt it on the previous stuff using async or dispatcher libs, message handlers, etc. and then we say „look mum, I made it event-driven and distributed“ 😅

It is even difficult to imagine an AS/400‘s memory model if you lived in a C landscape.

https://programmingsimplicity.substack.com/p/hardware-stockholm-syndrome

Мониторинг Celery. Pull-модель

В этой статье рассмотрим возможность получать метрики Celery непосредственно от самих воркеров, хитрости, на которые придётся пойти, чтобы решить эту задачу, и, самое главное, какие преимущества от этого можно получить по сравнению с классическим подходом к мониторингу Celery. Также продемонстрирую небольшой Django-проект и пример его конфигурации. Особое внимание будет уделено режиму мультипроцессинга и тому, как та или иная конфигурация запуска Celery будет влиять на сложность решения.

https://habr.com/ru/companies/domclick/articles/942584/

#celery #celery_worker #celery_flower #celery_beat #metrics #multiprocessing #multithreading #monitoring

There was something interesting going on on one of my systems:

If a certain function in #Python was called as a separate process with the #multiprocessing library, then the sort_values function of #Pandas would just hang (and therefore the process would never produce the output I was waiting for). Called from the main process was OK.

The solution was to change the sorting algorithm by the `kind="stable"` parameter. Weird.

#ArchLinux #Linux #programming #parallelization

There was something interesting going on on one of my systems:

If a certain function in #Python was called as a separate process with the #multiprocessing library, then the sort_values function of #Pandas would just hang (and therefore the process would never produce the output I was waiting for). Called from the main process was OK.

The solution was to change the sorting algorithm by the `kind="stable"` parameter. Weird.

#ArchLinux #Linux #programming #parallelization

Почему multiprocessing.Queue() тормозит и как обойти это с помощью shared_memory

Привет, Хабр! Вы запускаете многопроцессную задачу, кидаете данные в multiprocessing.Queue() , а потом вдруг замечаете... что всё тормозит. Муторно. Медленно. Местами прям отвратительно. Вы смотрите в монитор, на top, на htop, на код — и не понимаете: ну ведь должно же летать! А не летит.

https://habr.com/ru/companies/otus/articles/913200/

#python #multiprocessing #очередь #производительность #shared_memory #межпроцессное_взаимодействие #кольцевой_буфер

https://kitfucoda.medium.com/telegram-chatbots-evolution-decoupling-parallel-python-s-shared-state-for-clarity-e76880ce9b1f

My recent refactoring journey began with a take-home assignment's hidden pitfall: a hard-coded queue in settings.py. This seemingly small detail quickly grew into a global state headache, making testing and modularity a real challenge.

To tackle this, I focused on explicitly managing inter-process communication. The solution involved leveraging multiprocessing.Manager to centralize shared synchronization primitives, allowing for much cleaner dependency injection across parallel processes.

This refactoring delivered significant benefits. It drastically improved testability by eliminating the need for global patching. It also brought consistency and enhanced modularity, laying a robust foundation for future scalability.

It's been a valuable lesson, directly applicable to my own operational chatbot. While not a performance optimization, the structural clarity gained is immensely rewarding. I'm looking forward to diving deeper into the testing aspects of this setup soon!

#Python #Refactoring #Multiprocessing #CleanCode #SoftwareArchitecture #getfedihired #fedihire #OpenToWork

I've been working on a #space #visualization tool for our operators. It basically needs to always know, and be ready to plot, where every single one of 60k+ objects is down to millidegree/meter/second resolution just in case the sensor suddenly slews there

My own constraint is that it has to be 1) a single 2) #python executable because otherwise I'm not interested

Earlier this year, I found a great 30x faster technique for determining which #satellites are above the horizon. (In fact, it's far more general than that, but that's all the help it gives me to this problem.)

I also realized I could spawn a #multiprocessing child to do lookahead on data and then pass a huge #numpy array to my graphing process. (Investigated ~9 different ways, chose the best)

But there I was stuck.

At any given moment, there are ~4500 space objects above the horizon (at our latitude). Putting 4500 points with little persistence trails and labels and then updating all that at 1Hz let alone the 10Hz I'd like was taking too long, even using the amazing #pyqtgraph

So there I was stuck. Until this week.

I've been working on a #space #visualization tool for our operators. It basically needs to always know, and be ready to plot, where every single one of 60k+ objects is down to millidegree/meter/second resolution just in case the sensor suddenly slews there

My own constraint is that it has to be 1) a single 2) #python executable because otherwise I'm not interested

Earlier this year, I found a great 30x faster technique for determining which #satellites are above the horizon. (In fact, it's far more general than that, but that's all the help it gives me to this problem.)

I also realized I could spawn a #multiprocessing child to do lookahead on data and then pass a huge #numpy array to my graphing process. (Investigated ~9 different ways, chose the best)

But there I was stuck.

At any given moment, there are ~4500 space objects above the horizon (at our latitude). Putting 4500 points with little persistence trails and labels and then updating all that at 1Hz let alone the 10Hz I'd like was taking too long, even using the amazing #pyqtgraph

So there I was stuck. Until this week.

https://kitfucoda.medium.com/concurrency-vs-parallelism-achieving-scalability-with-processpoolexecutor-c366bfcc6207

Concurrency and parallelism are often confused in async programming discussions. Go's goroutines highlighted the difference: concurrency is doing many things at once, while parallelism is doing many things at the same time.

AsyncIO handles concurrency well for I/O, but CPU-bound tasks need parallelism. Python uses AsyncIO for concurrency, and ProcessPoolExecutor for parallelism, distributing work across CPU cores.

Process communication is harder than thread communication. AsyncIO's task cancellation differs from ProcessPoolExecutor's, requiring workarounds like event objects for reliable cancellation and shutdown.

Essentially, ProcessPoolExecutor enables parallelism for CPU-bound tasks, scaling them across multiple cores, while AsyncIO handles I/O concurrently.

#python #asyncio #concurrency #parallelism #multiprocessing #opentowork #getfedihired #fedihire #opentowork

https://kitfucoda.medium.com/concurrency-vs-parallelism-achieving-scalability-with-processpoolexecutor-c366bfcc6207

Concurrency and parallelism are often confused in async programming discussions. Go's goroutines highlighted the difference: concurrency is doing many things at once, while parallelism is doing many things at the same time.

AsyncIO handles concurrency well for I/O, but CPU-bound tasks need parallelism. Python uses AsyncIO for concurrency, and ProcessPoolExecutor for parallelism, distributing work across CPU cores.

Process communication is harder than thread communication. AsyncIO's task cancellation differs from ProcessPoolExecutor's, requiring workarounds like event objects for reliable cancellation and shutdown.

Essentially, ProcessPoolExecutor enables parallelism for CPU-bound tasks, scaling them across multiple cores, while AsyncIO handles I/O concurrently.

#python #asyncio #concurrency #parallelism #multiprocessing #opentowork #getfedihired #fedihire #opentowork

#question for #Python developers who do #multiprocessing

- What tips / tricks do you have for minimising the pickling / serialisation overhead?

- Are there any tools for #profiling what is being pickled?

- Bonus: what's your favourite Python profiler and why?

Параллельные вычисления, конкурентность и асинхронное программирование в Python: обзор для начинающих

Однопоточные приложения на Python ограничены в производительности: они выполняют задачи последовательно и не используют преимущества многоядерных процессоров. Кроме того, такие программы не справляются с обработкой множества операций одновременно, особенно если речь идет о задачах, связанных с вводом-выводом, например сетевыми запросами или чтением файлов. Производительность можно значительно улучшить, внедрив в код параллельные вычисления, конкурентность или асинхронное программирование. Для этого Python предлагает такие инструменты, как multiprocessing, threading и asyncio.

https://habr.com/ru/companies/skillfactory/articles/896160/

#python #multiprocessing #asyncio #threading

История эволюции веб-сервиса: от примера из доки до космолета

5k RPS, 5ms Latency и 100 экспериментов одновременно. История о том, как наша команда перестраивала веб-сервис для сплитования трафика в высокопроизводительную систему. С какими ограничениями Cpython и Gil столкнулись на пути, как обходили "узкие места" и оптимизировали сервис до микросекунд. В общем, всё о том, как мы построили "космолет" на Python и взлетели! Ну и, конечно же, ответ на вопрос: "Почему не Go? ".

https://habr.com/ru/companies/okko/articles/894462/

#python #multiprocessing #performance #okko #ab_testing

Как мы используем разделяемую память в Aqueduct

Привет. Меня зовут Денис Лисовик, я Backend-инженер в команде Data Science SWAT Авито. В этой статье рассказываю, как использовать разделяемую память в Aqueduct. Вместе мы шаг за шагом пройдем от сервиса, который едва держит один RPS, до сервиса, который может держать сотни запросов в секунду. В процессе вы узнаете, как использовать разделяемую память и как сделать так, чтобы она не утекала, а приложение не падало с Segmentation fault.

https://habr.com/ru/companies/avito/articles/866338/

#python #shared_memory #multiprocessing

@kentpitman @sigrid @mdhughes @awkravchuk

Featuring #unix_surrealism ! @prahou

@pkw ncurses #lisp #asdf #multiprocessing https://codeberg.org/pkw/open-borders

telnet lambda.moo.mud.org 8888
co guest
@join screwtape

#lambdaMOO !

I think @me bumped eir head on the resource limit in-MOO finally? Meet in paradise sushi!

You know it's time to stop when you have to implement __setstate__ and __getstate__ methods to exclude the boto3 s3 client from being serialized/deserialized by Pickle in multiprocessing.

#python #s3 #multiprocessing #pickle

You know it's time to stop when you have to implement __setstate__ and __getstate__ methods to exclude the boto3 s3 client from being serialized/deserialized by Pickle in multiprocessing.

#python #s3 #multiprocessing #pickle

I used to love #Python, but dealing with the honestly kind of scary #multiprocessing library was a reminder of the #GIL #threading situation really hurts working with #parallelism. 😰🫠

I've started learning #Kotlin after honestly really enjoying the threading library it provides. It's so's easy to work with when you understand it! 😀 (I would prefer to stay away from some of the #Java conventions after working with them for so long. 👍) #JVM #programming
♲ f.kawa-kun.com/display/881761a…

I used to love #Python, but dealing with the honestly kind of scary #multiprocessing library was a reminder of the #GIL #threading situation really hurts working with #parallelism. 😰🫠

I've started learning #Kotlin after honestly really enjoying the threading library it provides. It's so's easy to work with when you understand it! 😀 (I would prefer to stay away from some of the #Java conventions after working with them for so long. 👍) #JVM #programming
♲ f.kawa-kun.com/display/881761a…

Never, ever, use the fork context in multiprocessing. :blobfoxsweating: POSIX OSes (aside from macOS) have that as the default. :blobfoxangrylaugh: Always use the spawn context! :blobfoxdead:

Of course, unless you have an extreme edge case. :blobfoxgoogly: (Yes, the names are funny. :blobfoxgooglytrash: )

#Python #multiprocessing

Have you ever programmed a human computer? Having 30 people walking around the room to exchange information between RAM addresses and CPU registers, and human CPUs execute operations on the clock is a very special experience*.

This week I learned more than in a ~year of self-study, thanks to the 16th Advanced Scientific Programming in #Python https://aspp.school
We covered version control, packaging, testing, debugging, computer architecture, some #numpy and #pandas -fu, programming patterns aka what goes into a class and what doesn't, big-O to understand the scaling of various operations and how to find the fastest one for the given data type and size, and an intro to #multithreading and #multiprocessing 🍭

A personal highlight for me was pair programming. I never thought writing code in with a buddy would be so much fun, but I learned a lot from my buddies and now I don't want to go back to writing code alone 😅

Very indebted to the teachers and organizers; https://aspp.school/wiki/faculty if you ever meet one of those people, please buy them a drink for what they have done for a better code karma state in the universe

*our human computer didn't manage to execute the simplest sorting algorithm and the CPUs started to sweat; we experienced what happens when the code is ambiguous and imprecise 😱🫨

Have you ever programmed a human computer? Having 30 people walking around the room to exchange information between RAM addresses and CPU registers, and human CPUs execute operations on the clock is a very special experience*.

This week I learned more than in a ~year of self-study, thanks to the 16th Advanced Scientific Programming in #Python https://aspp.school
We covered version control, packaging, testing, debugging, computer architecture, some #numpy and #pandas -fu, programming patterns aka what goes into a class and what doesn't, big-O to understand the scaling of various operations and how to find the fastest one for the given data type and size, and an intro to #multithreading and #multiprocessing 🍭

A personal highlight for me was pair programming. I never thought writing code in with a buddy would be so much fun, but I learned a lot from my buddies and now I don't want to go back to writing code alone 😅

Very indebted to the teachers and organizers; https://aspp.school/wiki/faculty if you ever meet one of those people, please buy them a drink for what they have done for a better code karma state in the universe

*our human computer didn't manage to execute the simplest sorting algorithm and the CPUs started to sweat; we experienced what happens when the code is ambiguous and imprecise 😱🫨

Ускорение Python в 2 раза с помощью multiprocessing, async и MapReduce

Python действительно может считаться относительно медленным языком программирования по сравнению с некоторыми другими языками, такими как C++ или Java. Однако, существуют различные библиотеки и инструменты, которые позволяют ускорить выполнение счетных задач в Python. Рассмотрим как можно ускорить анализ данных в 2 раза!

https://habr.com/ru/articles/825206/

#python3 #python #asyncio #async/await #multiprocessing #mapreduce

Article: 10 times faster, running cases in parallel

In this article, we explore running optimization model cases in parallel. Specifically, we use the Python multiprocessing and mpi4py libraries to fully use the many CPU cores/threads in modern computers.

Our goals are to:
- Illustrate how to apply the multiprocessing and mpi4py libraries to running optimization model cases in parallel.
- Measure the performance of running cases in parallel compared with serially.
- Compare the performance of an old 4 core / 4 thread CPU with a new 20 core / 28 thread CPU, using the HiGHS solver.

https://www.solvermax.com/blog/10-times-faster-running-cases-in-parallel
#Python #pyomo #orms #optimization #modelling #HiGHS #multiprocessing #mpi4py

Article: 10 times faster, running cases in parallel

In this article, we explore running optimization model cases in parallel. Specifically, we use the Python multiprocessing and mpi4py libraries to fully use the many CPU cores/threads in modern computers.

Our goals are to:
- Illustrate how to apply the multiprocessing and mpi4py libraries to running optimization model cases in parallel.
- Measure the performance of running cases in parallel compared with serially.
- Compare the performance of an old 4 core / 4 thread CPU with a new 20 core / 28 thread CPU, using the HiGHS solver.

https://www.solvermax.com/blog/10-times-faster-running-cases-in-parallel
#Python #pyomo #orms #optimization #modelling #HiGHS #multiprocessing #mpi4py

Разница между pool.map и pool.map_async в Python

Еще одна статья-шпаргалка о модуле multiprocessing в Python, без воды, от новичка для новичков многопроцессорного программирования. pool.map и pool.map_async являются методами модуля multiprocessing.Pool в Python, которые позволяют параллельно выполнять функции на нескольких процессах.

https://habr.com/ru/articles/804917/

#multiprocessing #многопроцессорность

Модули multiprocessing и threading в Python

Данная статья написана новичком для новичков, т.е. для тех, кто только начинает изучать возможности многопроцессорного и многопоточного программирования в Python. Статья намеренно пишется без воды и со скомканной теорией, в стиле шпаргалки.

https://habr.com/ru/articles/804799/

#multiprocessing #multithreading

Как работает multiprocessing в Python под капотом

Я довольно давно пишу на Python и во многих проектах использовал multiprocessing — пакет стандартной библиотеки языка Python, который предоставляет интерфейс для работы с процессами, очередями, пулами процессов и многими другими удобными инструментами для параллельного программирования. В какой-то момент я понял, что мне не хватает более детального понимания работы этой библиотеки. Мне захотелось залезть в исходники multiprocessing, разобраться и заодно написать статью. Данная статья в основном рассчитана на новичков в Python и тех, кто хочет подробнее разобраться в том, как именно создаются процессы и пулы в Python и погрузиться в детали реализации.

https://habr.com/ru/articles/803607/

#python #multiprocessing #параллельное_программирование #процессы

Scientists unveil PC breakthrough that gives 2x speed without hardware upgrades https://www.tweaktown.com/news/96815/scientists-unveil-pc-breakthrough-that-gives-2x-speed-without-hardware-upgrades/index.html

#PCMR
#Multiprocessing
#breakthrough
#CPU

Weil eine API so langsam ist will ich jetzt Dinge parallelisieren. Ich will aber ein globales Objekt für eben diese API-Calls verwenden. Mit #multiprocessing.Pool() konnte ich nicht auf eine globale Variable zugreifen. Wie würdet ihr sowas in #Python parallelisieren?

#Python folks: I'm trying to pass a bound method to a #multiprocessing pool's map method. The bound method obviously has a reference to self, which is an instance of a class that inherits from a lot other classes. Somewhere in the hierarchy, there's an unpicklable object, therefore my call the pool.map fails with "TypeError: cannot pickle ... object".

How would you solve this? 🤔

Took a fun couple hours creating a function that does his slowest part as a service (not doing his work for him--we realized this needs to exist outside of his work) and then also making a parallelized helper for it.

Yep, 10-15x speedup using 16 cores. I think the operational machines have 32 or more cores, so this is great. Gets our final runtimes down to <1s (easy case) and <10s (hard case).

#python #multiprocessing #threads #software #space #orbitalmechanics

@the_curiostech I don't have a particular go-to. My frequent #multiprocessing use-case is embarassingly parallel so I use #python mp.Pool.map().

No rpc, no queues or locks or anything. Just "please blast this code across 10000 items and give me the results".

I find if I write "base code" any more complicated than that the bugs I encounter are too hard for my tiny brain.

I recently tried to scrape about 5k links off the internet with Python, for a personal project.

I didn't worry too much about optimizing it since it was pretty much a one-off thing, but after about half an hour of running the script, I was getting impatient.

So, I did what every red-blooded engineer would do: I avoided doing all the other things I should have been doing, and started looking for a *simple* way to speed up the process.

Enter pandarallel.

1/

#python #multiprocessing #pandas

🚀 Supercharge your #Python projects with Aiomultiprocess! Easily integrate #multiprocessing & #asyncio with this powerful library. Learn how through a real-world web scraping example. 🔥

📖 Read more:
https://qtalen.medium.com/aiomultiprocess-super-easy-integrate-multiprocessing-asyncio-in-python-2e883b65ba46
#aiomultiprocess #webdevelopment #webdev #programming #coding

Oh wow #pyhton threading lib is sooo nice. I used the #multiprocessing lib a while back and found it cumbersome. Parallelism in my existing Artifactory calls will be a snap with this one. I update metadata on docker manifests with quality gate data, deployment operator and deployment approver. When I did this we had less than related 20 images. We now have 60+ and it’s time to boost the efficiency a bit. This will be easy. I haven’t had to do much unit testing around threading so this should be fun

@joxean Can you instead collect the status/results from each worker as it finishes the tasks? That's the usual way to do it.

If you want realtime communication from a worker then you can use a Queue or a Pipe, depending on whether you need one way or two way messaging.

You set the Queue or Pipe object as part of the initialisation of the worker, with one end in the controller script and the other in the worker, and it abstracts away locking for you.

#Python #multiprocessing

@joxean what state do you need to share across workers with a Pool?

Can you just do the thing needing synchronization at the controller process, instead of in the workers? e.g. do expensive work in the workers, and print results as they come back, instead of in each worker.

Sometimes when I think I need to sync my workers I realise I can dedupe the workload upfront.

Can you share the code or problem to understand why you need locking?

#multiprocessing #python #concurrency