#dask — Public Fediverse posts

Собираем ML-платформу на базе Kubernetes: Yandex Cloud, JupyterHub, Dask и S3

Привет! Я Алиса, DevOps-инженер в KTS . В этой статье я расскажу об одном из наших недавних проектов, на котором мы строили инфраструктуру для команды дата-инженеров и аналитиков. Сразу оговорюсь, что это была не платформа для инференса Production-моделей, а именно полигон для исследований. В общем, делюсь практическим опытом построения масштабируемой инфраструктуры с автоскейлингом. Если тема вам актуальна — приглашаю к прочтению.

https://habr.com/ru/companies/kts/articles/1021976/

#yandex_cloud #jupyterhub #dask #s3 #kubernetes #ML #MLOps #devops

Восемь высокопроизводительных Python-библиотек в копилку разработчикам

Когда в 1991 году Гвидо ван Россум представил миру Python, никто не мог предсказать, какое место через несколько десятилетий этот язык займет в веб-разработке, Data Science и Machine Learning. Сейчас Python продолжает развиваться: с новым поколением инструментов в прошлое уходят традиционные ограничения — производительность, GIL и сложность параллельных вычислений. Привет, Хабр! С вами Леша Жиряков, я руковожу бэкенд-направлением витрины KION, возглавляю гильдию по Python и пишу для блога MWS на Хабре. Я каждый день сталкиваюсь с вызовами высоконагруженных систем и сформировался пул инструментов, которые помогают решать критические проблемы современной разработки — от обработки данных с Polars до управления зависимостями с UV. В этом материале я сделаю обзор Python-библиотек, с которыми можно создавать системы, сравнимые по производительности с Go и Rust.

https://habr.com/ru/companies/ru_mts/articles/968776/

#библиотеки #python #fastapi #litestar #polars #httpx #dask #Pydantic_V2 #ruff #Pithon_UV

Questa settimana ho fatto la-due-giorni-a-Bologna 🚀🚀

Intensa di #talk ispiranti, #gadget bellissimi … e le persone hanno reso tutto davvero indimenticabile. ✨

Conosco i retroscena dell’organizzazione e i ragazzi del @grusp hanno resto tutto perfetto, leggero e spensierato .. sebbene non fosse per nulla facile 💪

E’ sempre un piacere essere accettati come #speaker ai loro eventi ❤️

Alla prossima !

#DataAnalysis #Dask #Kubernetes #ParallelComputing #Scalability #AWS #DevSecOpsDay #ContainerDay

Siamo pronti per gli eventi del @grusp ?

Parlerò al #ContainerDay25 tra #SoftwareArchitect, #CloudEngineer e #PlatformSpecialist .. un po' #outsider ?

Tutt’altro: i dati oggi viaggiano in produzione, dentro cluster orchestrati e hanno bisogno di essere scalati, gestiti, monitorati proprio come il resto del software !

📍 Ci vediamo a #Bologna !

#ContainerDay #cluster #dask #python #YARN #kubernetes

Ray and Dask are Python libraries that help data scientists work faster with parallel processing. Dask excels at scalable data analysis with familiar pandas-like syntax, perfect for large datasets and ETL tasks. Ray shines in distributed ML training, hyperparameter tuning and model serving with built-in libraries like Ray Tune and Ray Serve. Choose Dask for data processing; Ray for ML pipelines. #DataScience #Python #MachineLearning #BigData #Ray #Dask #DataProcessing #ML https://www.kdnuggets.com/ray-or-dask-a-practical-guide-for-data-scientists

Ray and Dask are Python libraries that help data scientists work faster with parallel processing. Dask excels at scalable data analysis with familiar pandas-like syntax, perfect for large datasets and ETL tasks. Ray shines in distributed ML training, hyperparameter tuning and model serving with built-in libraries like Ray Tune and Ray Serve. Choose Dask for data processing; Ray for ML pipelines. #DataScience #Python #MachineLearning #BigData #Ray #Dask #DataProcessing #ML https://www.kdnuggets.com/ray-or-dask-a-practical-guide-for-data-scientists

DASK ile Türk Reasürans arasındaki teknik işleticilik sözleşmesi yenilendi: Türk Reasürans'tan yapılan açıklamaya göre, Sigortacılık ve Özel Emeklilik Düzenleme ve Denetleme Kurumu'nun (SEDDK) 30 Temmuz tarihli kararı doğrultusunda, DASK ile Türk Reasürans arasındaki mevcut teknik işleticilik sözleşmesinin güncel koşullara uyarlanarak yenilenmesine karar verildi.

Bu kapsamda, 8 Ağustos… https://www.eshahaber.com.tr/haber/dask-ile-turk-reasurans-arasindaki-teknik-isleticilik-sozlesmesi-yenilendi-248399.html?utm_source=dlvr.it&utm_medium=mastodon EshaHaber.com.tr #DASK #TürkReasürans #sigortacılık #emeklilik #teknikSözleşme

DASK'ın hasar ödeme gücü 355 milyar liraya ulaştı: AA muhabirine değerlendirmelerde bulunan Koç, Türk Reasürans olarak, DASK'ın teknik işleticiliğini üstlendikleri 8 Ağustos 2020'den bu yana yürütülen çalışmalara ilişkin bilgi verdi.

Koç, 2022'de DASK Ana Veri Merkezi'ni, yedeği İstanbul'da kalmak üzere deprem riskinin daha az olduğu Ankara'ya taşıdıklarını, 6 Şubat Kahramanmaraş depremlerinin birkaç ay… https://www.eshahaber.com.tr/haber/dask-in-hasar-odeme-gucu-355-milyar-liraya-ulasti-226097.html?utm_source=dlvr.it&utm_medium=mastodon EshaHaber.com.tr #DASK #deprem #sigorta #ekonomi #reasürans

Thank you to #Kone & Mai and Tor Nessling Foundations for supporting this work. A quantitative work like this would not be possible without a robust suite of FOSS tools. My thanks to the maintainers of #QGIS, #pandas, #geopandas, #duckdb, #dask, #statsmodels, #jupyter and many more!

𝗚𝗲𝗼𝘀𝗽𝗮𝘁𝗶𝗮𝗹 𝗣𝘆𝘁𝗵𝗼𝗻 𝗧𝘂𝘁𝗼𝗿𝗶𝗮𝗹𝘀
SpatialThoughts provides tutorials which cover a broad range of geospatial topics and technologies, e.g., #GeoPandas, #XArray, #dask, and more. Each technology is described in a notebook with step-by-step explanation. Check it out.
https://www.geopythontutorials.com

𝗚𝗲𝗼𝘀𝗽𝗮𝘁𝗶𝗮𝗹 𝗣𝘆𝘁𝗵𝗼𝗻 𝗧𝘂𝘁𝗼𝗿𝗶𝗮𝗹𝘀
SpatialThoughts provides tutorials which cover a broad range of geospatial topics and technologies, e.g., #GeoPandas, #XArray, #dask, and more. Each technology is described in a notebook with step-by-step explanation. Check it out.
https://www.geopythontutorials.com

#DuckDB (and a tonne of RAM) have absolutely saved my behind these last few months while dealing with huge biological datasets.

If you do any #data munging at all on a daily basis, well worth picking up DuckDB. Don't let the DB part fool you, it's more like #Dask or #Spark but #SQL .

https://duckdb.org/

#DuckDB (and a tonne of RAM) have absolutely saved my behind these last few months while dealing with huge biological datasets.

If you do any #data munging at all on a daily basis, well worth picking up DuckDB. Don't let the DB part fool you, it's more like #Dask or #Spark but #SQL .

https://duckdb.org/

News Haber EshaHaber Türkiye Sigorta, deprem bölgesinde 6,5 milyar lira hasar ödemesi yaptı: AA muhabirine açıklamalarda bulunan Çakmak, doğal afetlerle mücadelede en etkili yolun toplumsal bilincin güçlendirilmesi olduğunu söyledi.

Çakmak, Türkiye'nin büyük bir bölümünün aktif fay hatları üzerinde olduğunu anımsatarak, "Bu durum bizlere her zaman doğal afetlere karşı hazırlıklı ve tedbirli olmamız… https://www.eshahaber.com.tr/haber/turkiye-sigorta-deprem-bolgesinde-6-5-milyar-lira-hasar-odemesi-yapti-203294.html?utm_source=dlvr.it&utm_medium=mastodon EshaHaber.com.tr #deprem #sigorta #Türkiye #afet #DASK

News Haber EshaHaber Türkiye Sigorta, deprem bölgesinde 6,5 milyar lira ödeme yaptı: Türkiye Sigorta Genel Müdürü Çakmak, doğal afetlerle mücadelede en etkili yolun toplumsal bilincin güçlendirilmesi olduğunu söyledi.

Çakmak, Türkiye'nin büyük bir bölümünün aktif fay hatları üzerinde olduğunu anımsatarak, "Bu durum bizlere her zaman doğal afetlere karşı hazırlıklı ve tedbirli olmamız gerektiğini… https://www.eshahaber.com.tr/haber/turkiye-sigorta-deprem-bolgesinde-6-5-milyar-lira-odeme-yapti-203257.html?utm_source=dlvr.it&utm_medium=mastodon EshaHaber.com.tr #TürkiyeSigorta #Deprem #Sigorta #DASK #AfetBilinci

News Haber EshaHaber DASK, "asrın felaketi" sonrası yaklaşık 40 milyar lira hasar ödemesi yaptı: Türkiye’nin kalbinde derin izler bırakan ve yüzyılın felaketi olarak kayıtlara geçen 6 Şubat Kahramanmaraş merkezli depremlerin üzerinden 2 yıl geçti.

Bu sürede Türkiye genelinde sağlanan birlik ve beraberlik hareketi ile sarsıntının etkilediği 11 ilin yeniden ayağa kaldırılması için çalışmalar devam… https://www.eshahaber.com.tr/haber/dask-asrin-felaketi-sonrasi-yaklasik-40-milyar-lira-hasar-odemesi-yapti-202541.html?utm_source=dlvr.it&utm_medium=mastodon EshaHaber.com.tr #deprem #Kahramanmaraş #DASK #sigorta #afet

Det blir som sagt ingen metakalender i år, så denne honningkakeoppskriften kan dere nyte uten tanke på sesongen.

https://youtu.be/-ETiobx0g8U?si=uroFSoldiAz7aGVe

#kalender #advent #dask #honningkake

State of my toolchain 2024

An overview of the software, hardware and other personal productivity toolchains I’m using as of December 2024.

https://blog.kathyreid.id.au/2024/12/08/state-of-my-toolchain-2024/

Dask для анализа временных рядов

Привет, Хабр! Сегодня расскажем, как с помощью Dask можно анализировать временные ряды. С временными рядами всегда заморочек много: большие данные, сложные расчеты. Но Dask отлично с этим справляется.

https://habr.com/ru/companies/otus/articles/855408/

#dask #временные_ряды

Dear #gis users and #gischat . I just wrote my first (real) post on my block. I tried to learn some modern frameworks. Therefore, I compared the execution speed for the Intersection for the buildings of a whole German state with their parcels and land usage. I compared #Geopandas #duckdb #apachesedona and #dask GeoPandas.

Sedona and Dask-GeoPandas were the fastest. DuckDB's had some problems. Btw.: DuckDB did have the smallest memory footprint.
Here is the entry: https://sehheiden.github.io/posts/speed_comparision_gis_intersection/

I am moving all my computing libraries to #xarray, no regrets. It is a natural way to manipulate datasets of rectangular arrays, with named coordinates and dimensions: https://xarray.dev/
There are several possible backends, including #dask which allows lazy data loading.
I had the pleasure of meeting some of the devs last week, who showed me a preview of the upcoming `DataTree` structure which is going to make this library even more versatile!

#Python #numpy #ScientificComputing

[Перевод] Уроки, извлеченные из масштабирования до многотерабайтных датасетов

В этой статье я расскажу об уроках, которые вынес при работе с многотерабайтными датасетами. Объясню, с какими сложностями столкнулся при увеличении масштабов датасета и как их удалось решить. Я разделил статью на две части: первая посвящена масштабированию на отдельной машине, вторая — масштабированию на множестве машин. Наша цель — максимизировать доступные ресурсы и как можно быстрее выполнить поставленные задачи.

https://habr.com/ru/companies/magnus-tech/articles/834506/

#датасеты #big_data #joblib #машинное+обучение #параллелизация #spark #dask #виртуализация #инстансы #виртуальная_машина

Spent the morning playing with pystac-client and Dask. It's interesting for small areas but I still need to figure out how to scale it when working with huge extents.

#python #stac #remotesensing #dask #jalisco #mexico #laprimavera #incendios #wildfires

Spent the morning playing with pystac-client and Dask. It's interesting for small areas but I still need to figure out how to scale it when working with huge extents.

#python #stac #remotesensing #dask #jalisco #mexico #laprimavera #incendios #wildfires

If, like me, you are confused about all the terminology around geospatial cloud computing and how it all fits together, I recommend this video. Great explanation! #STAC, #COG, #Zarr, #Dask, #AWS, #EarthEngine
https://www.youtube.com/watch?v=YPno-89l54Q

If, like me, you are confused about all the terminology around geospatial cloud computing and how it all fits together, I recommend this video. Great explanation! #STAC, #COG, #Zarr, #Dask, #AWS, #EarthEngine
https://www.youtube.com/watch?v=YPno-89l54Q

Client libraries are better when they have no API: https://csvbase.com/blog/7

#CSV #pandas #polars #dask #curl #fsspec

Client libraries are better when they have no API: https://csvbase.com/blog/7

#CSV #pandas #polars #dask #curl #fsspec

Me: Groks #dask, teaches herself @matplotlib pretty fluent in @pandas_dev and #python.

Also me: signs an index incorrectly, spends 2 hours debugging a list index out of range error before spotting it 🙃

Now imagine how this #scales with tools like #Copilot and #GenerativeAI #coding tools ...

As part of my #PhD work, I recently had to perform computation on two very large files using @pandas_dev and I turned to #dask - a set of libraries on top of #pandas, aimed at scaling #python workloads from the laptop to the cluster.

Here's what I learned!

https://blog.kathyreid.id.au/2024/01/27/scaling-python-dask/

#DataScience #jupyter #ParallelProcessing

Good morning folks! It's been a while since I did one of my #TwitterMigration #Introduction #ConnectionList posts where I curate interesting people for you to follow on the #Fediverse :fediverse:

Today, I'd like you to meet:

@LMonteroSantos Lola is a #PhD #researcher at #EUI interested in #data #regulation, digital #economy and #AntiTrust, passionate about #DataScience and #programming. New to Mastodon, please make welcome 👋 🇪🇺

@danlockton is a #Professor at @TUEindhoven where he works in #design, #imagination and #climate #futures. He often posts interesting things around co-design and #collaboration 🇳🇱

@1sabelR is a #researcher @ANUResearch where she is into #SolarPunk and @scicomm 🇦🇺 She co-hosts the #SciBurst #podcast - worth a listen!

@timrichards is a #travel #writer based in #Naarm / #Melbourne in Australia, specialising in #rail 🇦🇺

@microstevens is a #DataScience facilitator at #UWMadison and she works in #OpenScience and #genomics 💻 🧬

@mrocklin does amazing things with #dask in #python, and I am very grateful in recent weeks for his posts and #StackOverflow responses. Thank you 🙏 🐍

@everythingopen is Australia's premier open #technology conference, covering #linux, #OpenSource, #OpenData, #OpenGov, #OpenGLAM, #OpenScience and everything else open. You should check it out! 🐧 🇦🇺

That's all for today - don't forget to share your own lists so we can more richly connect the :fediverse: and curate the conversations we want to have ❤️

My #dask coding worked and I got my data! I have been trying to get this data for three weeks 😎

Today's job is to manually validate it.

Random #research idea while babysitting a #dask process:

- I wonder if there's a way to save a bunch of terms of service documents to be able to version them, and show how they have changed over time, particularly in respect to arbitration, copyright and other #datafication processes?

Why yes I partitioned a 1Gb file into 2000 partitions with #dask, why do you ask?