#airflow — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #airflow, aggregated by home.social.
-
FlatAdapter для преобразования иерархических структур
Превращаем иерархические структуры в набор плоских строк готовых для импорта в БД flat-adapter преобразует нетипизированные вложенные mapping в детерминированный список плоских строк. Библиотека умеет приводить scalar-типы, обрабатывать optional-поля, извлекать значения по путям, разворачивать вложенные FlatAdapter-ы и строить декартово произведение списков.
https://habr.com/ru/articles/1068172/
#etl #airflow #python #backend #data_compose #data_mining #dwh
-
Заставляем Airflow самого заводить задачи в YouTrack без смс и регистрации
Работая единственным QA в команде, в обязанности которого активно входит мониторинг, я столкнулась с проблемами тайм-менеджмента: очень много времени уходит на анализ упавших пайплайнов, заведение однотипных задач, поднятие тревоги, отмена тревоги, потому что проблема уже исправляется и прочие прелести тестерской работы. В один прекрасный осенний денек, сидя на окне и думая о нем (мониторинге), я решила - хватит это терпеть. Посидела, погуглила, вспомнила родительские слова “Хочешь сделать что-то хорошо - сделай это сама”. Что ж таков путь.
https://habr.com/ru/articles/1060944/
#airflow #apache_airflow #etl #listener #dag #dags #youtrack #api #python #listeners
-
The Summertime Failure of Minergie and the Case for AC
Reading Time: 4 minutesMinergie is a fantastic concept for a cold planet, and to keep cool in winter. It traps heat with ease during the morning and radiates it for the rest of the day into the night. It supplements to solar radiation heat with geothermal heat, which it gets from the heat pumps. That heat is then spread across the buidling with hot water pipes.
In winter this is fantastic because it saves on energy and provides a comfortable environment. In reality it’s not unusual for me to notice 25°c in winter, so in effect it might be too warm, for some.
It’s in summer that what makes minergie so awesome in winter becomes a death trap in summer. More than once during the recent heat waves I have had the same head ache as after cycling up to Marchairuz with a sporty cycling group. The difference is that I was sitting, in shorts, at a laptop. I was not exerting myself.
I liken living in a minergie building, especially in the attic as living in a car left in the sun from April until September or later. Within minutes it’s oppressively hot, as soon as you turn off air conditioning, if it’s too hot to open windows, and that’s one of the issues with minergie, more than with traditional buildings.
With a traditional building, if you get too hot you can open to ventilate when it’s cool enough, and the Specific heat capacity will dump the heat that had accumulated. When I lived in an apartment during the 2003 heatwave I opened a window on each side, and the cross breeze provided cooling for the entire day, and key to this, when I closed the windows to avoid mosquitos I didn’t feel radiant heat as soon as I closed the windows.
I often try to find the motivation to wake at five or six in the morning, to open all the windows, including the hallway, to attempt to get cool air into the building, and it works. I go from 29°c to 28°c or even 26°c if I start early enough, and for a few minutes it feels excellent.
The problem is that as soon as the sun hits the balcony it heats the tiles, and the huge windows that are great for bringing heat in, in winter, trap it in summer. You can’t cool from one side once the sun has made its appearance.
I still cool via the veluxes until the outside air temperature (OAT) is similar to the indoor temperature. That’s when everything needs to be closed, and that’s when the oppressive heat stroke provoking level heat begins.
The thing about this Minergie building heat stroke risk is that during the night the air is comfortably cold. It’s 19°c to 21°c. Indoors it’s 30°c or more. That means that with the nocturnal heat exchange, during summer heat waves the building could cool.
People will say “but if you keep the windows open at night you do the same thing for free, or almost, and you’re right, except that the heat is structural, not air.
The hallway that connects all the apartments is sealed from the top. Neither the lift, nor the stairs have a vent. This means that in summer the heat from outside that gets in rises, and gets trapped heating the apartment walls, but also apartment floors. This means that the apartments can be sealed from the outside heat, but if anyone has bad habits, in the floors beneath, then the heat gets trapped, and then rises, for hours at a time. It’s also worth noting that the garage, that is cool in the morning, does heat up, so heat from the garage also makes its way up the stairs.
Three Good Nights
It is important to highlight that I hate the concept of air conditioning because to me, cooling air, by heating air, is negating the entire concept. For me, a simple cross breeze is good, when possible, or taking advantage of nocturnal cooling is also good.
I slept much better three times this year, once when I used air conditioning because I was getting over Puce de Canard and this was an emergency situation. I also found that a night of storms and rain helped make sleep much better, and finally for at least one night I found that having a simple fan, to shift air provided enough of a cooling effect to make sleep more consistent.
The Minergie Standard Update
While writing this blog post I read that the Minergie standard has been updated. It states that buildings should not exceed 26.5°c for more than 100 hours per year. This means that my building should be modernised with cooling systems to match the latest norms.
Minergie is now expected to make spaces more comfortable in summer as well as winter. This means that the renovation to meet this norm is justified in a shared property, especially since I have data showing that the temperatures are currently three to five degrees higher than they should be.
The Self Regulation Optic
With normal buildings, you close windows to keep them warm, and you open windows to cool them. The structure is designed with small windows which reduces heat loss in winter, and heat ingress in summer. Minergie buildings are designed to maximise heat gains in winter, but in summer that makes them uncomfortable.
Their materials makes them hermetic. Once the heat comes in there are no drafts, and thus no cooling. This means that the structure must be able to breath and release heat. A heat efficient building is stiffling in summer, unless you open windows but the paradox is that opening windows changes internal pressure and thus the thermostat behaviour.
With a minergie ventilation only changes the air but the structure stays warm.
That’s why the 2023 amendment to the Minergie standard offers hope. From 2023 onwards buildings must be below 26.5°c. They can be above 26.5°c for 100 hours before they are not within the new norms. I see this with enthusiasm, because it could help justify upgrading the setup of the building I live in.
#airFlow #buildings #drafts #heat #summer #trapped #windows #winter -
The Summertime Failure of Minergie and the Case for AC
Reading Time: 4 minutesMinergie is a fantastic concept for a cold planet, and to keep cool in winter. It traps heat with ease during the morning and radiates it for the rest of the day into the night. It supplements to solar radiation heat with geothermal heat, which it gets from the heat pumps. That heat is then spread across the buidling with hot water pipes.
In winter this is fantastic because it saves on energy and provides a comfortable environment. In reality it’s not unusual for me to notice 25°c in winter, so in effect it might be too warm, for some.
It’s in summer that what makes minergie so awesome in winter becomes a death trap in summer. More than once during the recent heat waves I have had the same head ache as after cycling up to Marchairuz with a sporty cycling group. The difference is that I was sitting, in shorts, at a laptop. I was not exerting myself.
I liken living in a minergie building, especially in the attic as living in a car left in the sun from April until September or later. Within minutes it’s oppressively hot, as soon as you turn off air conditioning, if it’s too hot to open windows, and that’s one of the issues with minergie, more than with traditional buildings.
With a traditional building, if you get too hot you can open to ventilate when it’s cool enough, and the Specific heat capacity will dump the heat that had accumulated. When I lived in an apartment during the 2003 heatwave I opened a window on each side, and the cross breeze provided cooling for the entire day, and key to this, when I closed the windows to avoid mosquitos I didn’t feel radiant heat as soon as I closed the windows.
I often try to find the motivation to wake at five or six in the morning, to open all the windows, including the hallway, to attempt to get cool air into the building, and it works. I go from 29°c to 28°c or even 26°c if I start early enough, and for a few minutes it feels excellent.
The problem is that as soon as the sun hits the balcony it heats the tiles, and the huge windows that are great for bringing heat in, in winter, trap it in summer. You can’t cool from one side once the sun has made its appearance.
I still cool via the veluxes until the outside air temperature (OAT) is similar to the indoor temperature. That’s when everything needs to be closed, and that’s when the oppressive heat stroke provoking level heat begins.
The thing about this Minergie building heat stroke risk is that during the night the air is comfortably cold. It’s 19°c to 21°c. Indoors it’s 30°c or more. That means that with the nocturnal heat exchange, during summer heat waves the building could cool.
People will say “but if you keep the windows open at night you do the same thing for free, or almost, and you’re right, except that the heat is structural, not air.
The hallway that connects all the apartments is sealed from the top. Neither the lift, nor the stairs have a vent. This means that in summer the heat from outside that gets in rises, and gets trapped heating the apartment walls, but also apartment floors. This means that the apartments can be sealed from the outside heat, but if anyone has bad habits, in the floors beneath, then the heat gets trapped, and then rises, for hours at a time. It’s also worth noting that the garage, that is cool in the morning, does heat up, so heat from the garage also makes its way up the stairs.
Three Good Nights
It is important to highlight that I hate the concept of air conditioning because to me, cooling air, by heating air, is negating the entire concept. For me, a simple cross breeze is good, when possible, or taking advantage of nocturnal cooling is also good.
I slept much better three times this year, once when I used air conditioning because I was getting over Puce de Canard and this was an emergency situation. I also found that a night of storms and rain helped make sleep much better, and finally for at least one night I found that having a simple fan, to shift air provided enough of a cooling effect to make sleep more consistent.
The Minergie Standard Update
While writing this blog post I read that the Minergie standard has been updated. It states that buildings should not exceed 26.5°c for more than 100 hours per year. This means that my building should be modernised with cooling systems to match the latest norms.
Minergie is now expected to make spaces more comfortable in summer as well as winter. This means that the renovation to meet this norm is justified in a shared property, especially since I have data showing that the temperatures are currently three to five degrees higher than they should be.
The Self Regulation Optic
With normal buildings, you close windows to keep them warm, and you open windows to cool them. The structure is designed with small windows which reduces heat loss in winter, and heat ingress in summer. Minergie buildings are designed to maximise heat gains in winter, but in summer that makes them uncomfortable.
Their materials makes them hermetic. Once the heat comes in there are no drafts, and thus no cooling. This means that the structure must be able to breath and release heat. A heat efficient building is stiffling in summer, unless you open windows but the paradox is that opening windows changes internal pressure and thus the thermostat behaviour.
With a minergie ventilation only changes the air but the structure stays warm.
That’s why the 2023 amendment to the Minergie standard offers hope. From 2023 onwards buildings must be below 26.5°c. They can be above 26.5°c for 100 hours before they are not within the new norms. I see this with enthusiasm, because it could help justify upgrading the setup of the building I live in.
#airFlow #buildings #drafts #heat #summer #trapped #windows #winter -
🚀🌀 Behold, the riveting saga of big, empty #tunnels where air goes whoosh! Because who doesn't want a photo album of NASA's oversized hairdryers? 📸💨 Surely, nothing screams relevance like peering into the void! 🙄
https://www.theatlantic.com/photo/2018/05/historic-photos-of-nasas-cavernous-wind-tunnels/560660/ #NASA #Airflow #Photography #Engineering #Humor #HackerNews #ngated -
🚀🌀 Behold, the riveting saga of big, empty #tunnels where air goes whoosh! Because who doesn't want a photo album of NASA's oversized hairdryers? 📸💨 Surely, nothing screams relevance like peering into the void! 🙄
https://www.theatlantic.com/photo/2018/05/historic-photos-of-nasas-cavernous-wind-tunnels/560660/ #NASA #Airflow #Photography #Engineering #Humor #HackerNews #ngated -
CVE-2026-33264: Apache Airflow <3.3.0 has a CRITICAL deserialization flaw. DAG authors can trigger remote code execution on Scheduler/API Server. Upgrade to 3.3.0+ and restrict allowed_deserialization_classes. Details: https://radar.offseq.com/threat/cve-2026-33264-cwe-502-deserialization-of-untruste-8bbc1aa55d6c3415 #OffSeq #Airflow #Infosec #CVE202633264
-
Votre pipeline est un *marathon* ? *Optimisez-le*. #DataEngineering #Pipelines #Tech #NoBullshit #Airflow #Spark ... https://www.linkedin.com/posts/gabriel-chandesris_dataengineering-pipelines-tech-share-7479514384663384064-Slvh/
-
Votre pipeline est un *marathon* ? *Optimisez-le*. #DataEngineering #Pipelines #Tech #NoBullshit #Airflow #Spark ... https://www.linkedin.com/posts/gabriel-chandesris_dataengineering-pipelines-tech-share-7479514384663384064-Slvh/
-
Jeppesen ForeFlight launches aviation AI engine designed to minimize flight plan errors
https://1ban.news/foreflight-airflow-aviation-ai-engine/
#1ban #foreflight #airflow #aviation #engine #tech -
RAG без downtime: настраиваем инкрементальное обновление документов на Qdrant и LangChain
PM: Нам нужно актуализировать базу знаний для ИИ-ассистента, там изменилась инструкция по смене пароля. DevOps: Не проблема, сейчас запущу скрипт, через два часа всё обновится. Предупреди Заказчика о недоступности сервиса. Знакомая ситуация? Полная зачистка векторной базы и реиндексация всех имеющихся документов с остановкой сервиса - решение простое и надёжное, но «прощается» только на этапе прототипа. В продакшене могут быть сотни тысяч документов, живые пользователи и SLA с требованием по доступности сервиса. Обновилась одна, пусть и очень важная инструкция, и сервис недоступен два часа. А если таких обновлений десятки в неделю? Давайте исправим это и напишем ETL-скрипт , который умеет добавлять, обновлять и удалять отдельные документы без остановки сервиса.
https://habr.com/ru/articles/1038456/
#rag #etlпайплайн #qdrant #qdrant_vector_store #масштабирование #airflow #python #rag_система
-
Строим машину времени для данных (SCD-2) на движке Trino под управлением Airflow
Сегодня SCD-2-таблицы не только остаются актуальными для медленно меняющихся данных, но и, на мой взгляд, становятся гораздо проще в реализации благодаря новым технологиям и инструментам. Мне поручили пересобрать витрину в ходе миграции в наше новое хранилище данных. Итак, в этой статье мы будем: — строить Iceberg-таблицы SCD-2 с помощью Trino, SQL и Python; — попутно освоим прекрасные функции merge, MD5 и другие полезные инструменты; — напишем свой собственный оператор для Airflow для автоматизации ETL-процесса.
https://habr.com/ru/companies/X5Tech/articles/1034656/
#scd_type_2 #trino #iceberg #airflow #lakehouse #etl #slowly_changing_dimensions #data_mesh #витрина_данных #хранилище_данных
-
Код как документация: как мы строим самодокументируемые витрины данных в Почте Mail
В аналитике больших данных есть старая проблема: код ETL-витрин живет своей жизнью, а документация — своей. Изменяешь логику, забываешь обновить описание колонки — и через месяц никто не помнит, что означает wallet_cards_category_hits. В Почте Mail (VK) мы решили эту проблему системно, разработав внутренний фреймворк, который делает код витрины и ее документацию неразрывными. На связи Дима Швеенков. Я все так же руковожу направлением аналитики в команде и отвечаю за данные в Почте Mail , а теперь еще и отвечаю за DWH в VK Tech . В предыдущих статьях я подробно рассказывал о нашем Data Driven-подходе к работе с данными, а также, в частности, как мы работаем со Spark и какие ключевые проблемы с данными мы решили, чтобы построить свое хранилище данных. Сегодня хотел бы остановиться на более узкой теме — как держать в порядке документацию, если у вас такое же огромное хранилище, как и у нас. Материал короткий, но, надеюсь, будет для вас полезным.
https://habr.com/ru/companies/vktech/articles/1032686/
#big_data #apache_spark #airflow #clickhouse #sql #документация #dwh #metadata #dbt #vk_tech
-
Airflow TaskFlow API: внутреннее устройство современного способа писать DAG-и
Немного погрузимся во внутреннее устройство Apache Airflow и разберёмся, что на самом деле происходит за красивым синтаксисом TaskFlow API. Посмотрим, как работают декораторы @task и @dag , каким образом обычные Python-функции превращаются в задачи Airflow и за счёт какой «магии» строится граф зависимостей. А заодно напишем собственный мини-пример, чтобы лучше понять архитектурные идеи, на которых построен современный Airflow. Заглянем под капот Airflow TaskFlow API
-
Heat Recovery Ventilator (HRV) — Operating Principle
A recuperator (heat recovery unit) transfers heat from exhaust air to incoming fresh air without mixing the two streams.
---
How It Works
Two airflows:
Exhaust air (warm, from indoors)
Supply air (cold, from outside)
They pass through a heat exchanger:
separated by plates or channels
no direct mixing
heat transfers through the material (conduction)
Result: → supply air is preheated
→ exhaust air is cooled
→ overall heat loss is reduced---
Types of Recuperators
1. Plate Heat Exchanger
aluminum or plastic plates
efficiency: ~60–90%
no moving parts
2. Rotary (Wheel) Heat Exchanger
rotating drum
transfers heat and some moisture
efficiency: up to ~85–90%
3. Counterflow Heat Exchanger
air streams move in opposite directions
highest efficiency: up to ~95%
---
What Is Transferred
heat (primary)
sometimes moisture (in enthalpy units)
---
Efficiency Example
outside: 0°C
indoor: +22°C
after recovery: ~16–20°C
---
Advantages
reduced heating energy demand
continuous ventilation without major heat loss
improved indoor air quality
---
Limitations
frost formation in winter (needs bypass or preheater)
filter maintenance required
upfront cost
---
Core Idea
A recuperator doesn’t generate heat — it recovers and reuses it.
#HVAC #HeatRecovery #HRV #ERV #EnergyEfficiency #Ventilation #IndoorAirQuality #AirExchange #HeatExchanger #SustainableLiving #GreenBuilding #EnergySaving #HomeComfort #SmartHome #BuildingEngineering #ClimateControl #EcoTech #Airflow #FreshAir #LowEnergy #PassiveHouse #NetZero #HomeImprovement #Engineering #CleanAir
-
Единое окно инженера данных: как мы построили веб-среду разработки для 12 000 потоков данных
Привет, Хабр! Меня зовут Никита Калганов, я ведущий системный инженер данных в X5 Tech. Мы с командой проектируем и развиваем высоконагруженную систему потоков данных и поддерживаем её в условиях постоянного роста, как по числу потоков, так и по количеству команд, которые с ней работают. В нашей системе обработки данных живёт больше двенадцати тысяч потоков данных, и их число растёт в среднем на несколько десятков в день. Всё нормально работает, но мы решили сделать её ещё лучше. В этой статье расскажу, как мы построили единое окно инженера данных, сделали собственный DDL-мигратор с поддержкой зависимостей и при этом сохранили то, что уже работает.
https://habr.com/ru/companies/X5Tech/articles/1026382/
#большие_данные #big_data #lakehouseплатформа_данных #data_pipelines #airflow #yaml #gitlab_cicd #микросервисная_архитектура #оркестрация #ddl
-
🖥️ Donne du style et de la puissance à ton setup !
Découvre notre sélection de boîtiers PC gaming 🔥
RGB, airflow optimisé, design futuriste… tout pour sublimer ta config 💡⚡ Que tu sois gamer ou créateur, trouve le boîtier parfait pour ton build !
👉 Explore maintenant la catégorie et upgrade ton setup !
https://inmedia.ma/product-category/accessoires-composants/boitiers-pc/
#pcgaming, #boitierpc, #setupgaming, #rgb, #gamingsetup, #pcbuilder, #airflow, #gaminggear
-
Big Data больше не для гигантов: связка Airflow + ClickHouse вытеснила Airflow + PostgreSQL
Открываю ноутбук. Захожу в метрики своего интернет-магазина. 500 тысяч посетителей в день. Каждый клик — событие. Каждый просмотр товара — строка в логах. Каждое добавление в корзину — ещё одна строка. За месяц — 15 миллионов событий. За год — почти 200 миллионов. Это не Google. Не Amazon. Это обычный средний магазин на ~100 человек. А теперь умножьте это на количество таблиц: пользователи, заказы, платежи, доставки, отзывы, просмотры, лайки, рефералы, купоны, возвраты... Поздравляю. Вы уже работаете с Big Data. В 2026 году это уже не привилегия корпораций, а стандарт ведения цифрового бизнеса. И как следствие этой "гонки вооружений" произошла тектоническая смена ориентиров. Классическая связка Airflow + PostgreSQL, которая ещё вчера считалась золотым стандартом, сегодня стремительно сдает позиции. Её место уверенно занимает дуэт Airflow + ClickHouse — технологический фундамент современной инженерии данных.
https://habr.com/ru/articles/1022460/
#clickhouse #postgresql #data_engineer #dwh #airflow #big_data #аналитика #рынок_труда #sql #python
-
The Easy Pea Trellis DIY for Healthier, Happier Plants https://www.allforgardening.com/1707690/the-easy-pea-trellis-diy-for-healthier-happier-plants/ #Airflow #DecorTrends #diy #garden #GardenTwine #gardener #gardening #GardeningTips&Hacks #GardeningTools #harvesting #HomeHacks #LivingWallOfGreenery #MildewPrevention #news #OrganicGardening #PeaPlants #PlantingPlans #Stakes #trellis #VegetableGarden #VegetableGardening
-
The Easy Pea Trellis DIY for Healthier, Happier Plants https://www.allforgardening.com/1707690/the-easy-pea-trellis-diy-for-healthier-happier-plants/ #Airflow #DecorTrends #diy #garden #GardenTwine #gardener #gardening #GardeningTips&Hacks #GardeningTools #harvesting #HomeHacks #LivingWallOfGreenery #MildewPrevention #news #OrganicGardening #PeaPlants #PlantingPlans #Stakes #trellis #VegetableGarden #VegetableGardening
-
Is the airflow going to be anything good here?
#fans #airflow #engineering #onlyfans #fluiddynamics #air #hvac
-
Built an end-to-end data pipeline using GCP, Airflow, PySpark, and BigQuery to analyze thermal anomaly data (India 🇮🇳 vs USA 🇺🇸).
Uncovered patterns in fire frequency, intensity, and seasonality through interactive dashboards.
#DataEngineering #GCP #Airflow #BigQuery #PySpark -
How important is ventillation for a small makeshift homelab?
https://quokk.au/c/homelab/p/788275/how-important-is-ventillation-for-a-small-makeshift-homelab
-
𝗔𝗶𝗿𝗳𝗹𝗼𝘄:
#Workflow #Orchestration #Airflow
https://thewhale.cc/posts/airflow
Apache Airflow is a **flexible and powerful platform for workflow orchestration**, which uses Python, offers an intuitive user interface, and has numerous integrations, all within a scalable and open-source framework.
-
TCO или Полная Стоимость Владение современных подходов в ETL для DB MPP
О чем эта статья : В данной статье я хочу сравнить TCO старых добрых ETL как например Informatica, ODI, MarkitEDM и подобных им vs DBT + AirFlow и подобных им Очень легко проанализировать стоимость лицензий или вычислений и хранения в случае облачной БД, но очень сложно — TCO. Стоимость разработки одной фичи, стоимость поддержки, стоимость сопровождения, стоимость изменений. Очень заманчиво учитывать только расходы на лицензии и вычисления и предполагать, что все остальные расходы одинаковы, хотя это не так. По умолчанию облачные MPP-базы обычно дешевле по хранению и вычислениям и не имеют лицензионной платы, и возникает соблазн использовать такой же безлицензионный подход в ETL, но есть недостатки :
-
✨ #AI in #BI is like a self-driving car: that still needs you at the wheel.
Maxime Beauchemin, creator of Apache #Superset and Apache #Airflow, explains why #analytics aren't ready to go full autopilot and how we can embrace smart assistance instead. Because 90% isn’t good enough.
🎥 Watch now! https://abracadata.space/sessions/2025/ai-in-bi/
-
Как мы мигрировали с Zeppelin и что из этого вышло. Часть 1. Рассылки
Для задач аналитики Zeppelin - это чуть ли не находка. Он может в одной книжке исполнять код на любом языке (был бы интерпретатор), выводить его в виде красивых табличек, графиков и в любом другом виде, который удобен. И на Хабре есть много статей , посвященных плюсом данного решения для задач аналитики. В результате в аналитический отдел ТКБ Инвестмент Партнерс данный инструмент проник достаточно глубоко: на нем писались аналитические отчеты для разных отделов, расчетные таблицы, которые затем отсылались в дашборды, также данными книжками могли пользоваться все, кто знал какие-либо языки программирования. И мы попали в зависимость от него. Панацея, которая должна была стать легким решением для (почти) любой задачи превратилась в наше проклятие. Из-за проблем с данной системой на нас приходило по 5-7 тикеров в неделю, а также потерей доверия к нашим сервисам. Как мы с этим справились?
-
Как мы разгрузили базу данных в проде и не сломали систему
Наверное, всем знакома ситуация, когда собираешь красивый отчет в BI-системе, пользуешься им полгода, а потом он с каждым днем тормозит все сильнее? Графики обновляются дольше обычного, таблицы медленно грузятся. А все потому, что BI работает на одной базе на пределе своих возможностей из-за перегруженности. Пример классической схемы, с которой начинают почти все компании: BI + база данных. Пока стартапы не могут позволить себе дорогостоящее озеро данных или DWH, они устанавливают базу данных, заполняют ее и прикручивают рядом BI-систему. Вот только данные в растущей компании имеют свойство прибывать. К примеру, у стартапа в первые месяцы работы было пять заказов в день, затем 100, 1 000 и еще больше. День за днем, год за годом все данные складываются в одну базу данных, и в итоге получается тяжелая, медленная система, которой банально неудобно пользоваться. Можно увеличить мощность базы данных, но это лишь отложит проблему на время, а не решит ее. А можно снять нагрузку с базы и доверить обработку накопленных данных другим инструментам и системам хранения. В этой статье я расскажу, как решить проблему с базой данных, которая трещит по швам, с помощью облачных managed-сервисов.
https://habr.com/ru/companies/cloud_ru/articles/987934/
#хранение_данных #bi #s3 #база_данных #хранение_данных_в_облаке #trino #metastore #airflow
-
Hello #Fediverse! Looks like I’m an #introduction away from joining the conversation so here goes!
I lead data teams for a healthcare system in New Mexico. My healthcare path started in acute care nursing before moving into #Informatics. These days, I’m deep into #Python, #Airflow, and #LLMs. I also love #Soccer (#Football), #Sailing, and doing anything my kids let me tag along for.
Looking to connect on #HealthIT, #DataEngineering, and #DataScience or just chat about other interests.
-
Hello #Fediverse! Looks like I’m an #introduction away from joining the conversation so here goes!
I lead data teams for a healthcare system in New Mexico. My healthcare path started in acute care nursing before moving into #Informatics. These days, I’m deep into #Python, #Airflow, and #LLMs. I also love #Soccer (#Football), #Sailing, and doing anything my kids let me tag along for.
Looking to connect on #HealthIT, #DataEngineering, and #DataScience or just chat about other interests.
-
There are better, but after designing and printing my own fan shroud for a case fan to add better airflow for my gpu, I'm kinda proud of myself, thats a 10C difference. #gpu #burn #airflow #3dprint #Temperature
-
💡 Apache Airflow 2025 Recap
2026 has arrived which marks a great opportunity to review and assess the changes the last year has brought to the ever evolving landscape of open source data tools.
In our new #blog post we look at Apache Airflow and how the leading open source orchestration platform has changed in the last year with the bit v3 major release.
#apacheairflow #airflow #opensource #dataengineering #datascience
-
💡 Apache Airflow 2025 Recap
2026 has arrived which marks a great opportunity to review and assess the changes the last year has brought to the ever evolving landscape of open source data tools.
In our new #blog post we look at Apache Airflow and how the leading open source orchestration platform has changed in the last year with the bit v3 major release.
#apacheairflow #airflow #opensource #dataengineering #datascience
-
Just when I thought I was almost out of yaks (and finally installing #Airflow!), #terragrunt got confused and started demanding to create resources that already exist, which broke #openEBS, which broke... sigh ...
Another 2 days of work later, #argocd is installed in my neurons and my cluster, and most of my config is refactored "enough". I swear we'll actually get to do some #datascience someday folks...
Big data on a tiny budget is hard!
-
"The release served as a crucial turning point for the project. Downloads from its GitHub repository increased, and more enterprises adopted the software. Encouraged by this growth, the team envisioned the next generation of Airflow: a modular architecture, a more modern user interface, and a “run anywhere, anytime” feature, enabling it to operate on premises, in the cloud, or on edge devices and handle event-driven and ad hoc scenarios in addition to scheduled tasks. The team delivered on this vision with the launch of Airflow 3.0 last April.
“It was amazing that we managed to ‘rebuild the plane while flying it’ when we worked on Airflow 3—even if we had some temporary issues and glitches,” says Jarek Potiuk, one of the foremost contributors to Airflow and now a member of its project-management committee. “We had to refactor and move a lot of pieces of the software while keeping Airflow 2 running and providing some bug fixes for it.”
Compared with Airflow’s second version, which Koka says had only a few hundred to a thousand downloads per month on GitHub, “now we’re averaging somewhere between 35 to 40 million downloads a month,” he says. The project’s community also soared, with more than 3,000 developers of all skill levels from around the world contributing to Airflow."
https://spectrum.ieee.org/apache-airflow-3-programmatic-workflows
#AirFlow #ApacheAirflow #AirBnB #OpenSource #FLOSS #WorkflowOrchestratror #Python #DataPipelines
-
"The release served as a crucial turning point for the project. Downloads from its GitHub repository increased, and more enterprises adopted the software. Encouraged by this growth, the team envisioned the next generation of Airflow: a modular architecture, a more modern user interface, and a “run anywhere, anytime” feature, enabling it to operate on premises, in the cloud, or on edge devices and handle event-driven and ad hoc scenarios in addition to scheduled tasks. The team delivered on this vision with the launch of Airflow 3.0 last April.
“It was amazing that we managed to ‘rebuild the plane while flying it’ when we worked on Airflow 3—even if we had some temporary issues and glitches,” says Jarek Potiuk, one of the foremost contributors to Airflow and now a member of its project-management committee. “We had to refactor and move a lot of pieces of the software while keeping Airflow 2 running and providing some bug fixes for it.”
Compared with Airflow’s second version, which Koka says had only a few hundred to a thousand downloads per month on GitHub, “now we’re averaging somewhere between 35 to 40 million downloads a month,” he says. The project’s community also soared, with more than 3,000 developers of all skill levels from around the world contributing to Airflow."
https://spectrum.ieee.org/apache-airflow-3-programmatic-workflows
#AirFlow #ApacheAirflow #AirBnB #OpenSource #FLOSS #WorkflowOrchestratror #Python #DataPipelines
-
One terminal to rule them all.
🌸 **flowrs** — A TUI for Apache Airflow
🌀 Monitor, inspect & manage DAGs right from your terminal
⚙️ Supports AWS MWAA, GCP Composer, Astronomer & more.🦀 Written in Rust & built with @ratatui_rs
⭐ GitHub: https://github.com/jvanbuel/flowrs#rustlang #ratatui #tui #airflow #devops #terminal #opensource
-
Is anyone else considering or using #Apache #Airflow for their #cybersecurity orchestration, #automation and response platform? (#soar)
My #development team builds and shares #opensource recipes, but our current closed source SOAR solution is limiting our ability to share solutions among our peers.
I'm interested in hearing about anyone's experiences with #airflow, good, bad or neutral.
Thanks!
-
#getfedihired I have a full #dataengineering project team capable of developing #datapipelines #airflow #orchestration and #automation ready for project work.
We work on #openstack #aws and the other guys. We try to stay in the #linux and #python realms.
We also do #sqlserver and #postgresql administration design and development and are accepting projects and providing a limited number of assessments.
https://infocusdata.com for info.
Thanks