#apacheairflow — Public Fediverse posts

Pinterest Deploys Production-Scale Model Context Protocol Ecosystem for AI Agent Workflows

Pinterest engineering teams have deployed an internal Model Context Protocol (MCP) ecosystem to power AI agents that automate…
#NewsBeep #News #Technology #Agents #ApacheAirflow #Architecture&Design #Artificialintelligence #CA #Canada #Development #Ecosystem #ide #integration #ModelContextProtocol(MCP) #pinterestmcpecosystem #Presto #spark #Workflow/BPM
https://www.newsbeep.com/ca/575647/

💡 Apache Airflow 2025 Recap

2026 has arrived which marks a great opportunity to review and assess the changes the last year has brought to the ever evolving landscape of open source data tools.

In our new #blog post we look at Apache Airflow and how the leading open source orchestration platform has changed in the last year with the bit v3 major release.

🔗 https://www.nextlytics.com/blog/apache-airflow-updates-2025-a-deep-dive-into-features-added-after-3.0

#apacheairflow #airflow #opensource #dataengineering #datascience

"The release served as a crucial turning point for the project. Downloads from its GitHub repository increased, and more enterprises adopted the software. Encouraged by this growth, the team envisioned the next generation of Airflow: a modular architecture, a more modern user interface, and a “run anywhere, anytime” feature, enabling it to operate on premises, in the cloud, or on edge devices and handle event-driven and ad hoc scenarios in addition to scheduled tasks. The team delivered on this vision with the launch of Airflow 3.0 last April.

“It was amazing that we managed to ‘rebuild the plane while flying it’ when we worked on Airflow 3—even if we had some temporary issues and glitches,” says Jarek Potiuk, one of the foremost contributors to Airflow and now a member of its project-management committee. “We had to refactor and move a lot of pieces of the software while keeping Airflow 2 running and providing some bug fixes for it.”

Compared with Airflow’s second version, which Koka says had only a few hundred to a thousand downloads per month on GitHub, “now we’re averaging somewhere between 35 to 40 million downloads a month,” he says. The project’s community also soared, with more than 3,000 developers of all skill levels from around the world contributing to Airflow."

https://spectrum.ieee.org/apache-airflow-3-programmatic-workflows

#AirFlow #ApacheAirflow #AirBnB #OpenSource #FLOSS #WorkflowOrchestratror #Python #DataPipelines

New to #ApacheAirflow? Build your first workflow!

Learn how this #opensource orchestration tool helps you schedule, monitor & manage data pipelines in #Python. Watch Ricardo Sueiras guide beginners through hands-on steps.

Click here: https://youtu.be/sdBimSGxAcs

🎤Ne manquez pas notre conférence aux #RPLL2025 :
Manipulation de la #GED avec Apache Airflow

🗓️jeudi 15 mai à 12:00
📍Hôtel de la Métropole, Lyon
🙋Présentée par @jeremielesage

👉https://jeci.fr/fr/evenements/rpll-atelier-2025/
#LogicielLibre #floss #ApacheAirflow

🤖🚀 Ah yes, because nothing says "simple" like melding Apache Airflow and LLM workflows with a side of AI Agents. Just what every developer dreams of: wrestling with Pydantic AI while pretending to control the automation beast. 🛠️✨
https://github.com/astronomer/airflow-ai-sdk #ApacheAirflow #LLMworkflows #AIAgents #PydanticAutomation #DeveloperDreams #HackerNews #ngated

LLM Workflows then Agents: Getting Started with Apache Airflow

https://github.com/astronomer/airflow-ai-sdk

#HackerNews #LLM #Workflows #Agents #ApacheAirflow #DataEngineering #AIWorkflow

🍹 C'est l'heure du cocktail de bienvenue proposé par Satya !

- une dose de #ModernDataStack,
- un trait de géo,
- un zeste d'Open Source,
- et beaucoup d'amour 💗.

Cette recette vous est servie dans cet article qui détaille comment le Gard valorise ses géo-données.

:geotribu: https://geotribu.fr/articles/2025/2025-02-25_stack_data_gard/

Relecture 🧐 : @geojulien & Michaël Galien

#PostgreSQL #PostGIS #GDAL #OGR #DBT #Metabase #ApacheAirflow

Just caught up with the recent Delta Lake webinar,

> Revolutionizing Delta Lake workflows on AWS Lambda with Polars, DuckDB, Daft & Rust

Some interesting hints there regarding lightweight processing of big-ish data. Easy to relate to any other framework instead of Lambda, e.g. #ApacheAirflow tasks

https://youtu.be/BR9oFD0QMAs

#dataengineering #datascience #duckdb #daft #polars #pandas #python #spark #deltalake #databricks #airflow #bigdata #smalldata

Azure Data Factory And Apache Airflow Integration Flaws Let Attackers Gain Write Access https://cybersecuritynews.com/azure-airflow-security-flaw/ #InformationSecurityNews #SecurityVulnerabilities #CyberSecurityNews #VulnerabilityNews #AzureDataFactory #ApacheAirflow #vulnerability #cloud

If you're using #ApacheAirflow, I'm interested in hearing your use cases for it.

#DevOps

📝 "How to build a data extraction pipeline with Apache Airflow"

👤 Déborah Mesquita

🔗 https://towardsdatascience.com/how-to-build-a-data-extraction-pipeline-with-apache-airflow-fa83cb8dbcdf?source=rss-dd9e06a0a640------2

#pyladies #python #oldiebutgoodie #apacheairflow #nlp #handsontutorials #editorspick #datascience

Still, #dagster has less dependencies, and after some battling and downgrading version, I managed to start the dev environment...

Building a #Apacheairflow container, is pure chaos. Installing via pip, is another kind of hell, with broken builds all over the place (google-re2)... A tool that has 9 years in the market, being so overwhelming its installation process

At least I can make #dagster run.

I guess we are all ill served with workflow orchestration tools anyway, on the open source world

I take back what I said yesterday... #dagster installation process is quite clumsy. And I detected some issues in terms of documentation/instructions.
I manage to install and run #Apacheairflow in 2 clicks...

How we orchestrate 2000+ #DBT models in #ApacheAirflow

https://medium.com/@aleexmagno/how-we-orchestrate-2000-dbt-models-in-apache-airflow-90901504032d

Going to #PyConUS for the first time. Looking forward to actually meet the community folks. Just in time to gear up for development streamlining discussions for the upcoming
#apacheAirflow 3.0.

https://us.pycon.org/2024/

Tenable Research discovered a one-click account takeover vulnerability in the AWS Managed Workflows Apache Airflow service, and that could have resulted in remote code execution (RCE) on the underlying instance, and in lateral movement to other services. Additional research revealed that numerous shared-parent service domains in AWS, Azure and GCP were misconfigured, putting cloud customers at considerable risk. No CVE ID associated. 🔗 https://www.tenable.com/blog/flowfixation-aws-apache-airflow-service-takeover-vulnerability-and-why-neglecting-guardrails

#FlowFixation #AWS #MWAA #ApacheAirflow #RCE

FlowFixation: AWS Apache Airflow Service Takeover Vulnerability

Date: March 21, 2024
CVE: Not specified
Sources: Tenable Blog

Issue Summary

Tenable Research discovered a vulnerability, named FlowFixation, in AWS Managed Workflows for Apache Airflow (MWAA) that could allow session hijacking leading to a full takeover of the victim's web management panel.

Technical Key findings

FlowFixation combines session fixation and XSS via Amazon AWS domain misconfiguration, enabling attackers to authenticate known sessions and gain control over victim's Apache Airflow management panels.

Vulnerable products

• AWS Managed Workflows for Apache Airflow (MWAA)

Impact assessment

Potential for remote code execution on underlying instances and lateral movement to other services.

Patches or workaround

AWS has addressed the vulnerability. Users should ensure they are using updated services.

Tags

#AWS #ApacheAirflow #CloudSecurity #SessionHijacking #Vulnerability

LLMs are good coders, useless writers

https://samfirke.com/2023/12/22/llms-are-good-coders-useless-writers/

🌪️ Just published my first blog post on Apache Airflow: a zero-to-one introduction to this tool. It details how to get it up and running using Docker 🐳, guides you through creating and scheduling simple pipelines 📊, and shows how to keep an eye on them with the Airflow UI.

https://www.franciscoyira.com/post/data-pipelines-cloud-intro-airflow-docker/
#ApacheAirflow #DataScience

I find #scopecreep to be a real problem with #selfGuidedLearning, particularly online. Do others find the same and, if so, how do you deal with it?

As an example, I sat down to dig into #ApacheSuperset but that led to #ApacheAirflow, which lead to the Astronomer.io service, which led to #ApacheFlink. By the end of the day I’d had a great overview of some really neat stuff but was not better positioned to be productive in Superset than I was at the start of the day.

Tech news digest from Alex, July’23

#apacheairflow #dbt #aws #snowflake

https://agordienko.medium.com/tech-news-digest-from-alex-july23-19950d00b22a