#dataengineering — Public Fediverse posts

Every transformation tested, every model documented, lineage you can trace end to end. My dbt project treats analytics like software.
#dbt #DataEngineering #Analytics #SQL

Every transformation tested, every model documented, lineage you can trace end to end. My dbt project treats analytics like software.
#dbt #DataEngineering #Analytics #SQL

Every transformation tested, every model documented, lineage you can trace end to end. My dbt project treats analytics like software.
#dbt #DataEngineering #Analytics #SQL

Every transformation tested, every model documented, lineage you can trace end to end. My dbt project treats analytics like software.
#dbt #DataEngineering #Analytics #SQL

Zero data loss in migrations isn’t a promise—it’s an engineering outcome. Learn how mapping, validation, exception handling, and auditability make it provable. https://hackernoon.com/zero-data-loss-is-a-design-problem-not-a-promise #dataengineering

Zero data loss in migrations isn’t a promise—it’s an engineering outcome. Learn how mapping, validation, exception handling, and auditability make it provable. https://hackernoon.com/zero-data-loss-is-a-design-problem-not-a-promise #dataengineering

Zero data loss in migrations isn’t a promise—it’s an engineering outcome. Learn how mapping, validation, exception handling, and auditability make it provable. https://hackernoon.com/zero-data-loss-is-a-design-problem-not-a-promise #dataengineering

Zero data loss in migrations isn’t a promise—it’s an engineering outcome. Learn how mapping, validation, exception handling, and auditability make it provable. https://hackernoon.com/zero-data-loss-is-a-design-problem-not-a-promise #dataengineering

Zero data loss in migrations isn’t a promise—it’s an engineering outcome. Learn how mapping, validation, exception handling, and auditability make it provable. https://hackernoon.com/zero-data-loss-is-a-design-problem-not-a-promise #dataengineering

🐍PyBay 2026 Speaker Highlight🎤 Rodrigo Silva Ferreira

Building scientific observatories with Python.

Rodrigo Silva Ferreira is a developer tooling and AI practitioner with a background in scientific computing and software quality.

📍 Oct. 03, 2026, San Francisco, CA: https://pybay.org/
🎤 More talks: https://pybay.org/speaking/speaker-profiles/
🎟️ Get your tickets at: https://pybay.org/get-tickets/

#PyBay #Python #PyBay2026 #Data #DataEngineering #DataVisualization #PythonForScience

🐍PyBay 2026 Speaker Highlight🎤 Rodrigo Silva Ferreira

Building scientific observatories with Python.

Rodrigo Silva Ferreira is a developer tooling and AI practitioner with a background in scientific computing and software quality.

📍 Oct. 03, 2026, San Francisco, CA: https://pybay.org/
🎤 More talks: https://pybay.org/speaking/speaker-profiles/
🎟️ Get your tickets at: https://pybay.org/get-tickets/

#PyBay #Python #PyBay2026 #Data #DataEngineering #DataVisualization #PythonForScience

🐍PyBay 2026 Speaker Highlight🎤 Rodrigo Silva Ferreira

Building scientific observatories with Python.

Rodrigo Silva Ferreira is a developer tooling and AI practitioner with a background in scientific computing and software quality.

📍 Oct. 03, 2026, San Francisco, CA: https://pybay.org/
🎤 More talks: https://pybay.org/speaking/speaker-profiles/
🎟️ Get your tickets at: https://pybay.org/get-tickets/

#PyBay #Python #PyBay2026 #Data #DataEngineering #DataVisualization #PythonForScience

🐍PyBay 2026 Speaker Highlight🎤 Rodrigo Silva Ferreira

Building scientific observatories with Python.

Rodrigo Silva Ferreira is a developer tooling and AI practitioner with a background in scientific computing and software quality.

📍 Oct. 03, 2026, San Francisco, CA: https://pybay.org/
🎤 More talks: https://pybay.org/speaking/speaker-profiles/
🎟️ Get your tickets at: https://pybay.org/get-tickets/

#PyBay #Python #PyBay2026 #Data #DataEngineering #DataVisualization #PythonForScience

🐍PyBay 2026 Speaker Highlight🎤 Rodrigo Silva Ferreira

Building scientific observatories with Python.

Rodrigo Silva Ferreira is a developer tooling and AI practitioner with a background in scientific computing and software quality.

📍 Oct. 03, 2026, San Francisco, CA: https://pybay.org/
🎤 More talks: https://pybay.org/speaking/speaker-profiles/
🎟️ Get your tickets at: https://pybay.org/get-tickets/

#PyBay #Python #PyBay2026 #Data #DataEngineering #DataVisualization #PythonForScience

AI staff augmentation in Miami gives tech leaders a direct route to senior ML and data engineering talent without the multi-month recruiting timeline. Plug verified specialists into your existing team and infrastructure. #AI #Miami #StaffAugmentation #MachineLearning #DataEngineering

Explore how we can help at https://codeponents.com/ai-staff-augmentation-miami-engineered-delivery-throughput/?ref=2&rid=230

AI staff augmentation in Miami gives tech leaders a direct route to senior ML and data engineering talent without the multi-month recruiting timeline. Plug verified specialists into your existing team and infrastructure. #AI #Miami #StaffAugmentation #MachineLearning #DataEngineering

Explore how we can help at https://codeponents.com/ai-staff-augmentation-miami-engineered-delivery-throughput/?ref=2&rid=230

AI staff augmentation in Miami gives tech leaders a direct route to senior ML and data engineering talent without the multi-month recruiting timeline. Plug verified specialists into your existing team and infrastructure. #AI #Miami #StaffAugmentation #MachineLearning #DataEngineering

Explore how we can help at https://codeponents.com/ai-staff-augmentation-miami-engineered-delivery-throughput/?ref=2&rid=230

AI staff augmentation in Miami gives tech leaders a direct route to senior ML and data engineering talent without the multi-month recruiting timeline. Plug verified specialists into your existing team and infrastructure. #AI #Miami #StaffAugmentation #MachineLearning #DataEngineering

Explore how we can help at https://codeponents.com/ai-staff-augmentation-miami-engineered-delivery-throughput/?ref=2&rid=230

AI staff augmentation in Miami gives tech leaders a direct route to senior ML and data engineering talent without the multi-month recruiting timeline. Plug verified specialists into your existing team and infrastructure. #AI #Miami #StaffAugmentation #MachineLearning #DataEngineering

Explore how we can help at https://codeponents.com/ai-staff-augmentation-miami-engineered-delivery-throughput/?ref=2&rid=230

We have big news to share! 🎉

AWS has become a Platinum Sponsor of Psycopg!

For two decades, Psycopg has been the standard way Python talks to PostgreSQL. This sponsorship reflects the role Psycopg plays across the Python and PostgreSQL ecosystem, and will fund the ongoing development of Psycopg 3 and 2.

If your organisation depends on Psycopg, consider supporting us. Links in replies 👇

#PostgreSQL #Python #Psycopg #AWS #OpenSource #FreeSoftware #FOSS #DataEngineering #SponsorOpenSource

We have big news to share! 🎉

AWS has become a Platinum Sponsor of Psycopg!

For two decades, Psycopg has been the standard way Python talks to PostgreSQL. This sponsorship reflects the role Psycopg plays across the Python and PostgreSQL ecosystem, and will fund the ongoing development of Psycopg 3 and 2.

If your organisation depends on Psycopg, consider supporting us. Links in replies 👇

#PostgreSQL #Python #Psycopg #AWS #OpenSource #FreeSoftware #FOSS #DataEngineering #SponsorOpenSource

We have big news to share! 🎉

AWS has become a Platinum Sponsor of Psycopg!

For two decades, Psycopg has been the standard way Python talks to PostgreSQL. This sponsorship reflects the role Psycopg plays across the Python and PostgreSQL ecosystem, and will fund the ongoing development of Psycopg 3 and 2.

If your organisation depends on Psycopg, consider supporting us. Links in replies 👇

#PostgreSQL #Python #Psycopg #AWS #OpenSource #FreeSoftware #FOSS #DataEngineering #SponsorOpenSource

We have big news to share! 🎉

AWS has become a Platinum Sponsor of Psycopg!

For two decades, Psycopg has been the standard way Python talks to PostgreSQL. This sponsorship reflects the role Psycopg plays across the Python and PostgreSQL ecosystem, and will fund the ongoing development of Psycopg 3 and 2.

If your organisation depends on Psycopg, consider supporting us. Links in replies 👇

#PostgreSQL #Python #Psycopg #AWS #OpenSource #FreeSoftware #FOSS #DataEngineering #SponsorOpenSource

Billions of events, one streaming pipeline: Kafka into Spark, aggregations out to a live dashboard. One of eight big-data projects scoped to genuinely large public data.
#DataEngineering #Kafka #Spark #BigData

Billions of events, one streaming pipeline: Kafka into Spark, aggregations out to a live dashboard. One of eight big-data projects scoped to genuinely large public data.
#DataEngineering #Kafka #Spark #BigData

Billions of events, one streaming pipeline: Kafka into Spark, aggregations out to a live dashboard. One of eight big-data projects scoped to genuinely large public data.
#DataEngineering #Kafka #Spark #BigData

Агент пишет код. Что тогда остаётся инженеру данных?

В работе инженера данных хватает повторяющихся задач: забрать данные из API, проверить структуру, преобразовать поля, записать результат в хранилище. Вокруг этого — настройки, тесты, сообщения об ошибках и документация. Такую работу удобно поручать Агенту. Опять агенты

https://habr.com/ru/articles/1084096/

#сезон_код_будущего #ai #aiагенты #agents #dataengineering #dataengineer #data #python #ии #ииассистент

Агент пишет код. Что тогда остаётся инженеру данных?

В работе инженера данных хватает повторяющихся задач: забрать данные из API, проверить структуру, преобразовать поля, записать результат в хранилище. Вокруг этого — настройки, тесты, сообщения об ошибках и документация. Такую работу удобно поручать Агенту. Опять агенты

https://habr.com/ru/articles/1084096/

#сезон_код_будущего #ai #aiагенты #agents #dataengineering #dataengineer #data #python #ии #ииассистент

Managing and visualizing thousands of spatial data points per second requires specific, highly scalable infrastructure. Here is a technical breakdown of an open-source GIS architecture designed specifically to ingest, process, and map real-time IoT sensor data and telemetry with minimal latency.

Click here: https://rsandgis.me/articles/open-source-iot-gis-architecture.html

#IoT #GIS #Architecture #DataEngineering #SpatialData #RealTime

Managing and visualizing thousands of spatial data points per second requires specific, highly scalable infrastructure. Here is a technical breakdown of an open-source GIS architecture designed specifically to ingest, process, and map real-time IoT sensor data and telemetry with minimal latency.

Click here: https://rsandgis.me/articles/open-source-iot-gis-architecture.html

#IoT #GIS #Architecture #DataEngineering #SpatialData #RealTime

Managing and visualizing thousands of spatial data points per second requires specific, highly scalable infrastructure. Here is a technical breakdown of an open-source GIS architecture designed specifically to ingest, process, and map real-time IoT sensor data and telemetry with minimal latency.

Click here: https://rsandgis.me/articles/open-source-iot-gis-architecture.html

#IoT #GIS #Architecture #DataEngineering #SpatialData #RealTime

Managing and visualizing thousands of spatial data points per second requires specific, highly scalable infrastructure. Here is a technical breakdown of an open-source GIS architecture designed specifically to ingest, process, and map real-time IoT sensor data and telemetry with minimal latency.

Click here: https://rsandgis.me/articles/open-source-iot-gis-architecture.html

#IoT #GIS #Architecture #DataEngineering #SpatialData #RealTime

Struggling with painfully slow spatial queries on large vector layers? Learn advanced indexing strategies, spatial clustering techniques, and critical PostgreSQL configuration tweaks to maximize the read and write performance of your open-source spatial database architectures.

Click here: https://rsandgis.me/articles/open-source-spatial-database-performance.html

#PostGIS #Database #Geospatial #DataEngineering #GIS #PostgreSQL

Struggling with painfully slow spatial queries on large vector layers? Learn advanced indexing strategies, spatial clustering techniques, and critical PostgreSQL configuration tweaks to maximize the read and write performance of your open-source spatial database architectures.

Click here: https://rsandgis.me/articles/open-source-spatial-database-performance.html

#PostGIS #Database #Geospatial #DataEngineering #GIS #PostgreSQL

Struggling with painfully slow spatial queries on large vector layers? Learn advanced indexing strategies, spatial clustering techniques, and critical PostgreSQL configuration tweaks to maximize the read and write performance of your open-source spatial database architectures.

Click here: https://rsandgis.me/articles/open-source-spatial-database-performance.html

#PostGIS #Database #Geospatial #DataEngineering #GIS #PostgreSQL

AWS cloud data loss in Bahrain and UAE raises resilience questions https://www.cloudcomputing-news.net/news/aws-cloud-data-loss-bahrain-uae-resilience/?utm_source=dlvr.it&utm_medium=mastodon #Cloud #Automation #Data #DataPlatforms #DataEngineering #AIStrategy #technology #ExecutiveLeadership

AWS cloud data loss in Bahrain and UAE raises resilience questions https://www.cloudcomputing-news.net/news/aws-cloud-data-loss-bahrain-uae-resilience/?utm_source=dlvr.it&utm_medium=mastodon #Cloud #Automation #Data #DataPlatforms #DataEngineering #AIStrategy #technology #ExecutiveLeadership

AWS cloud data loss in Bahrain and UAE raises resilience questions https://www.cloudcomputing-news.net/news/aws-cloud-data-loss-bahrain-uae-resilience/?utm_source=dlvr.it&utm_medium=mastodon #Cloud #Automation #Data #DataPlatforms #DataEngineering #AIStrategy #technology #ExecutiveLeadership

🚀✨ Congrats, techies! You've finally found a way to make #ClickHouse and #Postgres talk faster than a caffeinated squirrel 🐿️. Next up: convincing your boss why they should care about this monumental achievement in data yak-shaving. 🙄🔧
https://clickhouse.com/blog/introducing-walshadow #DataEngineering #TechAchievement #DataYakShaving #HackerNews #ngated

🚀✨ Congrats, techies! You've finally found a way to make #ClickHouse and #Postgres talk faster than a caffeinated squirrel 🐿️. Next up: convincing your boss why they should care about this monumental achievement in data yak-shaving. 🙄🔧
https://clickhouse.com/blog/introducing-walshadow #DataEngineering #TechAchievement #DataYakShaving #HackerNews #ngated

🚀✨ Congrats, techies! You've finally found a way to make #ClickHouse and #Postgres talk faster than a caffeinated squirrel 🐿️. Next up: convincing your boss why they should care about this monumental achievement in data yak-shaving. 🙄🔧
https://clickhouse.com/blog/introducing-walshadow #DataEngineering #TechAchievement #DataYakShaving #HackerNews #ngated

🚀✨ Congrats, techies! You've finally found a way to make #ClickHouse and #Postgres talk faster than a caffeinated squirrel 🐿️. Next up: convincing your boss why they should care about this monumental achievement in data yak-shaving. 🙄🔧
https://clickhouse.com/blog/introducing-walshadow #DataEngineering #TechAchievement #DataYakShaving #HackerNews #ngated

WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL

https://clickhouse.com/blog/introducing-walshadow

Comments: https://news.ycombinator.com/item?id=49660791

#HackerNews #WalShadow #Postgres #ClickHouse #replication #dataengineering #database #performance

WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL

https://clickhouse.com/blog/introducing-walshadow

Comments: https://news.ycombinator.com/item?id=49660791

#HackerNews #WalShadow #Postgres #ClickHouse #replication #dataengineering #database #performance

WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL

https://clickhouse.com/blog/introducing-walshadow

Comments: https://news.ycombinator.com/item?id=49660791

#HackerNews #WalShadow #Postgres #ClickHouse #replication #dataengineering #database #performance

WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL

https://clickhouse.com/blog/introducing-walshadow

Comments: https://news.ycombinator.com/item?id=49660791

#HackerNews #WalShadow #Postgres #ClickHouse #replication #dataengineering #database #performance

Zax looks like a very cool new tool. Bringing a SQL interface to high-dimensional array data in the cloud. Designed to bridge the gap between monster scientific datasets and the often tabular slices and aggregations that get used downstream.

https://www.earthmover.io/blog/compute-roadmap

#SQL #DataEngineering #Zarr #Xarray

Zax looks like a very cool new tool. Bringing a SQL interface to high-dimensional array data in the cloud. Designed to bridge the gap between monster scientific datasets and the often tabular slices and aggregations that get used downstream.

https://www.earthmover.io/blog/compute-roadmap

#SQL #DataEngineering #Zarr #Xarray

Zax looks like a very cool new tool. Bringing a SQL interface to high-dimensional array data in the cloud. Designed to bridge the gap between monster scientific datasets and the often tabular slices and aggregations that get used downstream.

https://www.earthmover.io/blog/compute-roadmap

#SQL #DataEngineering #Zarr #Xarray

Zax looks like a very cool new tool. Bringing a SQL interface to high-dimensional array data in the cloud. Designed to bridge the gap between monster scientific datasets and the often tabular slices and aggregations that get used downstream.

https://www.earthmover.io/blog/compute-roadmap

#SQL #DataEngineering #Zarr #Xarray

180 lines of Python using great-expectations to validate datasets against YAML rule sets and output reports. For data engineers who want checks in CI without building a https://www.valtersit.com/python/data-quality-rules-engine/ #python #dataengineering #greatexpectations

180 lines of Python using great-expectations to validate datasets against YAML rule sets and output reports. For data engineers who want checks in CI without building a https://www.valtersit.com/python/data-quality-rules-engine/ #python #dataengineering #greatexpectations

180 lines of Python using great-expectations to validate datasets against YAML rule sets and output reports. For data engineers who want checks in CI without building a https://www.valtersit.com/python/data-quality-rules-engine/ #python #dataengineering #greatexpectations

The real problem with enterprise data? Not multiple definitions - pretending there’s only one.

If Marketing and Finance can’t agree on what an “active customer” is, how can an AI agent?

Enterprise AI agents don’t fail because models are dumb. They fail because enterprise semantics are messy!

🔗 Watch Fabiane Nardon’s full #QConAI Boston talk: https://www.infoq.com/presentations/enterprise-data-architecture-ai-agents/

#AI #DataEngineering #EnterpriseAI #AIAgents #InfoQ

The real problem with enterprise data? Not multiple definitions - pretending there’s only one.

If Marketing and Finance can’t agree on what an “active customer” is, how can an AI agent?

Enterprise AI agents don’t fail because models are dumb. They fail because enterprise semantics are messy!

🔗 Watch Fabiane Nardon’s full #QConAI Boston talk: https://www.infoq.com/presentations/enterprise-data-architecture-ai-agents/

#AI #DataEngineering #EnterpriseAI #AIAgents #InfoQ

The real problem with enterprise data? Not multiple definitions - pretending there’s only one.

If Marketing and Finance can’t agree on what an “active customer” is, how can an AI agent?

Enterprise AI agents don’t fail because models are dumb. They fail because enterprise semantics are messy!

🔗 Watch Fabiane Nardon’s full #QConAI Boston talk: https://www.infoq.com/presentations/enterprise-data-architecture-ai-agents/

#AI #DataEngineering #EnterpriseAI #AIAgents #InfoQ

The real problem with enterprise data? Not multiple definitions - pretending there’s only one.

If Marketing and Finance can’t agree on what an “active customer” is, how can an AI agent?

Enterprise AI agents don’t fail because models are dumb. They fail because enterprise semantics are messy!

🔗 Watch Fabiane Nardon’s full #QConAI Boston talk: https://www.infoq.com/presentations/enterprise-data-architecture-ai-agents/

#AI #DataEngineering #EnterpriseAI #AIAgents #InfoQ

Stop wrestling with CSV bloat. New script converts CSVs to compressed Parquet with PyArrow, squeezing storage and speeding up your pipelines. Intermediate level, 180 lines. Try it: https://www.valtersit.com/python/parquet-file-writer-with-pyarrow/ #Python #DataEngineering

Stop wrestling with CSV bloat. New script converts CSVs to compressed Parquet with PyArrow, squeezing storage and speeding up your pipelines. Intermediate level, 180 lines. Try it: https://www.valtersit.com/python/parquet-file-writer-with-pyarrow/ #Python #DataEngineering

Stop wrestling with CSV bloat. New script converts CSVs to compressed Parquet with PyArrow, squeezing storage and speeding up your pipelines. Intermediate level, 180 lines. Try it: https://www.valtersit.com/python/parquet-file-writer-with-pyarrow/ #Python #DataEngineering