#dataengineering — Public Fediverse posts

Two habits keep our AI data pipeline bills sane: batch 20 to 30 items per LLM call instead of one-at-a-time, and use change detection so nothing gets re-enriched unless the source actually changed. https://go.upgradejs.com/e7w #AI #DataEngineering #LLM

Two habits keep our AI data pipeline bills sane: batch 20 to 30 items per LLM call instead of one-at-a-time, and use change detection so nothing gets re-enriched unless the source actually changed. https://go.upgradejs.com/e7w #AI #DataEngineering #LLM

Publications sur les Data Engineer Junior / ingénierie de données pour les juniors : 20k+ vues sur LinkedIn en quelques jours... Faisons du réseau ! Pas de conseils à vendre, juste échanger.

#DataEngineer #DataEngineering #Junior #Recrutement #RH #Senior #OpenToWork #LinkedIn !

New on the Psycopg blog: row-by-row streaming with server-side cursors 🐘🐍

One of the most common mistakes with large PostgreSQL datasets is calling fetchall() and loading everything into memory. Server-side cursors solve this cleanly: stream results row by row, control batch size, and use it with async too.

We walk through how it works, a practical export example, and what to watch out for. Link in replies 👇

#PostgreSQL #Python #Psycopg #OpenSource #FOSS #DataEngineering

New on the Psycopg blog: row-by-row streaming with server-side cursors 🐘🐍

One of the most common mistakes with large PostgreSQL datasets is calling fetchall() and loading everything into memory. Server-side cursors solve this cleanly: stream results row by row, control batch size, and use it with async too.

We walk through how it works, a practical export example, and what to watch out for. Link in replies 👇

#PostgreSQL #Python #Psycopg #OpenSource #FOSS #DataEngineering

Microsoft opens fourth cloud region in India https://www.cloudcomputing-news.net/news/microsoft-cloud-region-india-hyderabad/?utm_source=dlvr.it&utm_medium=mastodon #Cloud #Automation #Data #DataEngineering #CTO #ExecutiveLeadership #BusinessStrategy #DataScience

Microsoft opens fourth cloud region in India https://www.cloudcomputing-news.net/news/microsoft-cloud-region-india-hyderabad/?utm_source=dlvr.it&utm_medium=mastodon #Cloud #Automation #Data #DataEngineering #CTO #ExecutiveLeadership #BusinessStrategy #DataScience

How We Pushed CDC into Postgres

https://www.snowflake.com/en/blog/engineering/postgres-to-snowflake-replication-mirroring/

Comments: https://news.ycombinator.com/item?id=49238050

#HackerNews #Postgres #Snowflake #Replication #DataEngineering #TechInnovation

How We Pushed CDC into Postgres

https://www.snowflake.com/en/blog/engineering/postgres-to-snowflake-replication-mirroring/

Comments: https://news.ycombinator.com/item?id=49238050

#HackerNews #Postgres #Snowflake #Replication #DataEngineering #TechInnovation

Learn how to install OpenMetadata on Ubuntu using Docker Compose. Deploy the complete stack with Airflow, PostgreSQL, and Elasticsearch.

Full guide here: https://ostechnix.com/install-openmetadata-ubuntu-docker-compose/

#Openmetadata #Metadata #DataGovernance #DataCatalog #Dataops #Devops #DataEngineering #Ubuntu #Postgresql #Docker #DockerCompose #Elasticsearch #ApacheAirflow

Learn how to install OpenMetadata on Ubuntu using Docker Compose. Deploy the complete stack with Airflow, PostgreSQL, and Elasticsearch.

Full guide here: https://ostechnix.com/install-openmetadata-ubuntu-docker-compose/

#Openmetadata #Metadata #DataGovernance #DataCatalog #Dataops #Devops #DataEngineering #Ubuntu #Postgresql #Docker #DockerCompose #Elasticsearch #ApacheAirflow

New partner onboarding used to eat days of dev time. We rebuilt it as an AI pipeline where the LLM drafts the mapping and deterministic code validates every output before it ships. Same job, minutes of compute instead of days of debugging. Proposal engine, not decision maker.

https://go.upgradejs.com/uey

#AI #DataPipelines #LLM #DataEngineering

New partner onboarding used to eat days of dev time. We rebuilt it as an AI pipeline where the LLM drafts the mapping and deterministic code validates every output before it ships. Same job, minutes of compute instead of days of debugging. Proposal engine, not decision maker.

https://go.upgradejs.com/uey

#AI #DataPipelines #LLM #DataEngineering

Next Wednesday: the engineer who built ColdFront live on the architecture.

Most #Postgres databases pay SSD prices for data nobody queries. ColdFront gives you #PostgreSQL to #ApacheIceberg using the same #SQL and the same table names with writable cold tier.

On August 19, 8 AM PST, catch @vyruss on the engineering + TLA+-verified distributed writes. Paul Rothrock will give a live demo.

Q&A at the end.

📅 https://us02web.zoom.us/webinar/register/WN_3oRFbue0QL2UI7j9DmCbFA

🔗 https://github.com/pgEdge/ColdFront

#DuckDB #DataEngineering #OpenSource

Next Wednesday: the engineer who built ColdFront live on the architecture.

Most #Postgres databases pay SSD prices for data nobody queries. ColdFront gives you #PostgreSQL to #ApacheIceberg using the same #SQL and the same table names with writable cold tier.

On August 19, 8 AM PST, catch @vyruss on the engineering + TLA+-verified distributed writes. Paul Rothrock will give a live demo.

Q&A at the end.

📅 https://us02web.zoom.us/webinar/register/WN_3oRFbue0QL2UI7j9DmCbFA

🔗 https://github.com/pgEdge/ColdFront

#DuckDB #DataEngineering #OpenSource

This Microsoft article explains how Azure HorizonDB can support AI-powered retrieval directly inside PostgreSQL by combining operational data, vector search, keyword search, filters, indexing, and relevance ranking in one platform.
#PostgreSQL #AzureHorizonDB #MicrosoftAzure #ArtificialIntelligence #GenerativeAI #RAG #VectorSearch #DatabaseArchitecture #DataEngineering
https://techcommunity.microsoft.com/blog/adforpostgresql/ai-powered-retrieval-in-postgresql-with-azure-horizondb/4540160?wt.mc_id=DP-MVP-4015656

This Microsoft article explains how Azure HorizonDB can support AI-powered retrieval directly inside PostgreSQL by combining operational data, vector search, keyword search, filters, indexing, and relevance ranking in one platform.
#PostgreSQL #AzureHorizonDB #MicrosoftAzure #ArtificialIntelligence #GenerativeAI #RAG #VectorSearch #DatabaseArchitecture #DataEngineering
https://techcommunity.microsoft.com/blog/adforpostgresql/ai-powered-retrieval-in-postgresql-with-azure-horizondb/4540160?wt.mc_id=DP-MVP-4015656

Are you interested in understanding how Microsoft Fabric brings data engineering, analytics, data science, and AI together in one platform? Take this free course.
#MicrosoftFabric #DataAnalytics #DataEngineering #DataScience #ArtificialIntelligence #MicrosoftLearn #Learn

https://learn.microsoft.com/en-us/training/modules/introduction-end-analytics-use-microsoft-fabric/?wt.mc_id=DP-MVP-4015656

Are you interested in understanding how Microsoft Fabric brings data engineering, analytics, data science, and AI together in one platform? Take this free course.
#MicrosoftFabric #DataAnalytics #DataEngineering #DataScience #ArtificialIntelligence #MicrosoftLearn #Learn

https://learn.microsoft.com/en-us/training/modules/introduction-end-analytics-use-microsoft-fabric/?wt.mc_id=DP-MVP-4015656

Avrotize/Structurize schema conversion gallery #datascience #data #dataengineering

https://avrotize.com/gallery/

Avrotize/Structurize schema conversion gallery #datascience #data #dataengineering

https://avrotize.com/gallery/

Snowflake's pg_lake is now GA, and its data mirroring preview runs logical decoding inside Postgres via a snowflake_cdc extension rather than streaming to a consumer that knows nothing about the server.

Better engineering. Also a tighter coupling.

https://www.snowflake.com/en/blog/engineering/postgres-to-snowflake-replication-mirroring/

https://pgexperts.com

#PostgreSQL #DataEngineering

Snowflake's pg_lake is now GA, and its data mirroring preview runs logical decoding inside Postgres via a snowflake_cdc extension rather than streaming to a consumer that knows nothing about the server.

Better engineering. Also a tighter coupling.

https://www.snowflake.com/en/blog/engineering/postgres-to-snowflake-replication-mirroring/

https://pgexperts.com

#PostgreSQL #DataEngineering

The underrated part of an AI data pipeline is the trash filter. Classification catches CTAs and promo junk before anything hits the database, and every human correction feeds back into the extraction configs. The pipeline gets better with use instead of rotting. https://go.upgradejs.com/dxi #AI #DataPipelines #DataEngineering

The underrated part of an AI data pipeline is the trash filter. Classification catches CTAs and promo junk before anything hits the database, and every human correction feeds back into the extraction configs. The pipeline gets better with use instead of rotting. https://go.upgradejs.com/dxi #AI #DataPipelines #DataEngineering

Andy Pavlo Joins ClickHouse to Establish ClickHouse Labs

https://clickhouse.com/blog/andy-pavlo-joins-clickhouse

Comments: https://news.ycombinator.com/item?id=49156011

#HackerNews #AndyPavlo #ClickHouse #ClickHouseLabs #TechNews #DataEngineering

Andy Pavlo Joins ClickHouse to Establish ClickHouse Labs

https://clickhouse.com/blog/andy-pavlo-joins-clickhouse

Comments: https://news.ycombinator.com/item?id=49156011

#HackerNews #AndyPavlo #ClickHouse #ClickHouseLabs #TechNews #DataEngineering

Spec-Driven Data Engineering turns business rules, schemas, validation, and orchestration into versioned contracts that guide AI coding agents. https://hackernoon.com/why-ai-assisted-data-engineering-needs-executable-specifications #dataengineering

Spec-Driven Data Engineering turns business rules, schemas, validation, and orchestration into versioned contracts that guide AI coding agents. https://hackernoon.com/why-ai-assisted-data-engineering-needs-executable-specifications #dataengineering

🏅 Feliz de compartir la certificación Google Cloud Certified – Associate Data Practitioner (Certificación de Profesional Asociado en Datos) de Google Cloud.

☁️Credencial: https://www.credly.com/badges/a5332290-8e51-4bc2-897a-9165af03c203

#GoogleCloud #GoogleCloudCertified #DataEngineering #CloudComputing #DataAnalytics #Google #GCP

🏅 Feliz de compartir la certificación Google Cloud Certified – Associate Data Practitioner (Certificación de Profesional Asociado en Datos) de Google Cloud.

☁️Credencial: https://www.credly.com/badges/a5332290-8e51-4bc2-897a-9165af03c203

#GoogleCloud #GoogleCloudCertified #DataEngineering #CloudComputing #DataAnalytics #Google #GCP

🏅 Feliz de compartir la certificación Google Cloud Certified – Associate Data Practitioner (Certificación de Profesional Asociado en Datos) de Google Cloud.

☁️Credencial: https://www.credly.com/badges/a5332290-8e51-4bc2-897a-9165af03c203

#GoogleCloud #GoogleCloudCertified #DataEngineering #CloudComputing #DataAnalytics #Google #GCP

🏅 Feliz de compartir la certificación Google Cloud Certified – Associate Data Practitioner (Certificación de Profesional Asociado en Datos) de Google Cloud.

☁️Credencial: https://www.credly.com/badges/a5332290-8e51-4bc2-897a-9165af03c203

#GoogleCloud #GoogleCloudCertified #DataEngineering #CloudComputing #DataAnalytics #Google #GCP

Indexing the Data Lake for Online Point Queries

https://engineering.atspotify.com/2026/7/indexing-the-data-lake-for-online-point-queries

Comments: https://news.ycombinator.com/item?id=49135383

#HackerNews #Indexing #Data #Lake #Online #Queries #DataEngineering #Spotify

Indexing the Data Lake for Online Point Queries

https://engineering.atspotify.com/2026/7/indexing-the-data-lake-for-online-point-queries

Comments: https://news.ycombinator.com/item?id=49135383

#HackerNews #Indexing #Data #Lake #Online #Queries #DataEngineering #Spotify

Every CDC pipeline you run pulls a logical decoding stream from outside the server. Snowflake's new mirroring inverts it: an in-server extension (snowflake_cdc) pushes batches into Iceberg change logs plus a meta log, and the target replays that log as a state machine.

The interesting claim is about DDL sequencing, not throughput.

https://www.snowflake.com/en/blog/engineering/postgres-to-snowflake-replication-mirroring/

https://pgexperts.com

#PostgreSQL #DataEngineering #CDC

Every CDC pipeline you run pulls a logical decoding stream from outside the server. Snowflake's new mirroring inverts it: an in-server extension (snowflake_cdc) pushes batches into Iceberg change logs plus a meta log, and the target replays that log as a state machine.

The interesting claim is about DDL sequencing, not throughput.

https://www.snowflake.com/en/blog/engineering/postgres-to-snowflake-replication-mirroring/

https://pgexperts.com

#PostgreSQL #DataEngineering #CDC

Tired of Big Tech's unpredictable cloud egress fees? 💸☁️

While storing data in the public cloud is cheap, moving your AI datasets or Kubernetes workloads costs a fortune. The smartest move in 2026? Self-hosting your own S3-compatible Object Storage!

We just published a step-by-step guide on deploying a secure, high-performance #MinIO server on #Ubuntu 24.04.
https://www.servers99.com/tutorials/howto/install-minio-ubuntu-24-04/

#SelfHosting #Linux #DevOps #CloudComputing #SysAdmin #OpenSource #DataEngineering

Tired of Big Tech's unpredictable cloud egress fees? 💸☁️

While storing data in the public cloud is cheap, moving your AI datasets or Kubernetes workloads costs a fortune. The smartest move in 2026? Self-hosting your own S3-compatible Object Storage!

We just published a step-by-step guide on deploying a secure, high-performance #MinIO server on #Ubuntu 24.04.
https://www.servers99.com/tutorials/howto/install-minio-ubuntu-24-04/

#SelfHosting #Linux #DevOps #CloudComputing #SysAdmin #OpenSource #DataEngineering

Bridging Enterprise Scale and Local AI: Why I’ve Been Testing Gemma 4 12B Coder

https://atalupadhyay.wordpress.com/2026/07/31/running-gemma-4-12b-coder-locally/

Bridging Enterprise Scale and Local AI: Why I’ve Been Testing Gemma 4 12B Coder

https://atalupadhyay.wordpress.com/2026/07/31/running-gemma-4-12b-coder-locally/

SapixDB calls itself the world's first agent-native database — where every data domain is owned by an AI agent, every record is cryptographically signed and hash-linked, and schemas evolve under human governance. No migrations. No DBA gates. Built from the ground up for AI agents. #Database #AI #DataEngineering

#Database #AI #DataEngineering

SapixDB calls itself the world's first agent-native database — where every data domain is owned by an AI agent, every record is cryptographically signed and hash-linked, and schemas evolve under human governance. No migrations. No DBA gates. Built from the ground up for AI agents. #Database #AI #DataEngineering

#Database #AI #DataEngineering

How to Secure a Linux VPS Environment for Open Business Applications https://www.cloudcomputing-news.net/news/how-to-secure-a-linux-vps-environment-for-open-business-applications/?utm_source=dlvr.it&utm_medium=mastodon #Cloud #Automation #Data #AgenticAI #ExecutiveLeadership #Tech #Innovation #DataEngineering

Whether you’re migrating to Airflow 3, scaling data pipelines, bringing AI and ML workloads into production, or defining your organization’s data strategy, you’ll leave with ideas you can put into practice.

📅 August 31–September 2, 2026
📍 Hyatt Regency Austin

Don’t just watch the AI revolution unfold. Learn how to orchestrate it.

Secure your spot: https://airflowsummit.org/tickets/

#ApacheAirflow #DataEngineering #ArtificialIntelligence #MLOps #AirflowSummit

Whether you’re migrating to Airflow 3, scaling data pipelines, bringing AI and ML workloads into production, or defining your organization’s data strategy, you’ll leave with ideas you can put into practice.

📅 August 31–September 2, 2026
📍 Hyatt Regency Austin

Don’t just watch the AI revolution unfold. Learn how to orchestrate it.

Secure your spot: https://airflowsummit.org/tickets/

#ApacheAirflow #DataEngineering #ArtificialIntelligence #MLOps #AirflowSummit

Earlier this year, Daniele gave a talk at @pgconfdev in Vancouver titled "Psycopg: 20 years of mostly friendly coexistence with libpq" ❤️

Two decades of Python and PostgreSQL working together and there are still stories worth telling. The recording is now available to watch online. Link in replies 👇

#PostgreSQL #Python #Psycopg #OpenSource #FOSS #PGConf #DataEngineering

Earlier this year, Daniele gave a talk at @pgconfdev in Vancouver titled "Psycopg: 20 years of mostly friendly coexistence with libpq" ❤️

Two decades of Python and PostgreSQL working together and there are still stories worth telling. The recording is now available to watch online. Link in replies 👇

#PostgreSQL #Python #Psycopg #OpenSource #FOSS #PGConf #DataEngineering

Data products overview: "The Modern Data-Stack: Open-Source Edition", https://www.datafold.com/blog/the-modern-data-stack-open-source-edition/

#DataEngineering

Data products overview: "The Modern Data-Stack: Open-Source Edition", https://www.datafold.com/blog/the-modern-data-stack-open-source-edition/

#DataEngineering

How to Get Started with Databricks Free Edition (No Cloud Account Required)

Getting started with Databricks doesn’t require a corporate cloud subscription or a credit card. Databricks replaced its legacy Community Edition with the updated Databricks Free Edition, giving students, hobbyists, and practitioners free access to build AI applications, run notebooks, and learn real-world data engineering tools.

How to Access Databricks Free Edition

1. Go to the Free Edition PageHead directly to databricks.com/learn/free-edition.
2. Sign UpClick “Sign up for Free Edition”. Enter your basic details (name, email, and company/school).> Tip: You do not need a business email or a cloud account (AWS/Azure/GCP) to sign up.
3. Verify Your EmailCheck your inbox for a verification email from Databricks and click the link to set your password.
4. Launch Your WorkspaceLog in to access your free Data Intelligence Platform workspace. You can immediately create interactive Python/SQL notebooks, test LLMs via the Databricks Playground, and use the AI-powered Databricks Assistant to help you write code.

What’s Included (and What’s Not)

🛠️ pracdata/awesome-open-source-data-engineering

A curated list of open source tools used in analytics platforms and data engineering ecosystem

Compiles software tools across storage, streaming, integration, and metadata management for data architecture platforms.

⭐ Stars: 592
📅 Last Update: Jul 24, 2026

https://github.com/pracdata/awesome-open-source-data-engineering

#selfhosted #homelab #selfhost #selfhosting #opensource #dataengineering #databases