#data-engineering — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #data-engineering, aggregated by home.social.
-
Radim Marek's twelve-month AlloyDB evaluation is the most careful public writeup of it so far, and it is not a hit piece.
His summary: "AlloyDB's compatibility claim holds at the wire protocol, but the underlying engine diverges immediately at the storage layer." In managed AlloyDB the 8KB page is no longer the durable representation of your data; the WAL is, and pages are derived from it.
-
Tencent and Alibaba expand APAC cloud infrastructure https://www.cloudcomputing-news.net/news/tencent-alibaba-apac-cloud-infrastructure/?utm_source=dlvr.it&utm_medium=mastodon #Cloud #Automation #Data #AIStrategy #DataEngineering #DataScience #AgenticAI #EnterpriseAI
-
Two habits keep our AI data pipeline bills sane: batch 20 to 30 items per LLM call instead of one-at-a-time, and use change detection so nothing gets re-enriched unless the source actually changed. https://go.upgradejs.com/e7w #AI #DataEngineering #LLM
-
Publications sur les Data Engineer Junior / ingénierie de données pour les juniors : 20k+ vues sur LinkedIn en quelques jours... Faisons du réseau ! Pas de conseils à vendre, juste échanger.
#DataEngineer #DataEngineering #Junior #Recrutement #RH #Senior #OpenToWork #LinkedIn !
-
New on the Psycopg blog: row-by-row streaming with server-side cursors 🐘🐍
One of the most common mistakes with large PostgreSQL datasets is calling fetchall() and loading everything into memory. Server-side cursors solve this cleanly: stream results row by row, control batch size, and use it with async too.
We walk through how it works, a practical export example, and what to watch out for. Link in replies 👇
#PostgreSQL #Python #Psycopg #OpenSource #FOSS #DataEngineering
-
Microsoft opens fourth cloud region in India https://www.cloudcomputing-news.net/news/microsoft-cloud-region-india-hyderabad/?utm_source=dlvr.it&utm_medium=mastodon #Cloud #Automation #Data #DataEngineering #CTO #ExecutiveLeadership #BusinessStrategy #DataScience
-
Learn how to install OpenMetadata on Ubuntu using Docker Compose. Deploy the complete stack with Airflow, PostgreSQL, and Elasticsearch.
Full guide here: https://ostechnix.com/install-openmetadata-ubuntu-docker-compose/
#Openmetadata #Metadata #DataGovernance #DataCatalog #Dataops #Devops #DataEngineering #Ubuntu #Postgresql #Docker #DockerCompose #Elasticsearch #ApacheAirflow
-
New partner onboarding used to eat days of dev time. We rebuilt it as an AI pipeline where the LLM drafts the mapping and deterministic code validates every output before it ships. Same job, minutes of compute instead of days of debugging. Proposal engine, not decision maker.
-
Next Wednesday: the engineer who built ColdFront live on the architecture.
Most #Postgres databases pay SSD prices for data nobody queries. ColdFront gives you #PostgreSQL to #ApacheIceberg using the same #SQL and the same table names with writable cold tier.
On August 19, 8 AM PST, catch @vyruss on the engineering + TLA+-verified distributed writes. Paul Rothrock will give a live demo.
Q&A at the end.
📅 https://us02web.zoom.us/webinar/register/WN_3oRFbue0QL2UI7j9DmCbFA
-
This Microsoft article explains how Azure HorizonDB can support AI-powered retrieval directly inside PostgreSQL by combining operational data, vector search, keyword search, filters, indexing, and relevance ranking in one platform.
#PostgreSQL #AzureHorizonDB #MicrosoftAzure #ArtificialIntelligence #GenerativeAI #RAG #VectorSearch #DatabaseArchitecture #DataEngineering
https://techcommunity.microsoft.com/blog/adforpostgresql/ai-powered-retrieval-in-postgresql-with-azure-horizondb/4540160?wt.mc_id=DP-MVP-4015656 -
Are you interested in understanding how Microsoft Fabric brings data engineering, analytics, data science, and AI together in one platform? Take this free course.
#MicrosoftFabric #DataAnalytics #DataEngineering #DataScience #ArtificialIntelligence #MicrosoftLearn #Learn -
Avrotize/Structurize schema conversion gallery #datascience #data #dataengineering
-
Snowflake's pg_lake is now GA, and its data mirroring preview runs logical decoding inside Postgres via a snowflake_cdc extension rather than streaming to a consumer that knows nothing about the server.
Better engineering. Also a tighter coupling.
https://www.snowflake.com/en/blog/engineering/postgres-to-snowflake-replication-mirroring/
-
The underrated part of an AI data pipeline is the trash filter. Classification catches CTAs and promo junk before anything hits the database, and every human correction feeds back into the extraction configs. The pipeline gets better with use instead of rotting. https://go.upgradejs.com/dxi #AI #DataPipelines #DataEngineering
-
Andy Pavlo Joins ClickHouse to Establish ClickHouse Labs
https://clickhouse.com/blog/andy-pavlo-joins-clickhouse
Comments: https://news.ycombinator.com/item?id=49156011
#HackerNews #AndyPavlo #ClickHouse #ClickHouseLabs #TechNews #DataEngineering
-
Spec-Driven Data Engineering turns business rules, schemas, validation, and orchestration into versioned contracts that guide AI coding agents. https://hackernoon.com/why-ai-assisted-data-engineering-needs-executable-specifications #dataengineering
-
🏅 Feliz de compartir la certificación Google Cloud Certified – Associate Data Practitioner (Certificación de Profesional Asociado en Datos) de Google Cloud.
☁️Credencial: https://www.credly.com/badges/a5332290-8e51-4bc2-897a-9165af03c203
#GoogleCloud #GoogleCloudCertified #DataEngineering #CloudComputing #DataAnalytics #Google #GCP
-
🏅 Feliz de compartir la certificación Google Cloud Certified – Associate Data Practitioner (Certificación de Profesional Asociado en Datos) de Google Cloud.
☁️Credencial: https://www.credly.com/badges/a5332290-8e51-4bc2-897a-9165af03c203
#GoogleCloud #GoogleCloudCertified #DataEngineering #CloudComputing #DataAnalytics #Google #GCP
-
Indexing the Data Lake for Online Point Queries
https://engineering.atspotify.com/2026/7/indexing-the-data-lake-for-online-point-queries
Comments: https://news.ycombinator.com/item?id=49135383
#HackerNews #Indexing #Data #Lake #Online #Queries #DataEngineering #Spotify
-
Every CDC pipeline you run pulls a logical decoding stream from outside the server. Snowflake's new mirroring inverts it: an in-server extension (snowflake_cdc) pushes batches into Iceberg change logs plus a meta log, and the target replays that log as a state machine.
The interesting claim is about DDL sequencing, not throughput.
https://www.snowflake.com/en/blog/engineering/postgres-to-snowflake-replication-mirroring/
-
Tired of Big Tech's unpredictable cloud egress fees? 💸☁️
While storing data in the public cloud is cheap, moving your AI datasets or Kubernetes workloads costs a fortune. The smartest move in 2026? Self-hosting your own S3-compatible Object Storage!
We just published a step-by-step guide on deploying a secure, high-performance #MinIO server on #Ubuntu 24.04.
https://www.servers99.com/tutorials/howto/install-minio-ubuntu-24-04/#SelfHosting #Linux #DevOps #CloudComputing #SysAdmin #OpenSource #DataEngineering
-
Bridging Enterprise Scale and Local AI: Why I’ve Been Testing Gemma 4 12B Coder
https://atalupadhyay.wordpress.com/2026/07/31/running-gemma-4-12b-coder-locally/
#DataAnalyst #DataEngineering #Databricks #DatabricksCertifiedInstructor #Gemma412BCoder #GenerativeAI #LocalLLM #MachineLearningEngineer #Ollama -
SapixDB calls itself the world's first agent-native database — where every data domain is owned by an AI agent, every record is cryptographically signed and hash-linked, and schemas evolve under human governance. No migrations. No DBA gates. Built from the ground up for AI agents. #Database #AI #DataEngineering
-
Whether you’re migrating to Airflow 3, scaling data pipelines, bringing AI and ML workloads into production, or defining your organization’s data strategy, you’ll leave with ideas you can put into practice.
📅 August 31–September 2, 2026
📍 Hyatt Regency AustinDon’t just watch the AI revolution unfold. Learn how to orchestrate it.
Secure your spot: https://airflowsummit.org/tickets/
#ApacheAirflow #DataEngineering #ArtificialIntelligence #MLOps #AirflowSummit