Search
1000 results for “dataplane”
-
SAP acquires Dremio to power enterprise AI data platform
SAP agreed to acquire Dremio, an open data lakehouse platform, to expand its Business Data Cloud’s ability to…
#Germany #DE #Europe #EU #Europa #SAP #DataCloud #datalakehouse #datalineage #datasources #Dremio #sap #SAPHANA
https://www.europesays.com/germany/23414/ -
https://www.europesays.com/ie/498845/ Google Cloud Introduces Cross-Engine Iceberg Support in BigQuery #AI #ApacheIceberg #Architecture&Design #Cloud #DataCatalog #DataLake #DataPortability #Éire #GoogleBigQuery #GoogleCloud #GoogleCrossEngineIceberg #IE #Ireland #ML&DataEngineering #Technology
-
https://www.europesays.com/ro/186707/ Lansarea Oura Ring 5 se apropie; Ce ne rezrvă noul inel inteligent al companiei? #DataLansare #hardware #InelInteligent #lansare #OuraRing5 #RO #Română #Romania #Romanian #Technology #tehnologie
-
https://www.europesays.com/ro/186303/ Data de lansare a jocului GTA 6 este bătută în cuie! Vine pe 19 noiembrie, iar campania de marketing începe în vară #CatCosta #DataLansare #debut #GrandTheftAuto6 #GrandTheftAutoVI #Gta6 #GTAVI #joc #lansare #pret #RO #Română #Romania #Romanian #Technology #tehnologie
-
https://www.europesays.com/people/73507/ Inside Scale AI’s Business After Meta’s Bombshell $14 Billion Deal #DataLabeling #JasonDroege #LucyGuo #MarkZuckerberg #Mercor #Meta #ScaleAI #surge
-
Google and SpaceX discuss orbital AI data centres https://www.cloudcomputing-news.net/news/google-spacex-orbital-ai-data-centres/?utm_source=dlvr.it&utm_medium=mastodon #Cloud #Automation #Data #Tech #DataPlatforms #AgenticAI #CTO #CIO
-
Google and SpaceX discuss orbital AI data centres https://www.cloudcomputing-news.net/news/google-spacex-orbital-ai-data-centres/?utm_source=dlvr.it&utm_medium=mastodon #Cloud #Automation #Data #Tech #DataPlatforms #AgenticAI #CTO #CIO
-
https://www.europesays.com/people/71653/ Inside Scale AI’s Business After Meta’s Bombshell $14 Billion Deal #DataLabeling #JasonDroege #LucyGuo #MarkZuckerberg #Mercor #Meta #ScaleAI #surge
-
Inside Scale AI’s Business After Meta’s Bombshell $14 Billion Deal
Scale AI CEO Jason Droege Ian Tuttle for Scale AI As Alexandr Wang walked down the staircase to…
#UnitedStates #US #USA #america #business #datalabeling #jasondroege #lucyguo #markzuckerberg #Mercor #Meta #scale.ai #surge #unitedstatesofamerica
https://www.europesays.com/2989396/ -
QuadTrack Amiga Sequencer Brings 4-Track Groovebox Control to Mac, PC, and Amiga
#QuadTrack #Amiga #CommodoreAmiga #PinkParrotStudio #MrDataline #Dataline #AmigaMusic #RetroComputing #Groovebox #Sequencer
https://theoasisbbs.com/quadtrack-amiga-sequencer-brings-4-track-groovebox-control-to-mac-pc-and-amiga/?fsp_sid=6611 -
QuadTrack Amiga Sequencer Brings 4-Track Groovebox Control to Mac, PC, and Amiga
#QuadTrack #Amiga #CommodoreAmiga #PinkParrotStudio #MrDataline #Dataline #AmigaMusic #RetroComputing #Groovebox #Sequencer
https://theoasisbbs.com/quadtrack-amiga-sequencer-brings-4-track-groovebox-control-to-mac-pc-and-amiga/?fsp_sid=6611 -
QuadTrack Amiga Sequencer Brings 4-Track Groovebox Control to Mac, PC, and Amiga
#QuadTrack #Amiga #CommodoreAmiga #PinkParrotStudio #MrDataline #Dataline #AmigaMusic #RetroComputing #Groovebox #Sequencer
https://theoasisbbs.com/quadtrack-amiga-sequencer-brings-4-track-groovebox-control-to-mac-pc-and-amiga/?fsp_sid=6611 -
QuadTrack Amiga Sequencer Brings 4-Track Groovebox Control to Mac, PC, and Amiga
#QuadTrack #Amiga #CommodoreAmiga #PinkParrotStudio #MrDataline #Dataline #AmigaMusic #RetroComputing #Groovebox #Sequencer
https://theoasisbbs.com/quadtrack-amiga-sequencer-brings-4-track-groovebox-control-to-mac-pc-and-amiga/?fsp_sid=6611 -
QuadTrack Amiga Sequencer Brings 4-Track Groovebox Control to Mac, PC, and Amiga
#QuadTrack #Amiga #CommodoreAmiga #PinkParrotStudio #MrDataline #Dataline #AmigaMusic #RetroComputing #Groovebox #Sequencer
https://theoasisbbs.com/quadtrack-amiga-sequencer-brings-4-track-groovebox-control-to-mac-pc-and-amiga/?fsp_sid=6611 -
AWS expands Anthropic partnership with Claude Platform launch https://www.cloudcomputing-news.net/news/claude-platform-on-aws-anthropic-partnership/?utm_source=dlvr.it&utm_medium=mastodon #Cloud #Automation #Data #ExecutiveLeadership #DataPlatforms #Tech #DataScience #CloudComputing
-
Bringing Metadata to Life at Automattic with OpenMetadata
The metadata problem at scale
Automattic’s data ecosystem is large and highly interconnected: thousands of Iceberg tables and views queried via Trino, thousands of Airflow tasks producing and updating them via Spark jobs, and a growing catalogue of Looker and Superset dashboards and charts on top.
Most of the information about these assets exists somewhere, scattered across version control (like Git), schema registries (in some metastore like HiveMetastore), and the heads of whoever built the pipeline.
Anyone working with our data eventually hits the same wall:
- Which table should I query?
- Where did this data come from?
- Was it updated recently? When?
- Who owns this pipeline?
- Has this table grown recently, and was the latest run healthy?
These questions come up repeatedly. Answering even one usually requires knowing the right system to check, or the right person to ask. That doesn’t scale. It has become a bottleneck not just for humans, but also for the AI agents we increasingly want to put in front of our data.
We tried to address this with internal documentation pages describing our tables. They helped, but two gaps remained: people still struggled to find the right asset, and there was no lineage information. The most important question, “where does this data actually come from?”, kept going unanswered.
Why OpenMetadata
After surveying metadata solutions (where metadata is essentially “data about data”) and discovery solutions (DataHub, Marquez, Amundsen, Apache Atlas, Unity, ODD, OpenMetadata, and more), we picked OpenMetadata as the platform to consolidate everything: one user interface, one API, one graph that connects tables, dashboards, owners, profiles, quality checks, and lineage.
The platform is now live internally, and we are actively ingesting metadata from Trino, Airflow, Superset, and Looker.
What it gives us
A single source of truth for discovery
A centralized UI where anyone can find tables, dashboards, charts, and metrics; explore schemas and column‑level descriptions; browse ownership; check data quality results; and navigate relationships without needing to know where to look.
End-to-end lineage
Trace how a table was produced, which Airflow job ran, which upstream tables it depends on, and which dashboards consume it. Before changing a schema, you can assess downstream impact and understand the blast radius.
Data quality in the catalog
A database catalog is a centralized, self‑describing repository storing metadata about database objects such as tables, views, columns, users, and constraints. Quality tests check results live alongside the tables they validate, so users can confirm freshness and correctness without leaving the catalog.
Live profiling
Row counts and last‑updated timestamps are captured after each pipeline run, keeping the catalog in sync with the actual state of the data. This also helps answer questions like “how much has this table grown in the last 15 days?”
Auditing and change tracking
Schema, ownership, and annotation changes are versioned for debugging and compliance. Users can also be notified of schema updates, critical for catching issues that might break dashboards.
Why this matters for AI agents
OpenMetadata is not just for humans.
An AI agent querying Trino without metadata context produces unreliable results: it cannot know which of twelve similarly named tables is the authoritative one, whether a column is still maintained, or what a given metric actually represents. At scale, this becomes more than a discovery problem. It becomes an operational risk.
Increasingly, we want AI agents and automation systems to interact directly with our data platform: generate queries, investigate incidents, validate transformations, understand lineage, and reason about data quality. Without metadata, those systems are effectively blind and more prone to AI agent hallucinations.
The same context humans rely on, ownership, lineage, freshness, semantics, and quality signals, is also what allows agents to operate safely and intelligently on top of a modern data platform.
With a rich metadata graph in place—table descriptions, column semantics, lineage, ownership, and quality results—agents can make the same informed decisions as a data engineer.
A few concrete things a metadata‑aware agent can do that a blind agent cannot:
- Resolve ambiguous table references by checking descriptions, ownership, and lineage before writing a query.
- Warn before suggesting a transformation that would break a downstream dashboard by walking the lineage graph first.
- Surface data quality failures as context (e.g., “this table failed a column check 3 days ago”), and, if the agent has access to the repository and understands the pipeline, even fix the issue.
- Identify the right owner to contact when a table needs to be dropped or its schema changed.
We are currently testing an MCP server that exposes OpenMetadata to AI agents, allowing them to search assets, fetch schemas, navigate lineage, and retrieve quality results, with semantic search for more precise answers.
Semantic search in OpenMetadata currently only works with OpenSearch. Since we run Elasticsearch, we are contributing the Elasticsearch implementation upstream so the same capability works on both backends.
How we keep the catalog fresh
Two complementary ingestion paths feed OpenMetadata, keeping the catalog continuously up to date:
- Async (DAG‑based)
An Airflow DAG (openmetadata_ingestion) crawls Trino, Airflow, Superset, and Looker on a schedule. Each source runs as a YARN task (isolated) via a reusable OpenMetadataWorkflowOperator, using the official openmetadata‑ingestion library. This keeps the broader catalog up to date without coupling ingestion to individual pipeline runs. - Sync (per‑run)
An OpenMetadataSync module integrated into the Spark task execution lifecycle posts metadata updates after every job: table schema, column descriptions, ownership, lineage, row counts from the Iceberg snapshot, and the results of any data quality checks.
High-value, frequently updated tables stay continuously fresh without waiting for the next scheduled crawl (which still captures deleted and static tables).
Where out‑of‑the‑box connectors fall short, particularly for lineage edges they cannot infer, we close the gap with custom integrations using the OpenMetadata API directly. We also noticed that some Looker and Superset lineage and ownership mappings do not work as expected and require additional development.
Where we are, and what’s next
The OpenMetadata infrastructure and async ingestion DAG were recently deployed in our internal system. Even in its early stages, the catalog is already indexing tens of thousands of data assets across our analytics ecosystem, including Looker, Trino, Superset, and Airflow.
33k+ assets already indexedThe remaining work falls into a few areas:
- Sync ingestion of per‑run metadata from Spark jobs.
- Ownership and lineage enrichment for Superset and Looker.
- Column‑level lineage via the OpenLineage Spark connector.
- MCP and agent integration to fully connect the catalog with AI workflows.
We have also started contributing fixes and improvements back to the OpenMetadata open‑source project.
If you build on top of Automattic’s data, whether as a human being or an agent, this is the layer that should make every next answer faster, safer, and easier to find.
#ai #DataLineage #DataScience #Metadata #OpenMetadata -
Нагрузочное тестирование динамического маскирования в Apache Ranger: что происходит с производительностью запросов
Привет, Хабр! На связи Илья Амосов из команды поддержки Lakehouse-платформы данных Data Ocean Nova вендора Data Sapience. В сегодняшней публикации я раскрою тему влияния динамического сокрытия чувствительных данных на производительность SQL-запросов. Мы сравним различные методики маскирования, узнаем, как работает оптимизатор и движок со скрытыми полями, происходит ли деградация пропускной способности платформы, как влияет на производительность выбранный метод сокрытия чувствительных данных в случаях, если вы используете компонент на базе Apache Ranger.
https://habr.com/ru/companies/datasapience/articles/1033038/
-
Нагрузочное тестирование динамического маскирования в Apache Ranger: что происходит с производительностью запросов
Привет, Хабр! На связи Илья Амосов из команды поддержки Lakehouse-платформы данных Data Ocean Nova вендора Data Sapience. В сегодняшней публикации я раскрою тему влияния динамического сокрытия чувствительных данных на производительность SQL-запросов. Мы сравним различные методики маскирования, узнаем, как работает оптимизатор и движок со скрытыми полями, происходит ли деградация пропускной способности платформы, как влияет на производительность выбранный метод сокрытия чувствительных данных в случаях, если вы используете компонент на базе Apache Ranger.
https://habr.com/ru/companies/datasapience/articles/1033038/
-
Нагрузочное тестирование динамического маскирования в Apache Ranger: что происходит с производительностью запросов
Привет, Хабр! На связи Илья Амосов из команды поддержки Lakehouse-платформы данных Data Ocean Nova вендора Data Sapience. В сегодняшней публикации я раскрою тему влияния динамического сокрытия чувствительных данных на производительность SQL-запросов. Мы сравним различные методики маскирования, узнаем, как работает оптимизатор и движок со скрытыми полями, происходит ли деградация пропускной способности платформы, как влияет на производительность выбранный метод сокрытия чувствительных данных в случаях, если вы используете компонент на базе Apache Ranger.
https://habr.com/ru/companies/datasapience/articles/1033038/
-
DuckDB Labs released #DuckLake 1.0 - a data lake format that stores table metadata in a SQL database, rather than spreading it across object storage files.
Key features:
• catalog-stored small updates
• improved sorting and partitioning
• compatibility with Iceberg-style data featuresLearn more ⇨ https://bit.ly/48PsPIS
-
DuckDB Labs released #DuckLake 1.0 - a data lake format that stores table metadata in a SQL database, rather than spreading it across object storage files.
Key features:
• catalog-stored small updates
• improved sorting and partitioning
• compatibility with Iceberg-style data featuresLearn more ⇨ https://bit.ly/48PsPIS
-
DuckDB Labs released #DuckLake 1.0 - a data lake format that stores table metadata in a SQL database, rather than spreading it across object storage files.
Key features:
• catalog-stored small updates
• improved sorting and partitioning
• compatibility with Iceberg-style data featuresLearn more ⇨ https://bit.ly/48PsPIS
-
DuckDB Labs released #DuckLake 1.0 - a data lake format that stores table metadata in a SQL database, rather than spreading it across object storage files.
Key features:
• catalog-stored small updates
• improved sorting and partitioning
• compatibility with Iceberg-style data featuresLearn more ⇨ https://bit.ly/48PsPIS
-
The last piece in the DC construction puzzle: Ongoing operations https://www.cloudcomputing-news.net/news/data-centre-personnel-as-the-last-critical-element-of-dc-operations/?utm_source=dlvr.it&utm_medium=mastodon #Cloud #Automation #Data #DataEngineering #BusinessStrategy #DigitalTransformation #DataArchitecture #DataPlatforms
-
Wat gênant, #SIMYO van #KPN, als je zo je geld moet verdienen.
Ten eerste moet je het dataplafond zelf actief aanzetten. Dat kost je 50 cent per maand.* Als je dit niet doet, dan betaal je voor elke MB buiten je bundel 15 cent (150 euro per GB).
Ten tweede werkt dit dataplafond (waar je dan dus 50 cent per maand voor betaalt) niet als je buiten de EU bent (en dus met veel hogere tarieven te maken hebt).
* = Veel mensen weten niet dat je dit ook in je telefoon zelf kan instellen.
-
Wat gênant, #SIMYO van #KPN, als je zo je geld moet verdienen.
Ten eerste moet je het dataplafond zelf actief aanzetten. Dat kost je 50 cent per maand.* Als je dit niet doet, dan betaal je voor elke MB buiten je bundel 15 cent (150 euro per GB).
Ten tweede werkt dit dataplafond (waar je dan dus 50 cent per maand voor betaalt) niet als je buiten de EU bent (en dus met veel hogere tarieven te maken hebt).
* = Veel mensen weten niet dat je dit ook in je telefoon zelf kan instellen.
-
Wat gênant, #SIMYO van #KPN, als je zo je geld moet verdienen.
Ten eerste moet je het dataplafond zelf actief aanzetten. Dat kost je 50 cent per maand.* Als je dit niet doet, dan betaal je voor elke MB buiten je bundel 15 cent (150 euro per GB).
Ten tweede werkt dit dataplafond (waar je dan dus 50 cent per maand voor betaalt) niet als je buiten de EU bent (en dus met veel hogere tarieven te maken hebt).
* = Veel mensen weten niet dat je dit ook in je telefoon zelf kan instellen.
-
Wat gênant, #SIMYO van #KPN, als je zo je geld moet verdienen.
Ten eerste moet je het dataplafond zelf actief aanzetten. Dat kost je 50 cent per maand.* Als je dit niet doet, dan betaal je voor elke MB buiten je bundel 15 cent (150 euro per GB).
Ten tweede werkt dit dataplafond (waar je dan dus 50 cent per maand voor betaalt) niet als je buiten de EU bent (en dus met veel hogere tarieven te maken hebt).
* = Veel mensen weten niet dat je dit ook in je telefoon zelf kan instellen.
-
Wat gênant, #SIMYO van #KPN, als je zo je geld moet verdienen.
Ten eerste moet je het dataplafond zelf actief aanzetten. Dat kost je 50 cent per maand.* Als je dit niet doet, dan betaal je voor elke MB buiten je bundel 15 cent (150 euro per GB).
Ten tweede werkt dit dataplafond (waar je dan dus 50 cent per maand voor betaalt) niet als je buiten de EU bent (en dus met veel hogere tarieven te maken hebt).
* = Veel mensen weten niet dat je dit ook in je telefoon zelf kan instellen.
-
https://www.europesays.com/ie/450219/ Refik Anadol’s AI arts museum, Dataland, sets opening date #AiArtMuseum #AiArtwork #anadol #Arts #ArtsAndDesign #ArtsAndDesign #ArtsDesign #dataland #datum #Design #DigitalArt #Éire #Entertainment #FirstExhibition #FirstMuseum #form #humanity #IE #Image #ImportantTechnology #Ireland #MachineDreams #part #World