Search
1000 results for “data0”
-
The inner working of parquette/arrow data in R: https://blog.djnavarro.net/posts/2022-11-30_unpacking-arrow-datasets/ #rstats #arrow #parquette
-
The inner working of parquette/arrow data in R: https://blog.djnavarro.net/posts/2022-11-30_unpacking-arrow-datasets/ #rstats #arrow #parquette
-
The inner working of parquette/arrow data in R: https://blog.djnavarro.net/posts/2022-11-30_unpacking-arrow-datasets/ #rstats #arrow #parquette
-
The inner working of parquette/arrow data in R: https://blog.djnavarro.net/posts/2022-11-30_unpacking-arrow-datasets/ #rstats #arrow #parquette
-
Emission Repair and Private Inspection Facilities
Source: https://data.nj.gov/d/rggz-cv9v
#nj #data #dataviz -
Fabricated citations: an audit across 2·5 million biomedical papers
"We present findings from a reference-integrity audit of 2·5 million biomedical papers spanning 3 years, showing that fabricated references are embedded in the peer-reviewed literature at scale, and that the rate of fabrication is accelerating."
Topaz M, Roguin N, Gupta P et al. Fabricated citations: an audit across 2·5 million biomedical papers, The Lancet, 407, 1779-1781. DOI: https://doi.org/10.1016/S0140-6736(26)00603-3.
-
Bringing Metadata to Life at Automattic with OpenMetadata
The metadata problem at scale
Automattic’s data ecosystem is large and highly interconnected: thousands of Iceberg tables and views queried via Trino, thousands of Airflow tasks producing and updating them via Spark jobs, and a growing catalogue of Looker and Superset dashboards and charts on top.
Most of the information about these assets exists somewhere, scattered across version control (like Git), schema registries (in some metastore like HiveMetastore), and the heads of whoever built the pipeline.
Anyone working with our data eventually hits the same wall:
- Which table should I query?
- Where did this data come from?
- Was it updated recently? When?
- Who owns this pipeline?
- Has this table grown recently, and was the latest run healthy?
These questions come up repeatedly. Answering even one usually requires knowing the right system to check, or the right person to ask. That doesn’t scale. It has become a bottleneck not just for humans, but also for the AI agents we increasingly want to put in front of our data.
We tried to address this with internal documentation pages describing our tables. They helped, but two gaps remained: people still struggled to find the right asset, and there was no lineage information. The most important question, “where does this data actually come from?”, kept going unanswered.
Why OpenMetadata
After surveying metadata solutions (where metadata is essentially “data about data”) and discovery solutions (DataHub, Marquez, Amundsen, Apache Atlas, Unity, ODD, OpenMetadata, and more), we picked OpenMetadata as the platform to consolidate everything: one user interface, one API, one graph that connects tables, dashboards, owners, profiles, quality checks, and lineage.
The platform is now live internally, and we are actively ingesting metadata from Trino, Airflow, Superset, and Looker.
What it gives us
A single source of truth for discovery
A centralized UI where anyone can find tables, dashboards, charts, and metrics; explore schemas and column‑level descriptions; browse ownership; check data quality results; and navigate relationships without needing to know where to look.
End-to-end lineage
Trace how a table was produced, which Airflow job ran, which upstream tables it depends on, and which dashboards consume it. Before changing a schema, you can assess downstream impact and understand the blast radius.
Data quality in the catalog
A database catalog is a centralized, self‑describing repository storing metadata about database objects such as tables, views, columns, users, and constraints. Quality tests check results live alongside the tables they validate, so users can confirm freshness and correctness without leaving the catalog.
Live profiling
Row counts and last‑updated timestamps are captured after each pipeline run, keeping the catalog in sync with the actual state of the data. This also helps answer questions like “how much has this table grown in the last 15 days?”
Auditing and change tracking
Schema, ownership, and annotation changes are versioned for debugging and compliance. Users can also be notified of schema updates, critical for catching issues that might break dashboards.
Why this matters for AI agents
OpenMetadata is not just for humans.
An AI agent querying Trino without metadata context produces unreliable results: it cannot know which of twelve similarly named tables is the authoritative one, whether a column is still maintained, or what a given metric actually represents. At scale, this becomes more than a discovery problem. It becomes an operational risk.
Increasingly, we want AI agents and automation systems to interact directly with our data platform: generate queries, investigate incidents, validate transformations, understand lineage, and reason about data quality. Without metadata, those systems are effectively blind and more prone to AI agent hallucinations.
The same context humans rely on, ownership, lineage, freshness, semantics, and quality signals, is also what allows agents to operate safely and intelligently on top of a modern data platform.
With a rich metadata graph in place—table descriptions, column semantics, lineage, ownership, and quality results—agents can make the same informed decisions as a data engineer.
A few concrete things a metadata‑aware agent can do that a blind agent cannot:
- Resolve ambiguous table references by checking descriptions, ownership, and lineage before writing a query.
- Warn before suggesting a transformation that would break a downstream dashboard by walking the lineage graph first.
- Surface data quality failures as context (e.g., “this table failed a column check 3 days ago”), and, if the agent has access to the repository and understands the pipeline, even fix the issue.
- Identify the right owner to contact when a table needs to be dropped or its schema changed.
We are currently testing an MCP server that exposes OpenMetadata to AI agents, allowing them to search assets, fetch schemas, navigate lineage, and retrieve quality results, with semantic search for more precise answers.
Semantic search in OpenMetadata currently only works with OpenSearch. Since we run Elasticsearch, we are contributing the Elasticsearch implementation upstream so the same capability works on both backends.
How we keep the catalog fresh
Two complementary ingestion paths feed OpenMetadata, keeping the catalog continuously up to date:
- Async (DAG‑based)
An Airflow DAG (openmetadata_ingestion) crawls Trino, Airflow, Superset, and Looker on a schedule. Each source runs as a YARN task (isolated) via a reusable OpenMetadataWorkflowOperator, using the official openmetadata‑ingestion library. This keeps the broader catalog up to date without coupling ingestion to individual pipeline runs. - Sync (per‑run)
An OpenMetadataSync module integrated into the Spark task execution lifecycle posts metadata updates after every job: table schema, column descriptions, ownership, lineage, row counts from the Iceberg snapshot, and the results of any data quality checks.
High-value, frequently updated tables stay continuously fresh without waiting for the next scheduled crawl (which still captures deleted and static tables).
Where out‑of‑the‑box connectors fall short, particularly for lineage edges they cannot infer, we close the gap with custom integrations using the OpenMetadata API directly. We also noticed that some Looker and Superset lineage and ownership mappings do not work as expected and require additional development.
Where we are, and what’s next
The OpenMetadata infrastructure and async ingestion DAG were recently deployed in our internal system. Even in its early stages, the catalog is already indexing tens of thousands of data assets across our analytics ecosystem, including Looker, Trino, Superset, and Airflow.
33k+ assets already indexedThe remaining work falls into a few areas:
- Sync ingestion of per‑run metadata from Spark jobs.
- Ownership and lineage enrichment for Superset and Looker.
- Column‑level lineage via the OpenLineage Spark connector.
- MCP and agent integration to fully connect the catalog with AI workflows.
We have also started contributing fixes and improvements back to the OpenMetadata open‑source project.
If you build on top of Automattic’s data, whether as a human being or an agent, this is the layer that should make every next answer faster, safer, and easier to find.
#ai #DataLineage #DataScience #Metadata #OpenMetadata -
🐦🏙️ Researchers studied how urban environments influence bird singing patterns and regional dialects. #Data shows that city #birds sing at higher pitches and extend their daily vocal activity by nearly an hour due to artificial light.
#science #nature #biology #wildlife #urban #environment #learning #animals #ecology #research #communication #birdsong #birding #birdwatching
-
Power of #data:
I heard today in a talk that the average #salary in #Kenya should be EUR 300 - 500 ... knowing Kenya well, i could not recognize that.
The explanation? "Average salary" is measured among those with jobs in the formal sector - which is ... 17% of those in workforce.
The reality for the vast majority is way lower.Some data:
-
Power of #data:
I heard today in a talk that the average #salary in #Kenya should be EUR 300 - 500 ... knowing Kenya well, i could not recognize that.
The explanation? "Average salary" is measured among those with jobs in the formal sector - which is ... 17% of those in workforce.
The reality for the vast majority is way lower.Some data:
-
Power of #data:
I heard today in a talk that the average #salary in #Kenya should be EUR 300 - 500 ... knowing Kenya well, i could not recognize that.
The explanation? "Average salary" is measured among those with jobs in the formal sector - which is ... 17% of those in workforce.
The reality for the vast majority is way lower.Some data:
-
Power of #data:
I heard today in a talk that the average #salary in #Kenya should be EUR 300 - 500 ... knowing Kenya well, i could not recognize that.
The explanation? "Average salary" is measured among those with jobs in the formal sector - which is ... 17% of those in workforce.
The reality for the vast majority is way lower.Some data:
-
No Idling Pledge Map
Source: https://data.nj.gov/d/45a5-ngdt
#nj #data #dataviz -
Data center startup Fermi promised nuclear power for AI. But it couldn’t sign a single client
For a brief moment, no startup rode the data center boom better than Fermi Inc. Co-founded by a…
#Nuclear #Amazon #board #company #datumcenter #Fermi #gigawatt #Investor #nuclear #nuclearpower #Perry #potentialtenant #ProjectMatador #shareholder #site #statement #TobyNeugebauer
https://www.europesays.com/2974863/ -
Data Breaches Surge, Exposing Sensitive Info at AI Startups, Agencies
Data breaches are surging, with AI startups and agencies exposed, as seen in the alarming theft of 10 petabytes from a Chinese supercomputer and 4 terabytes from AI startup Mercor due to a supply-chain vulnerability. These incidents highlight the hidden risks of connecting data to AI models, creating sensitive blind spots that leave large…
-
Map of Reports by the Office of the State Comptroller in 2023
Source: https://data.nj.gov/d/hh2f-37ku
#nj #data #dataviz -
{dtrack} makes documentation of data wrangling part of the analysis and creates pretty flow charts: https://terminological.github.io/dtrackr/ #rstats #reproducibility
-
{dtrack} makes documentation of data wrangling part of the analysis and creates pretty flow charts: https://terminological.github.io/dtrackr/ #rstats #reproducibility
-
{dtrack} makes documentation of data wrangling part of the analysis and creates pretty flow charts: https://terminological.github.io/dtrackr/ #rstats #reproducibility
-
Acute-Care Facility Locations
Source: https://data.nj.gov/d/ars4-hngc
#nj #data #dataviz -
Data Extortion Groups Intensify Pressure On Global Aerospace Supply Chains
Pulse ID: 69fd6aa29449c2acb70596c9
Pulse Link: https://otx.alienvault.com/pulse/69fd6aa29449c2acb70596c9
Pulse Author: Tr1sa111
Created: 2026-05-08 04:46:26Be advised, this data is unverified and should be considered preliminary. Always do further verification.
#CyberSecurity #Extortion #InfoSec #OTX #OpenThreatExchange #SupplyChain #bot #Tr1sa111
-
Data Extortion Groups Intensify Pressure On Global Aerospace Supply Chains
Pulse ID: 69fd6aad801200fc94b242cc
Pulse Link: https://otx.alienvault.com/pulse/69fd6aad801200fc94b242cc
Pulse Author: Tr1sa111
Created: 2026-05-08 04:46:37Be advised, this data is unverified and should be considered preliminary. Always do further verification.
#CyberSecurity #Extortion #InfoSec #OTX #OpenThreatExchange #SupplyChain #bot #Tr1sa111
-
Map of Reports by the Office of the State Comptroller in 2023
Source: https://data.nj.gov/d/hh2f-37ku
#nj #data #dataviz -
Ultimate ML interpretability bundle: Interpretable Machine Learning + Interpreting Machine Learning Models With SHAP by Christoph Molnar is the featured bundle of ebooks 📚 on Leanpub!
-
Ultimate ML interpretability bundle: Interpretable Machine Learning + Interpreting Machine Learning Models With SHAP by Christoph Molnar is the featured bundle of ebooks 📚 on Leanpub!
-
Ultimate ML interpretability bundle: Interpretable Machine Learning + Interpreting Machine Learning Models With SHAP by Christoph Molnar is the featured bundle of ebooks 📚 on Leanpub!
-
Map of Accredited NJ Career and Technical Institutions
Source: https://data.nj.gov/d/jmk4-knzi
#nj #data #dataviz -
You've gathered data. Now what? This webinar demystifies data analysis and shows you how to turn numbers into compelling advocacy tools.
The third and final webinar in the series, Monitoring What Matters: Making Sense of the Squiggly Lines — Understanding and Using Your Monitoring Data is coming up next week on May 13, 1:00 - 2:00 EDT. Maddie Cooper will be presenting. There is still time to register for this free webinar. 👇
https://us06web.zoom.us/webinar/register/WN_RmlzDuBkRY-i8QDRQJTuDQ?hsCtaAttrib=205776071827#/registration #globalmuseum #webinar #data -
The Trump administration is deleting government data. From infant deaths to hunger, here are 5 ways it’s hurting Americans
This information was used to understand the problems Americans face. The consequences of its erasure, experts warn, could affect generations to come.
#GovernmentData #information #data
https://www.theguardian.com/us-news/ng-interactive/2026/may/07/trump-administration-deleting-data
-
Here is the #visualization that I was writing about yesterday. It's showing data by the #UN about immigrants/emigrants to/from all countries to/from all countries in 5-year intervals.
What do you think?