#datafusion — Public Fediverse posts

Panopticon or not to Panopticon

Bentham wrote it in 1787, in letters sent from a town in what's now Belarus. Cells in a ring, inspector in the middle, lit so he sees out and you can't see in. People always repeat the seeing part. The part that mattered was that you couldn't check. Is he up there right now? No way to know, so you act like he is. That's the whole trick. It runs on doubt, and doubt is cheap, which is how one guard covers a hundred cells.

Foucault took that in 1975 and turned it into a shape for modern power generally. School, barracks, factory, ward. Two things about it: you move through enclosures one at a time, and each one is trying to correct you.

Neither is true anymore, and Deleuze said so in 1992, in five pages in October. No more enclosures. No graduation, no discharge, no sentence you finish. And the individual splits into what he called the dividual - slices, attributes, fragments that travel without you. He wrote that before the web was commercial and still described a computer tracking your position and said, flatly, that this wasn't science fiction.

Zuboff filled in the money in 2019. Behavioral surplus: the exhaust off your ordinary life, kept for uses past improving the service, milled into predictions, sold. Which inverts the point of the thing. The old apparatus watched in order to fix you. This one watches in order to bet on you.

Then there's the piece Deleuze had no name for. Something has to put the fragments back together. That's data fusion, and it's why Palantir is the name people say. In February DHS signed for up to a billion over five years, Gotham and Foundry, any component agency drawing on it by task order without competing a contract. Enforcement records, biometrics, financial history, travel, resolved into one object you can act on.

The inputs aren't new. Credit bureaus are older than I am. What's new is the schema - the thing that decides your face and your rent history are attributes of the same entity. Deleuze's fragments, glued.

So there's no tower. Nothing to photograph, nothing to zone, nothing you can usefully subpoena. And responsibility spreads until nobody's holding any of it. The location broker sells to the aggregator, the aggregator to the vendor, the vendor licenses to the agency, and not one of them assembled the file.

I'm fairly sure "who's watching" is the wrong question, though. Nobody's watching? Which is worse?

#Surveillance #Panopticon #Foucault #Deleuze #Palantir #SurveillanceCapitalism #Privacy #DataFusion #Philosophy

Panopticon or not to Panopticon

Bentham wrote it in 1787, in letters sent from a town in what's now Belarus. Cells in a ring, inspector in the middle, lit so he sees out and you can't see in. People always repeat the seeing part. The part that mattered was that you couldn't check. Is he up there right now? No way to know, so you act like he is. That's the whole trick. It runs on doubt, and doubt is cheap, which is how one guard covers a hundred cells.

Foucault took that in 1975 and turned it into a shape for modern power generally. School, barracks, factory, ward. Two things about it: you move through enclosures one at a time, and each one is trying to correct you.

Neither is true anymore, and Deleuze said so in 1992, in five pages in October. No more enclosures. No graduation, no discharge, no sentence you finish. And the individual splits into what he called the dividual - slices, attributes, fragments that travel without you. He wrote that before the web was commercial and still described a computer tracking your position and said, flatly, that this wasn't science fiction.

Zuboff filled in the money in 2019. Behavioral surplus: the exhaust off your ordinary life, kept for uses past improving the service, milled into predictions, sold. Which inverts the point of the thing. The old apparatus watched in order to fix you. This one watches in order to bet on you.

Then there's the piece Deleuze had no name for. Something has to put the fragments back together. That's data fusion, and it's why Palantir is the name people say. In February DHS signed for up to a billion over five years, Gotham and Foundry, any component agency drawing on it by task order without competing a contract. Enforcement records, biometrics, financial history, travel, resolved into one object you can act on.

The inputs aren't new. Credit bureaus are older than I am. What's new is the schema - the thing that decides your face and your rent history are attributes of the same entity. Deleuze's fragments, glued.

So there's no tower. Nothing to photograph, nothing to zone, nothing you can usefully subpoena. And responsibility spreads until nobody's holding any of it. The location broker sells to the aggregator, the aggregator to the vendor, the vendor licenses to the agency, and not one of them assembled the file.

I'm fairly sure "who's watching" is the wrong question, though. Nobody's watching? Which is worse?

#Surveillance #Panopticon #Foucault #Deleuze #Palantir #SurveillanceCapitalism #Privacy #DataFusion #Philosophy

🤔 Ah yes, the classic "I fit a billion-scale graph into 10GB RAM" tale. 🤦‍♂️ This bold adventurer claims victory over insurmountable odds by merely offloading everything to disk and praying to the #DataFusion gods—while overlooking a few pesky #deadlocks. 🚀 Sure, it "works"... if you consider "frequent deadlocks" a feature. 🙄
https://semyonsinchenko.github.io/ssinchenko/post/datafusion-graphs-cc-2/ #billionScaleGraph #RAMOptimization #techHumor #HackerNews #ngated

🤔 Ah yes, the classic "I fit a billion-scale graph into 10GB RAM" tale. 🤦‍♂️ This bold adventurer claims victory over insurmountable odds by merely offloading everything to disk and praying to the #DataFusion gods—while overlooking a few pesky #deadlocks. 🚀 Sure, it "works"... if you consider "frequent deadlocks" a feature. 🙄
https://semyonsinchenko.github.io/ssinchenko/post/datafusion-graphs-cc-2/ #billionScaleGraph #RAMOptimization #techHumor #HackerNews #ngated

Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

https://semyonsinchenko.github.io/ssinchenko/post/datafusion-graphs-cc-2/

Comments: https://news.ycombinator.com/item?id=49124658

#HackerNews #Algorithms #DataFusion #Graphs #BigData #RAM #Performance

Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

https://semyonsinchenko.github.io/ssinchenko/post/datafusion-graphs-cc-2/

Comments: https://news.ycombinator.com/item?id=49124658

#HackerNews #Algorithms #DataFusion #Graphs #BigData #RAM #Performance

Missing RustWeek and want to hear me ramble about data processing in Rust? Head on over here ;) https://youtu.be/uvfiz7-blyU

#rustlang #datafusion #duckdb

Missing RustWeek and want to hear me ramble about data processing in Rust? Head on over here ;) https://youtu.be/uvfiz7-blyU

#rustlang #datafusion #duckdb

High-Resolution Mars Surface Data Unveiled Through Data Fusion

Scientists use new data fusion methods to create sharper maps of Mars, showing frost and rock shapes better. This helps understand Mars' weather and geology.

#MarsData, #SpaceExploration, #MarsGeology, #PlanetaryScience, #DataFusion

https://newsletter.tf/mars-surface-data-fusion-frost-rock-maps/

New maps of Mars are 10 times sharper than before, showing tiny details of frost and rocks. This is a big step in understanding the planet.

#MarsData, #SpaceExploration, #MarsGeology, #PlanetaryScience, #DataFusion
https://newsletter.tf/mars-surface-data-fusion-frost-rock-maps/

I’m excited to be heading to Los Angeles next week for the Air Sensors International Conference (ASIC) 2026. I’ll be sharing new findings from our work evaluating low‑cost air sensors for data fusion.

Presentation: In-Situ Evaluation of Low-Cost PM2.5 Sensor Networks Using a Novel Distance-Based Method Quantifies Sensor Uncertainty and Reveals Episodic Variability in Performance
Session: Air Sensor Degradation and Operating in Challenging Environments
Time: Thursday, May 7th, 2026 | 1:30–3:00 PM

If you’ll be at ASIC, I’d love to connect and exchange ideas.
#ASIC2026 #airquality #lowcostsensors #airpollution #datafusion

I’m excited to be heading to Los Angeles next week for the Air Sensors International Conference (ASIC) 2026. I’ll be sharing new findings from our work evaluating low‑cost air sensors for data fusion.

Presentation: In-Situ Evaluation of Low-Cost PM2.5 Sensor Networks Using a Novel Distance-Based Method Quantifies Sensor Uncertainty and Reveals Episodic Variability in Performance
Session: Air Sensor Degradation and Operating in Challenging Environments
Time: Thursday, May 7th, 2026 | 1:30–3:00 PM

If you’ll be at ASIC, I’d love to connect and exchange ideas.
#ASIC2026 #airquality #lowcostsensors #airpollution #datafusion

@thealexmerced thanks! Added to wish list in manning. Better 2buy there vs Amazon to get the ai features?

I guess Manning got rid of old option to buy coins 2 read individual pages? was a cool feature 2 bad.

Thanks for reminder about #datafusion i guess it & #polars have excellent #iceberg support & can be used from #rust

I was thinking about replacing a #pyspark glue job with a rust #lambda on #aws

Just found your excellent medium account. Best of luck at your upcoming talk!

If you are a person working on the software development industry, please do yourself a favor and read @andygrove book: "How Query Engines work":

https://howqueryengineswork.com/

The book is free to read on web, but if you like it, I'd suggest to buy the eBook version too.

Andy Groove is the creator of #DataFusion, and has been working with query engines for a big while.

If you are a person working on the software development industry, please do yourself a favor and read @andygrove book: "How Query Engines work":

https://howqueryengineswork.com/

The book is free to read on web, but if you like it, I'd suggest to buy the eBook version too.

Andy Groove is the creator of #DataFusion, and has been working with query engines for a big while.

Как мы переписали ядро Trino на Rust

CedrusData Engine — это lakehouse-движок, основанный на Trino. На реальных нагрузках наш продукт рутинно превосходит по производительности другие технологии (Trino, Doris, Dremio, StarRocks) в 1.5-3 раза, с еще более значительным отрывом от устаревших Greenplum и Impala. Эти результаты — следствие постоянных вложений в разработку новейших техник обработки больших данных. В этой статье я расскажу про проект Oxide — одну из наших ключевых инициатив прошлого года по переписыванию ядра Trino с Java на Rust.

https://habr.com/ru/companies/cedrusdata/articles/985602/

#trino #rust #datafusion #bigdata #sql

We've been working on something exciting in the Arrow/DataFusion ecosystem, which finally shipped with yesterday's release of DataFusion. You can now use Run-End-Encoded arrays in group by clauses!

#opensource #apache #arrow #datafusion #performance #database

https://github.com/apache/datafusion/issues/16011

We've been working on something exciting in the Arrow/DataFusion ecosystem, which finally shipped with yesterday's release of DataFusion. You can now use Run-End-Encoded arrays in group by clauses!

#opensource #apache #arrow #datafusion #performance #database

https://github.com/apache/datafusion/issues/16011

Embedding User-Defined Indexes in Apache Parquet

https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes/

#HackerNews #Embedding #User-Defined #Indexes #in #Apache #Parquet #ApacheParquet #UserDefinedIndexes #DataFusion #BigData #Analytics

Embedding User-Defined Indexes in Apache Parquet

https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes/

#HackerNews #Embedding #User-Defined #Indexes #in #Apache #Parquet #ApacheParquet #UserDefinedIndexes #DataFusion #BigData #Analytics

Improvement of the disk manager configuration on #apache #datafusion by introducing a builder.

https://github.com/apache/datafusion/pull/16191

#rust #rustlang

Со скоростью кометы: ускоряем Spark без переписывания кода

Привет, Хабр! Меня зовут Лев Маковеев. Я младший инженер по обработке данных в компании «Криптонит». В этой статье хочу поделиться с вами результатами небольшого исследования, в ходе которого мы протестировали ускоритель запросов Apache DataFusion Comet и пришли к довольно впечатляющим результатам. Забегая вперёд, отмечу, что в отдельных тестах ускорение было более чем десятикратным!

https://habr.com/ru/companies/kryptonite/articles/902872/

#spark #apache #comet #DataFusion #большие_данные #анализ_данных #data_engineering #data_scientist #big_data #оптимизация

I wrote a blog post on how to write user defined functions in Apache DataFusion. This includes how you can write Rust backed Python functions that operate at full native speed with zero copy operations of the data structures. Switching from pure python functions to these types of UDFs can lead to 10x speed improvements.

https://datafusion.apache.org/blog/2024/11/19/datafusion-python-udf-comparisons/

#ASF #DataFusion #DataScience

I wrote a blog post on how to write user defined functions in Apache DataFusion. This includes how you can write Rust backed Python functions that operate at full native speed with zero copy operations of the data structures. Switching from pure python functions to these types of UDFs can lead to 10x speed improvements.

https://datafusion.apache.org/blog/2024/11/19/datafusion-python-udf-comparisons/

#ASF #DataFusion #DataScience

The Apache Arrow ecosystem is an amazing enabler for a whole new range of database systems. Tonbo is a K/V store that uses Arrow+Parquet on storage systems ranging from OPFS (browser filesystem) to S3. And of course you get SQL for free with Apache DataFusion! #ApacheArrow #DataFusion #RustLang
https://tonbo.io/blog/introducing-tonbo

The Apache Arrow ecosystem is an amazing enabler for a whole new range of database systems. Tonbo is a K/V store that uses Arrow+Parquet on storage systems ranging from OPFS (browser filesystem) to S3. And of course you get SQL for free with Apache DataFusion! #ApacheArrow #DataFusion #RustLang
https://tonbo.io/blog/introducing-tonbo

Awesome reading list about Apache DataFusion. I started diving into it lately, hacking an extension to have Elasticsearch as a data source (aka a TableProvider). It's a wonderful piece of software and an impressive ecosystem. #DataFusion #RustLang https://datafusion.apache.org/user-guide/concepts-readings-events.html

Awesome reading list about Apache DataFusion. I started diving into it lately, hacking an extension to have Elasticsearch as a data source (aka a TableProvider). It's a wonderful piece of software and an impressive ecosystem. #DataFusion #RustLang https://datafusion.apache.org/user-guide/concepts-readings-events.html

Very nice introduction to Apache DataFusion and its internals. I started to dive into it lately, and it's really a wonderful piece of software. #datafusion #rustlang https://www.youtube.com/watch?v=iJhRbDFJjbg

Very nice introduction to Apache DataFusion and its internals. I started to dive into it lately, and it's really a wonderful piece of software. #datafusion #rustlang https://www.youtube.com/watch?v=iJhRbDFJjbg

And the code is up! Still an early prototype, but we can write SQL queries with DataFusion that join Elasticsearch indices with other sources like SQL databases, CVS files, Parquet data on S3, etc 🤯 DataFusion is powerful and fun to use! #elasticsearch #datafusion #rustlang https://github.com/swallez/elasticsearch-datafusion-tableprovider

And the code is up! Still an early prototype, but we can write SQL queries with DataFusion that join Elasticsearch indices with other sources like SQL databases, CVS files, Parquet data on S3, etc 🤯 DataFusion is powerful and fun to use! #elasticsearch #datafusion #rustlang https://github.com/swallez/elasticsearch-datafusion-tableprovider

Starting a "space time" week at Elastic, where we can work on whatever we want that is loosely related to our products. For me, this will be experimenting with Apache DataFusion to see how we can use/integrate it with Elasticsearch.
#elasticsearch #datafusion

Starting a "space time" week at Elastic, where we can work on whatever we want that is loosely related to our products. For me, this will be experimenting with Apache DataFusion to see how we can use/integrate it with Elasticsearch.
#elasticsearch #datafusion

I recently worked on an update to the Apache DataFusion project. The goal of this project is to provide a fast, modular approach to building large scale data processing.

The update adds in significant improvements to the python interface, to the point where I would now recommend giving it a try to people who haven't used it or had success.

I wrote up a blog post about the changes, hosted on the Apache DataFusion site: https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0/

#DataFusion #ASF #DataScience #DataEngineering

I recently worked on an update to the Apache DataFusion project. The goal of this project is to provide a fast, modular approach to building large scale data processing.

The update adds in significant improvements to the python interface, to the point where I would now recommend giving it a try to people who haven't used it or had success.

I wrote up a blog post about the changes, hosted on the Apache DataFusion site: https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0/

#DataFusion #ASF #DataScience #DataEngineering

🔥⏲️ Fudge Sunday #newsletter "Events and Efficiency" A look at what’s next in events and efficiency in #runtimes #vectorized #analytics #velox #comet #datafusion #serverless #spark

https://fudge.org/archive/events-and-efficiency/

I put in an open source PR today!

I've been playing around with Apache #DataFusion and wanted to see how it performed against a real life non-trivial problem I have. I ran into a blocker in that one of the basic functions that is exposed in the rust code underneath isn't exposed in python, so I just wrote it myself and put up a PR. I've never really been much of an open source contributor, so looking forward to seeing how this goes.

🌐 Five of our prototypes will be using advanced #DataFusion methods to bring EO products to the #nextlevel . Our scientists are at the forefront of developing a multi-modal architecture, leveraging Single Image Super Resolution #SIRS techniques fuelled by deep Convolutional Neural Networks #CNN and Generative Adversarial Networks #GAN 🖥

Explore further 👉 https://www.evo-land.eu/method/data-fusion/

#CLMS #CopernicusLand #RemoteSensing 🛰️🔍

@EU_HaDEA
@euenvironment
@inglada
@DLR
@cnes

#AI #datafusion could boost #sustainability

https://www.inderscience.com/info/inarticle.php?artid=134562
https://techxplore.com/news/2023-10-ai-fusion-boost-sustainability.html