#apachespark — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #apachespark, aggregated by home.social.
-
Stackable Data Platform 26.7: Security, SBOMs, and Flexible Registries
Stackable SDP 26.7 focuses on quality and supply chain security, bringing SLSA provenance and breaking changes to registry logic.
#ApacheSpark #Containerisierung #IT #Kubernetes #OpenSource #Updates #news
-
Stackable Data Platform 26.7: Sicherheit, SBOMs und flexible Registries
Stackable stellt die Data Platform 26.7 auf Qualität und Supply-Chain-Sicherheit um. Neben SLSA-Provenance drohen bei der Registry-Logik Breaking Changes.
#ApacheSpark #Containerisierung #IT #Kubernetes #OpenSource #Updates #news
-
AQE (Adaptive Query Execution) : adapte le plan d'exécution en temps réel
DPP (Dynamic Partition Pruning) : ne lit que les partitions utiles pendant une jointure
SPJ (Storage Partition Join) : évite le shuffle en utilisant le partitionnement existant
-
https://luminousmen.com/post/the-apache-spark-optimization-checklist/ (en)
Comment optimiser Apache Spark ?
1. Utiliser les API DataFrame / Dataset, pas RDD.
2. Filtrer tôt, filtrer fort.
3. Trouver le data skew.
4. Connaitre AQE, DPP, SPJ.
5. Regarder l'UI. -
Join me Tuesday for my next Python Data Science & AI Full Throttle! https://deitel.com/PYDSFT
O'Reilly Media Pearson #deitel #python #machinelearning #deeplearning #NLP #datamining #ApacheSpark #BigData #IoT #GenAI
-
96% fewer out-of-memory (OOM) failures!
#Pinterest shared how it improved the reliability of its #ApacheSpark workloads.
By focusing on:
✅ Enhanced observability
✅ Configuration tuning
✅ Automatic memory retriesThe changes addressed persistent job failures affecting recommendation systems and large-scale data processing.
Details here ⇨ https://bit.ly/4smqrQD
#SoftwareArchitecture #BigData #CostOptimization #Memory #DistributedSystems #Observability #InfoQ
-
Bellevue / Seattle area friends: I’m super stoked for next week’s Spark Community Spring (Friday Mar 13th: spooky 👻).
If you’ve ever wanted to contribute to Apache Spark, come hang out and get your first Spark PR started with Felix Cheung, Huaxin Gao, Devin Petersohn, and myself :)
We’ll help folks find starter issues, get their dev environments set up, and walk through the contribution process.
There will be free lunch, and if enough people show up… maybe even Taco Bell for an afternoon snack*.
#ApacheSpark #OSS #hackathon #freelunch #tacofridaymaaaaybe
(* Depends on attendance)
-
#Pinterest launched a next-gen CDC-based ingestion framework.
Using #ApacheKafka, #ApacheFlink, #ApacheSpark & #ApacheIceberg, they achieved:
• Latency cut from 24+ hours to 15 minutes
• Processing of only changed records
• Support for incremental updates & deletions
• Petabyte-scale data across 1,000+ pipelinesWin: optimized cost & efficiency!
Read the architectural deep dive on InfoQ 👉 https://bit.ly/4rMJB2H
-
In this #InfoQ article, Hina Gandhi explores a #ReinforcementLearning (RL) approach built on #ApacheSpark, enabling distributed computing systems to autonomously learn optimal configurations.
📰 Read now: https://bit.ly/4r0VdyP
-
Pinterest just shared a deep dive into Moka - its new blueprint for the future of large-scale data processing.
The company is migrating core workloads from ageing Hadoop infrastructure to a Kubernetes-based platform on Amazon EKS, with Apache Spark as the primary engine - and support for additional frameworks coming soon.
Curious to learn more? Read on #InfoQ 👉 https://bit.ly/4qCs4JP
-
#CaseStudy - Agoda consolidated multiple independent data pipelines into a central #ApacheSpark platform, eliminating financial data inconsistencies.
A multi-layered quality framework - with automated checks, ML anomaly detection, and data contracts - ensures accurate financial metrics while handling millions of daily bookings.
Deep dive into the architecture here ⇨ https://bit.ly/4a109NP
-
Discover how Decathlon, one of the world’s leading sports retailers, adopted the #opensource library #Polars to optimize its data workflows.
By migrating from Apache Spark to Polars for small input datasets, Decathlon achieved:
• Significant speed
• Meaningful cost savings👉 Learn more: https://bit.ly/4qmb2zc
-
#CaseStudy - #Lyft rearchitected its ML platform, LyftLearn, into a hybrid system!
Offline workloads now run on AWS SageMaker, while Kubernetes continues to power online model serving.
The result❓ Read #InfoQ and find out 👉 https://bit.ly/3Y3hTBG
-
In this #InfoQ #podcast, Vivek Yadav shares his journey in building a testing system based on multi-year worth of data.
Discover why he chose #ApacheSpark and how it integrates with “traditional” engineering practices.
🎧Listen now: https://bit.ly/487zXQM
-
Apache Spark and its Python counterpart, PySpark, have emerged as groundbreaking solutions reshaping how data is processed, analyzed, and leveraged. https://hackernoon.com/spark-and-pyspark-redefining-distributed-data-processing #apachespark
-
ICYMI: Abe Sharp looks at Volcano, a #CNCF project that optimizes high-performance workloads on Kubernetes to avoid deadlocks
https://www.admin-magazine.com/Archive/2025/86/Application-aware-batch-scheduler?utm_source=mam
#Kubernetes #scheduler #Volcano #Queue #PodGroup #ApacheSpark #PyTorch #MachineLearning -
Databricks is contributing the tech behind Delta Live Tables (DLT) to the #ApacheSpark project!
It will now be known as Spark Declarative Pipelines, making it easier to develop & maintain streaming pipelines for all Spark users.
🔗 Learn more: https://bit.ly/3IkaM3a
-
Today is the DBA Appreciation Day!
Bring your DBAs a cake and a coffee, please. And don't drop any tables in production, pretty please. It's weekend ...
#PostgreSQL #SQLServer #Oracle #DB2 #MySQL #MariaDB #Snowflake #SQLite #Neo4j #Teradata #SAPHana #Aerospike #ApacheSpark #Clickhouse #Informix #WarehousePG #Greenplum #Adabas
-
New post about how to write data from a Apache Spark DataFrame into a Elasticsearch/Opensearch database #datascience #databricks #elasticsearch #opensearch #bigdata #apachespark #spark #tech #programming #python:
https://pedro-faria.netlify.app/posts/2025/2025-03-16-spark-elasticsearch/en/
-
Easier to use: DuckDB gets local web user interface
As of version 1.2.1, the DuckDB in-process database can be conveniently operated via a local UI, which is installed as an extension, as an alternative to CLI.
-
Einfacher bedienen: DuckDB erhält lokale Web-Benutzeroberfläche
Die In-Process-Datenbank DuckDB lässt sich ab Version 1.2.1 alternativ zur CLI komfortabel über ein lokales UI bedienen, das als Extension installiert wird.
-
TIL: You can get a list of Spark-enabled GATK tools with the command
gatk --list | grep Spark
(The website doesn't seem to have a list anywhere)
-
Spark Connect is revolutionizing the way we run Spark applications. With version 3.4 and beyond, remote client applications written in Scala or Python can now run on a Spark cluster, offering more flexibility than ever before. Read Sergey Kotlov's latest article now.
https://towardsdatascience.com/adopting-spark-connect-cdd6de69fa98
-
🎃The October issue of #CheckpointChronicle is now out 🌟
It covers Ververica's Fluss, #ApacheFlink 2.0, Iggy.rs, Strimzi's support for #ApacheKafka 4.0, tons of OTF material from @vanlightly, Christian Hollinger's write up of ngrok's data platform, nice detail of how SmartNews use #ApacheIceberg with Flink and #ApacheSpark, a good writeup from Sudhendu Pandey on #ApachePolaris, notes from Kir Titievsky on Kafka's Avro serialisers, and much more!
