#apachespark — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #apachespark, aggregated by home.social.
-
Treating SparkContext as a control tower shifts how you think about Spark: not just as an API, but as the coordinator for your entire distributed engine.
Read More: https://zalt.me/blog/2026/05/sparkcontext-control-tower
-
AQE (Adaptive Query Execution) : adapte le plan d'exécution en temps réel
DPP (Dynamic Partition Pruning) : ne lit que les partitions utiles pendant une jointure
SPJ (Storage Partition Join) : évite le shuffle en utilisant le partitionnement existant
-
https://luminousmen.com/post/the-apache-spark-optimization-checklist/ (en)
Comment optimiser Apache Spark ?
1. Utiliser les API DataFrame / Dataset, pas RDD.
2. Filtrer tôt, filtrer fort.
3. Trouver le data skew.
4. Connaitre AQE, DPP, SPJ.
5. Regarder l'UI. -
Join me Tuesday for my next Python Data Science & AI Full Throttle! https://deitel.com/PYDSFT
O'Reilly Media Pearson #deitel #python #machinelearning #deeplearning #NLP #datamining #ApacheSpark #BigData #IoT #GenAI
-
Join me Tuesday for my next Python Data Science & AI Full Throttle! https://deitel.com/PYDSFT
O'Reilly Media Pearson #deitel #python #machinelearning #deeplearning #NLP #datamining #ApacheSpark #BigData #IoT #GenAI
-
96% fewer out-of-memory (OOM) failures!
#Pinterest shared how it improved the reliability of its #ApacheSpark workloads.
By focusing on:
✅ Enhanced observability
✅ Configuration tuning
✅ Automatic memory retriesThe changes addressed persistent job failures affecting recommendation systems and large-scale data processing.
Details here ⇨ https://bit.ly/4smqrQD
#SoftwareArchitecture #BigData #CostOptimization #Memory #DistributedSystems #Observability #InfoQ
-
96% fewer out-of-memory (OOM) failures!
#Pinterest shared how it improved the reliability of its #ApacheSpark workloads.
By focusing on:
✅ Enhanced observability
✅ Configuration tuning
✅ Automatic memory retriesThe changes addressed persistent job failures affecting recommendation systems and large-scale data processing.
Details here ⇨ https://bit.ly/4smqrQD
#SoftwareArchitecture #BigData #CostOptimization #Memory #DistributedSystems #Observability #InfoQ
-
The Data Lakehouse Explained: Why Apache Iceberg Is Quietly Running the Show
https://techlife.blog/posts/data-lakehouse-iceberg
#ApacheIceberg #DataLakehouse #DataWarehouse #DataLake #Snowflake #ApacheSpark #DataEngineering
-
The Data Lakehouse Explained: Why Apache Iceberg Is Quietly Running the Show
https://techlife.blog/posts/data-lakehouse-iceberg
#ApacheIceberg #DataLakehouse #DataWarehouse #DataLake #Snowflake #ApacheSpark #DataEngineering
-
Bellevue / Seattle area friends: I’m super stoked for next week’s Spark Community Spring (Friday Mar 13th: spooky 👻).
If you’ve ever wanted to contribute to Apache Spark, come hang out and get your first Spark PR started with Felix Cheung, Huaxin Gao, Devin Petersohn, and myself :)
We’ll help folks find starter issues, get their dev environments set up, and walk through the contribution process.
There will be free lunch, and if enough people show up… maybe even Taco Bell for an afternoon snack*.
#ApacheSpark #OSS #hackathon #freelunch #tacofridaymaaaaybe
(* Depends on attendance)
-
Bellevue / Seattle area friends: I’m super stoked for next week’s Spark Community Spring (Friday Mar 13th: spooky 👻).
If you’ve ever wanted to contribute to Apache Spark, come hang out and get your first Spark PR started with Felix Cheung, Huaxin Gao, Devin Petersohn, and myself :)
We’ll help folks find starter issues, get their dev environments set up, and walk through the contribution process.
There will be free lunch, and if enough people show up… maybe even Taco Bell for an afternoon snack*.
#ApacheSpark #OSS #hackathon #freelunch #tacofridaymaaaaybe
(* Depends on attendance)
-
#Pinterest launched a next-gen CDC-based ingestion framework.
Using #ApacheKafka, #ApacheFlink, #ApacheSpark & #ApacheIceberg, they achieved:
• Latency cut from 24+ hours to 15 minutes
• Processing of only changed records
• Support for incremental updates & deletions
• Petabyte-scale data across 1,000+ pipelinesWin: optimized cost & efficiency!
Read the architectural deep dive on InfoQ 👉 https://bit.ly/4rMJB2H
-
#Pinterest launched a next-gen CDC-based ingestion framework.
Using #ApacheKafka, #ApacheFlink, #ApacheSpark & #ApacheIceberg, they achieved:
• Latency cut from 24+ hours to 15 minutes
• Processing of only changed records
• Support for incremental updates & deletions
• Petabyte-scale data across 1,000+ pipelinesWin: optimized cost & efficiency!
Read the architectural deep dive on InfoQ 👉 https://bit.ly/4rMJB2H
-
#Pinterest launched a next-gen CDC-based ingestion framework.
Using #ApacheKafka, #ApacheFlink, #ApacheSpark & #ApacheIceberg, they achieved:
• Latency cut from 24+ hours to 15 minutes
• Processing of only changed records
• Support for incremental updates & deletions
• Petabyte-scale data across 1,000+ pipelinesWin: optimized cost & efficiency!
Read the architectural deep dive on InfoQ 👉 https://bit.ly/4rMJB2H
-
#Pinterest launched a next-gen CDC-based ingestion framework.
Using #ApacheKafka, #ApacheFlink, #ApacheSpark & #ApacheIceberg, they achieved:
• Latency cut from 24+ hours to 15 minutes
• Processing of only changed records
• Support for incremental updates & deletions
• Petabyte-scale data across 1,000+ pipelinesWin: optimized cost & efficiency!
Read the architectural deep dive on InfoQ 👉 https://bit.ly/4rMJB2H
-
#Pinterest launched a next-gen CDC-based ingestion framework.
Using #ApacheKafka, #ApacheFlink, #ApacheSpark & #ApacheIceberg, they achieved:
• Latency cut from 24+ hours to 15 minutes
• Processing of only changed records
• Support for incremental updates & deletions
• Petabyte-scale data across 1,000+ pipelinesWin: optimized cost & efficiency!
Read the architectural deep dive on InfoQ 👉 https://bit.ly/4rMJB2H
-
In this #InfoQ article, Hina Gandhi explores a #ReinforcementLearning (RL) approach built on #ApacheSpark, enabling distributed computing systems to autonomously learn optimal configurations.
📰 Read now: https://bit.ly/4r0VdyP
-
In this #InfoQ article, Hina Gandhi explores a #ReinforcementLearning (RL) approach built on #ApacheSpark, enabling distributed computing systems to autonomously learn optimal configurations.
📰 Read now: https://bit.ly/4r0VdyP
-
Pinterest just shared a deep dive into Moka - its new blueprint for the future of large-scale data processing.
The company is migrating core workloads from ageing Hadoop infrastructure to a Kubernetes-based platform on Amazon EKS, with Apache Spark as the primary engine - and support for additional frameworks coming soon.
Curious to learn more? Read on #InfoQ 👉 https://bit.ly/4qCs4JP
-
Pinterest just shared a deep dive into Moka - its new blueprint for the future of large-scale data processing.
The company is migrating core workloads from ageing Hadoop infrastructure to a Kubernetes-based platform on Amazon EKS, with Apache Spark as the primary engine - and support for additional frameworks coming soon.
Curious to learn more? Read on #InfoQ 👉 https://bit.ly/4qCs4JP
-
#CaseStudy - Agoda consolidated multiple independent data pipelines into a central #ApacheSpark platform, eliminating financial data inconsistencies.
A multi-layered quality framework - with automated checks, ML anomaly detection, and data contracts - ensures accurate financial metrics while handling millions of daily bookings.
Deep dive into the architecture here ⇨ https://bit.ly/4a109NP
-
#CaseStudy - Agoda consolidated multiple independent data pipelines into a central #ApacheSpark platform, eliminating financial data inconsistencies.
A multi-layered quality framework - with automated checks, ML anomaly detection, and data contracts - ensures accurate financial metrics while handling millions of daily bookings.
Deep dive into the architecture here ⇨ https://bit.ly/4a109NP
-
【ハンズオン】OCI Data Flowで始めるApache Spark｜ETLからMLまで体験してみよう！
https://qiita.com/yushibats/items/559b65f72efeccf8865b?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items -
Apache Spark's new Declarative Pipelines framework simplifies ETL development by letting engineers focus on defining transformations while automating orchestration & error handling. This open-source solution handles batch/streaming workloads via Python/SQL interfaces, significantly reducing boilerplate code. Promising productivity gains for teams managing complex Spark pipelines. What impact might declarative approaches have on your workflow? #ApacheSpark #ETL #OpenSource
-
Discover how Decathlon, one of the world’s leading sports retailers, adopted the #opensource library #Polars to optimize its data workflows.
By migrating from Apache Spark to Polars for small input datasets, Decathlon achieved:
• Significant speed
• Meaningful cost savings👉 Learn more: https://bit.ly/4qmb2zc
-
Discover how Decathlon, one of the world’s leading sports retailers, adopted the #opensource library #Polars to optimize its data workflows.
By migrating from Apache Spark to Polars for small input datasets, Decathlon achieved:
• Significant speed
• Meaningful cost savings👉 Learn more: https://bit.ly/4qmb2zc
-
#CaseStudy - #Lyft rearchitected its ML platform, LyftLearn, into a hybrid system!
Offline workloads now run on AWS SageMaker, while Kubernetes continues to power online model serving.
The result❓ Read #InfoQ and find out 👉 https://bit.ly/3Y3hTBG
-
#CaseStudy - #Lyft rearchitected its ML platform, LyftLearn, into a hybrid system!
Offline workloads now run on AWS SageMaker, while Kubernetes continues to power online model serving.
The result❓ Read #InfoQ and find out 👉 https://bit.ly/3Y3hTBG
-
In this #InfoQ #podcast, Vivek Yadav shares his journey in building a testing system based on multi-year worth of data.
Discover why he chose #ApacheSpark and how it integrates with “traditional” engineering practices.
🎧Listen now: https://bit.ly/487zXQM
-
In this #InfoQ #podcast, Vivek Yadav shares his journey in building a testing system based on multi-year worth of data.
Discover why he chose #ApacheSpark and how it integrates with “traditional” engineering practices.
🎧Listen now: https://bit.ly/487zXQM
-
Dew Drop – September 23, 2025 (#4503)
https://fed.brid.gy/r/https://alvinashcraft.com/2025/09/23/dew-drop-september-23-2025-4503/
-
Apache Spark and its Python counterpart, PySpark, have emerged as groundbreaking solutions reshaping how data is processed, analyzed, and leveraged. https://hackernoon.com/spark-and-pyspark-redefining-distributed-data-processing #apachespark
-
Apache Spark and its Python counterpart, PySpark, have emerged as groundbreaking solutions reshaping how data is processed, analyzed, and leveraged. https://hackernoon.com/spark-and-pyspark-redefining-distributed-data-processing #apachespark
-
ICYMI: Abe Sharp looks at Volcano, a #CNCF project that optimizes high-performance workloads on Kubernetes to avoid deadlocks
https://www.admin-magazine.com/Archive/2025/86/Application-aware-batch-scheduler?utm_source=mam
#Kubernetes #scheduler #Volcano #Queue #PodGroup #ApacheSpark #PyTorch #MachineLearning -
ICYMI: Abe Sharp looks at Volcano, a #CNCF project that optimizes high-performance workloads on Kubernetes to avoid deadlocks
https://www.admin-magazine.com/Archive/2025/86/Application-aware-batch-scheduler?utm_source=mam
#Kubernetes #scheduler #Volcano #Queue #PodGroup #ApacheSpark #PyTorch #MachineLearning -
Databricks is contributing the tech behind Delta Live Tables (DLT) to the #ApacheSpark project!
It will now be known as Spark Declarative Pipelines, making it easier to develop & maintain streaming pipelines for all Spark users.
🔗 Learn more: https://bit.ly/3IkaM3a
-
Databricks is contributing the tech behind Delta Live Tables (DLT) to the #ApacheSpark project!
It will now be known as Spark Declarative Pipelines, making it easier to develop & maintain streaming pipelines for all Spark users.
🔗 Learn more: https://bit.ly/3IkaM3a
-
Today is the DBA Appreciation Day!
Bring your DBAs a cake and a coffee, please. And don't drop any tables in production, pretty please. It's weekend ...
#PostgreSQL #SQLServer #Oracle #DB2 #MySQL #MariaDB #Snowflake #SQLite #Neo4j #Teradata #SAPHana #Aerospike #ApacheSpark #Clickhouse #Informix #WarehousePG #Greenplum #Adabas
-
Today is the DBA Appreciation Day!
Bring your DBAs a cake and a coffee, please. And don't drop any tables in production, pretty please. It's weekend ...
#PostgreSQL #SQLServer #Oracle #DB2 #MySQL #MariaDB #Snowflake #SQLite #Neo4j #Teradata #SAPHana #Aerospike #ApacheSpark #Clickhouse #Informix #WarehousePG #Greenplum #Adabas
-
Today is the DBA Appreciation Day!
Bring your DBAs a cake and a coffee, please. And don't drop any tables in production, pretty please. It's weekend ...
#PostgreSQL #SQLServer #Oracle #DB2 #MySQL #MariaDB #Snowflake #SQLite #Neo4j #Teradata #SAPHana #Aerospike #ApacheSpark #Clickhouse #Informix #WarehousePG #Greenplum #Adabas
-
Today is the DBA Appreciation Day!
Bring your DBAs a cake and a coffee, please. And don't drop any tables in production, pretty please. It's weekend ...
#PostgreSQL #SQLServer #Oracle #DB2 #MySQL #MariaDB #Snowflake #SQLite #Neo4j #Teradata #SAPHana #Aerospike #ApacheSpark #Clickhouse #Informix #WarehousePG #Greenplum #Adabas
-
Today is the DBA Appreciation Day!
Bring your DBAs a cake and a coffee, please. And don't drop any tables in production, pretty please. It's weekend ...
#PostgreSQL #SQLServer #Oracle #DB2 #MySQL #MariaDB #Snowflake #SQLite #Neo4j #Teradata #SAPHana #Aerospike #ApacheSpark #Clickhouse #Informix #WarehousePG #Greenplum #Adabas
-
New post about how to write data from a Apache Spark DataFrame into a Elasticsearch/Opensearch database #datascience #databricks #elasticsearch #opensearch #bigdata #apachespark #spark #tech #programming #python:
https://pedro-faria.netlify.app/posts/2025/2025-03-16-spark-elasticsearch/en/
-
Easier to use: DuckDB gets local web user interface
As of version 1.2.1, the DuckDB in-process database can be conveniently operated via a local UI, which is installed as an extension, as an alternative to CLI.
-
Easier to use: DuckDB gets local web user interface
As of version 1.2.1, the DuckDB in-process database can be conveniently operated via a local UI, which is installed as an extension, as an alternative to CLI.
-
Einfacher bedienen: DuckDB erhält lokale Web-Benutzeroberfläche
Die In-Process-Datenbank DuckDB lässt sich ab Version 1.2.1 alternativ zur CLI komfortabel über ein lokales UI bedienen, das als Extension installiert wird.
-
Einfacher bedienen: DuckDB erhält lokale Web-Benutzeroberfläche
Die In-Process-Datenbank DuckDB lässt sich ab Version 1.2.1 alternativ zur CLI komfortabel über ein lokales UI bedienen, das als Extension installiert wird.
-
TIL: You can get a list of Spark-enabled GATK tools with the command
gatk --list | grep Spark
(The website doesn't seem to have a list anywhere)
-
Spark Connect is revolutionizing the way we run Spark applications. With version 3.4 and beyond, remote client applications written in Scala or Python can now run on a Spark cluster, offering more flexibility than ever before. Read Sergey Kotlov's latest article now.
https://towardsdatascience.com/adopting-spark-connect-cdd6de69fa98
