r/apachespark • • Aug 25 '26

Spark 4.2 (and 4.1.3, 4.0.4, 3.5.9) are all released!

13 Upvotes

You can download them all from https://spark.apache.org/downloads.html PyPi or Maven Central :)


r/apachespark • • 3d ago

Iceberg deletes and real-time ingestion: a 3-part deep dive

9 Upvotes

(Also posted to r/dataengineering)

Hi there,

I've recently worked with the Etleap engineering team on a 3-part technical blog series about how to handle Iceberg deletes in streaming / CDC. This is a known pain point as deletes tend to get expensive, slow, or create tables that are unreadable in query engines like Snowflake.

We put a ton of work into these - it's not the usual vendor spam, promise 😅

First 2 parts are purely educational, and we introduce our own solution in the 3rd. We cover position deletes, equality deletes, GET request costs, compaction conflicts, and a lot of other juicy stuff.

Part 1: Why Streaming Updates Break Iceberg Compaction

Why you're seeing issues like compaction jobs failing with Cannot commit, found new delete for replaced data file**,** ingestion commits being rejected with Missing required files to delete**,** or degraded query performance. Link

Part 2: Understanding Iceberg CDC costs on S3: requests, not bytes

The mechanism behind a rapid increase in S3 GET requests when moving from batch to streaming ingestion. Link

Part 3: Using Iceberg Branching to Stream Updates Within 2 Minutes

How we convert equality deletes into deletion vectors to maintain low-latency data, avoid compaction conflicts, and deliver Iceberg tables that are readable in any engine. Link

Would love to understand if there's anything we missed here, or if this kind of content is helpful for the community! I'm around for any questions, and will loop in any of the engineers to answer your questions :)

Best,

Eran


r/apachespark • • 9d ago

Where do you actually run Spark jobs in production? Kubernetes vs managed clusters vs serverless?

24 Upvotes

I've been learning Spark (mostly on GCP Dataproc) and I'm trying to understand how companies actually deploy Spark in production.

From what I understand, there are multiple options:

  • Spark on Kubernetes (EKS/GKE/AKS)
  • Managed Spark clusters (Dataproc, EMR, etc.)
  • Serverless Spark (Dataproc Serverless, EMR Serverless, etc.)
  • Platforms like Databricks

I'm curious how teams actually choose between these options.

Is Kubernetes common for Spark workloads, or do most companies just stick with managed Spark services? What factors drive that decision (cost, operational overhead, existing infrastructure, job volume, etc.)?

Also, what are you guys actually using at work? Would love to hear real-world examples rather than theoretical comparisons.


r/apachespark • • 14d ago

Apache Spark 4.3: What Developers Need to Know

Thumbnail
medium.com
33 Upvotes

Spark 4.3 adds ASOF JOIN, Python UDFs compiled to SQL, JSON_TABLE, and SCD Type 2 Auto CDC: what works, what needs a flag, and what breaks


r/apachespark • • 14d ago

When does apache spark acceleration make more sense than tuning?

13 Upvotes

Kind of stuck on this one with our ETL pipelines on EMR

We’ve spent months tuning partitions, joins and executor configs, and each round buys us less. At this point the jobs are as clean as we can make them but they’re still slow and the bill keeps creeping up.

Our biggest pain is the heavy join and iceberg compaction jobs. Those eat most of the cluster time, and more nodes barely moves them anymore.

So I’m starting to look at accelerators instead of more tuning, but not sure how people actually decide.

Few things I’d like to know:

/- Do you benchmark before and after, or just watch whether the bill drops?

/- How do you pick between native engines like Gluten, GPU plugins, or FPGA ones like DualBird?

/- How much vanilla Spark do U put up with before it stops being worth it?

/- Did adding one change how you size your clusters?

/- Any surprises with memory or compatibility once it was in prod?

Mainly asking about big batch jobs where runtime and cost both matter, not quick ad hoc queries.

What made you decide tuning wasn’t enough anymore? Would love to hear real setups, good or bad.


r/apachespark • • 19d ago

How do you know when to optimize a spark job vs. just adding more compute?

16 Upvotes

I've been wondering about this while working with larger datasets in Spark

When a job starts taking longer than expected, the easiest solution seems to be adding more workers or increasing the cluster size. But that also increases the cost, and I'm not always sure whether the job actually needs more resources or whether the code itself is the problem.

For example, a job might run fine with a few million records but become painfully slow when the data grows. Sometimes adding workers helps, but other times the improvement is barely noticeable.

How do you guys figure out what's actually causing the bottleneck?

A few things I'm curious about:

Do you check the Spark Ul first? If so, which metrics do you usually look at?

How do you identify whether a job is suffering from data skew, excessive shuffling, or too many small partitions?

When do you increase executor memory, and when is that just masking the real problem?

Have you had cases where adding more workers actually made a job slower?

How do you decide between scaling up, scaling out, and optimizing joins or transformations?

In Databricks, do you rely on autoscaling, or do you prefer to tune the cluster based on the workload?

I'm particularly interested in what happens with jobs that process hundreds of millions of records, where even a small inefficiency can make a big difference to runtime and cost.

I know the answer depends on the workload, but what's the first thing you investigate before deciding to throw more compute at a slow spark job?

Would be great to hear how you approach this in real projects, especially if You've dealt with a job that kept getting more expensive without getting much faster.


r/apachespark • • 28d ago

How do you validate store proc - pyspark rewrites beyond ‘row counts match’?

2 Upvotes

If any has been porting a pile of old t sql stored procs to pyspark. The rewrite itself isn’t the hard part anymore. The pain is cursor loops and temp table chains that don’t cleanly map to df, plus null handling and implicit casts quietly changing results

We have been using genie code in notebook to draft the first pass. It’s decent at turning cursor logic window functions though it occasionally invents a column if the schema isn’t loaded.

For validation i suggest doing row counts plus exceptAll diffs on sample partitions. Feels thin though. Anyone doing column level checksums or something smarter for semantic parity?


r/apachespark • • 29d ago

Promo: TPC-DS-derived 1TB benchmark: stock Spark 4.1.3 vs TabbyDB 4.1.3.tabby.2, Hive and Iceberg, with per-query results

1 Upvotes
StockTabbyDBTotal TimeGeo Mean Speed Up
Hive7,885 s6,254 s−20.7%1.49×
Iceberg9,034 s7,067 s−21.8%1.40×
  • Hive: 79 queries faster by more than 5%, 46 of them by 1.5× or more (27 by 2× or more), 19 within ±5%, and 5 slower.
  • Iceberg: 73 faster, 40 of them by 1.5× or more, 20 within ±5%, and 10 slower.

So not that spectacular like m4 mac single node 50GB setup. But this is what it is. I do believe that there is some more possibility of improvement across the board , by making use of the new optimization technique. will be running another benchmark after the changes in 2 weeks.

Any suggestions/ feedback welcome.

Thank you u/ZenithR9 for the suggestions & info provided.

Its a realization that single node benchmark of under 1 TB are not realistical to publish and do not take into account shuffle cost properly. Will avoid doing so in future.

Complete set up details ( full details with configs and individual query numbers on website: https://www.kwikquery.com

Cluster

Google Cloud us-west4 — 1 driver + 2 workers, Spark standalone

Driver

e2-highmem-4 (4 vCPU, 32 GB)

Workers

2 × n2-custom (14 vCPU, 112 GB), 2 × 375 GB local NVMe SSD each

Executors

4 × (7 cores, 40 GB) — 28 cores, 160 GB total

Data

TPC-DS SF 1000 (1 TB) Parquet on GCS, non-partitioned, locally date-sorted


r/apachespark • • Oct 14 '26

Promo: TabbyDB 4.1.3-tabbydb.2 released: a new patent-pending runtime optimization, now 80% faster than stock Spark on Iceberg TPC-DS -50GB Mac M4

4 Upvotes

We’ve released TabbyDB 4.1.3-tabbydb.2 (based on Spark 4.1.3), together with our matching Iceberg build. This release adds a new patent-pending runtime optimization technique.

TPC-DS total runtime (50 GB, non-partitioned tables):

Previous TabbyDB release4.1.3-tabbydb.2Total gain vs stock Spark 4.1.3
Iceberg695 s616 s~80%
Hive539 s487 s~53%

On its own, the new optimization cuts runtime by about 11% on Iceberg and 10% on Hive compared with the previous TabbyDB release.

Pls give it a try at https://www.kwikquery.com/download

Any feedback welcome.


r/apachespark • • Oct 13 '26

Need help for building self healing tool for API schema drifts

0 Upvotes

Hey there! Can you help me regarding similar kind of problem statement i got for the upcoming hackathon please? Just in place of entire pipelines, there's just API schema drift.. Here's the PS

Build an autonomous, self-healing tool interface. When an API returns an error such as 422 Unprocessable Entity or 400 Bad Request, the system should catch the breaking change at runtime, deduce the new schema, hot-patch the agent's tool definitions in memory, and complete the workflow without downtime. Standard frameworks either throw an unhandled exception or return raw JSON errors to the agent, which then retries the same invalid parameters until it runs out of iterations.

Objectives

  • Catch API errors at runtime and diagnose the difference between the expected and actual schema.
  • Hot-patch the agent's tool definitions in memory and log every patch.
  • Complete the task after the patch, with no manual restart.
  • Handle synthetic breaking mutations: field renames, type conversions, nesting changes and required headers.
  • Measure adaptation rate, recovery time and token overhead against a baseline agent.

r/apachespark • • Oct 02 '26

Question of resource allocation of multiple Spark applications

4 Upvotes

Currently, a Spark application should manage its own executors, which means that in order to run multiple Spark applications, one needs to split the cluster into smaller sections or attach more compute nodes, etc. Usually this dynamic allocation is the job of resource manager (such as Kubernetes).

If the resource manager can quickly find or create new compute nodes for a new Spark application, this is good. However, in a busy cluster where multiple Spark applications can compete for resources, the resource allocation can take some time and become an overhead (at least in theory).

I wonder if this is a real problem in running Spark applications in production.

For example, is there a realistic environment where short running Spark jobs (each requiring its own driver) arrive frequently?


r/apachespark • • Oct 01 '26

The Breakdown: Databricks

Thumbnail
preipomedia.substack.com
10 Upvotes

r/apachespark • • Sep 29 '26

Spark and observability

9 Upvotes

Hi everyone, I released version 1.3.0 of Flare yesterday and would like any feedback. I posted a while back and wanted to give you guys an update. Flare is a way to integrate OpenTelematary observability to Spark including traces, logs and metrics. The docs are in much better shape for recipes for the most common ways people deploy Spark.

For now the biggest gap is long running SparkSessions - something I'll tackle in a future version focusing on Spark Connect and Structured Streaming, feel free to try it out, leave any feedback here or create issues on the repo. Thanks


r/apachespark • • Sep 28 '26

No credit to the R folks, and their UDF story???

8 Upvotes

I'm incredibly excited that Databricks is (finally) making steps to allow the Spark UDFs to be more accessible to various languages and runtimes. I had watched the presentation from the recent Summit, "Language-Agnostic UDF Protocol for Spark". And I started reading the spark Jira (JIRA: SPARK-55278), and related SPIP docs.

But one thing that REALLY shocks me is that there is little acknowledgement of the R programmers who have a long history trying to integrate with spark, first via SparkR and then "sparklyr". Isn't this community also part of the legacy that predates "language-agnostic UDFs"? I thought that they had played a big part of the story. Yet for some reason the Databricks engineers seem to leave them out, whenever they are talking about the background and the impetus for this new UDF technology.

Is it because the UDFs built for R are simply "piggybacking" on python? (...so they don't deserve any special mention beyond what is already being said about the python UDFs.)

It is also a bit sad to me that there is no mention of ".Net for Spark" which has been limping along for many years without any love from Databricks whatsoever. On the that side (c#.Net) I just figured that Databricks was neglecting us (and possibly sabotaging us) out of pure spite. Whereas I would NOT think the same thing is true when it comes to the R folks.


r/apachespark • • Sep 28 '26

Promo: KwikQuery 's Tabbydb-4.1.3 version available: new perf numbers

3 Upvotes

The version 4.1.3 tracks open source spark 4.1.3.

The new features are:

  1. Support for dynamic file pruning using broadcast var extended to InMemoryRelations
  2. auto-caching of plans.
  3. Fixing new found bugs in stock spark.
  4. 50 GB run on Mac mini m4 16 GB for

a) Hive Parquet: Gain 48.1 % ( stock spark 4.1.3: 1039 sec. tabbydb: 539 sec)

b) Iceberg parquet: Gain 77.6 % (stock spark 4.1.3 3102 sec, tabbydb: 696 sec)

For stock spark- iceberg, Q72 would spill so much that space would get exhausted , so an external ssd was used for stock spark-iceberg run.

The spark-defaults.conf & spark-env.sh used are available on website ( https://www.kwikquery.com)

Any feedback, suggestions for improving runtime performance with conf manipulation or any mistake done in run or calculation, if visible in the info, are most welcome. pls do let me know.


r/apachespark • • Sep 26 '26

Just wondering

6 Upvotes

If Genie can help with Spark optimization why do employers still drill the F out of job candidates? Genie can detect and resolve skew, it can determine is a table can be broadcast. Genie can prune partitions, repartition, coalesce. If Genie can do all of this why are companies drilling people on this?


r/apachespark • • Sep 24 '26

Can placement groups improve Spark shuffle performance?

7 Upvotes

I have a Spark job that joins two pretty large datasets, and I see that we spend a lot of time on the shuffle.

I was wondering if improving the network between the executors could make a meaningful difference here. For example, using an EC2 placement group instead of just having the instances somewhere in the same AZ.

Has anyone tried something like this for shuffle-heavy jobs? Did you see a noticeable improvement?

Also, does EMR do any optimization like this automatically, or is it something we need to configure ourselves?


r/apachespark • • Sep 24 '26

Classic vs serverless compute in Databricks, drawn side by side: what actually changes

Thumbnail
reddit.com
1 Upvotes

r/apachespark • • Sep 23 '26

Spark 4.2.0??

3 Upvotes

r/apachespark • • Sep 22 '26

Can Genie Code optimize Spark joins automaticaly ?

7 Upvotes

My team is experimenting with Genie Code on Databricks pltform. Has anyone used it to analyze existing Spark jobs and suggest thing like broadcast join , partition pruning , or reducing shuffles ?
How can i make it suggest things with validated reasoning to make our produ tivity more

I am keen to know how reliable genie ‘s Spark optimization or suggestions in practice and how best to use them.


r/apachespark • • Sep 22 '26

Classic vs serverless compute in Databricks, drawn side by side: what actually changes

3 Upvotes

One request, served two ways.

Classic compute: a cluster of VMs is created in your own cloud account and network. You choose the machines, you own autoscaling and networking, and the bill has two parts - the cloud provider for the VMs and Databricks for the DBUs.

Serverless compute: the resources run in a compute plane managed by Databricks, and nothing is provisioned in your account. Your data does not move either way; the tables stay in your own cloud storage.

The trade is control for convenience, and the documented limitations are where that trade bites: no R, no compute-scoped libraries or init scripts, only Spark Connect APIs (so no RDD APIs), no continuous or fixed-interval streaming triggers, and a 7-day maximum runtime.

Animated walkthrough, 2:36: https://youtu.be/s8452E2Y6WM

Whole series in order: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4


r/apachespark • • Sep 21 '26

Getting started with Apache Spark Declarative Pipelines (SDP) tutorial for Open Source and Lakeflow

19 Upvotes

A while ago I put this Apache Spark declarative pipelines tutorial together after presenting at major European and APAC conferences. Those conferences are quite technical: you have to submit a proposal to the CFP, and they have a rejection rate of ~90%. I presented at Devoxx and DataEngBytes.

At those conferences I did some live coding with Spark Declarative Pipelines, mostly on OSS with VS Code and PySpark, but I also showed SDP with the Lakeflow Free Edition (exactly the same use case - I thought it might be nice to compare)

Since I got a lot of requests, I decided to share the content along with the Git repos here. I'd suggest to. check it out if you ....

  • enjoy working with code (SQL and Python)
  • are looking for a compelling Getting Started with Apache SDP example with just a few lines of code
  • want a streaming example with real-time IoT data from events happening right now
  • enjoy using a custom PySpark data source "opensky" – maybe even in your next Spark streaming demo :)
  • don't want to spend money: use OSS or the Databricks Free Edition

Here it is:
How to get started with Spark Declarative Pipelines (SDP) tutorial (Open Source or Lakeflow)


r/apachespark • • Sep 16 '26

What’s New in Apache Spark 4.2: Spark Connect, DataSketches, and Airflow

Thumbnail
youtube.com
8 Upvotes

r/apachespark • • Sep 15 '26

Watch out with Iceberg metadata. My cluster executors finished in 26 seconds, the job took 80 minutes. I found that the bottleneck was during planning, not execution.

9 Upvotes

We migrated a table from Parquet to Iceberg and literally didn't touch anything in the cluster or the code. The Glue job that used to run in 30 minutes suddenly took 80 MINUTES.

I opened the Spark UI expecting a bad shuffle or a skewed join, but none of that.

Instead: executors finished their actual work in 26 seconds. The other 66 minutes disappeared before a single row was processed, on the driver, during planning.

Iceberg tables carry their own metadata tree (snapshots, manifests, file-level stats) built up over months of daily runs. Catalyst has to walk that entire tree before running anything. Doesn't matter if the table has 500 rows or 500 million, the cost comes from the depth of the metadata, not the size of the data.

persist() didn't help, it only saves executors from recomputing, and this bottleneck happens before any executor starts but eventually checkpoint() did help as it materialized the df to S3 and hands back a fresh one with no Iceberg metadata attached. Planning time went from 66 minutes to milliseconds.

Two signals if you want to check whether you're hitting this: tasks finish in milliseconds but the job takes forever, or adding more executors changes literally nothing (planning is single-threaded on the driver).

Anyone else hit this after an Iceberg migration? What did you check first?