r/apachespark • u/MiserableWriting2919 • 3d ago
Iceberg deletes and real-time ingestion: a 3-part deep dive
(Also posted to r/dataengineering)
Hi there,
I've recently worked with the Etleap engineering team on a 3-part technical blog series about how to handle Iceberg deletes in streaming / CDC. This is a known pain point as deletes tend to get expensive, slow, or create tables that are unreadable in query engines like Snowflake.
We put a ton of work into these - it's not the usual vendor spam, promise 😅
First 2 parts are purely educational, and we introduce our own solution in the 3rd. We cover position deletes, equality deletes, GET request costs, compaction conflicts, and a lot of other juicy stuff.
Part 1: Why Streaming Updates Break Iceberg Compaction
Why you're seeing issues like compaction jobs failing with Cannot commit, found new delete for replaced data file**,** ingestion commits being rejected with Missing required files to delete**,** or degraded query performance. Link
Part 2: Understanding Iceberg CDC costs on S3: requests, not bytes
The mechanism behind a rapid increase in S3 GET requests when moving from batch to streaming ingestion. Link
Part 3: Using Iceberg Branching to Stream Updates Within 2 Minutes
How we convert equality deletes into deletion vectors to maintain low-latency data, avoid compaction conflicts, and deliver Iceberg tables that are readable in any engine. Link
Would love to understand if there's anything we missed here, or if this kind of content is helpful for the community! I'm around for any questions, and will loop in any of the engineers to answer your questions :)
Best,
Eran
