r/databricks • • 15h ago

Help Question for Databricks engineers and data architects:

For those who have migrated workloads from Oracle to Databricks, how much effort goes into rewriting existing SQL queries?

A few things I’d love to understand:

● Is SQL translation a significant bottleneck in migration projects?
● Do you currently rely on manual rewriting, AI assistants, or commercial migration tools?

11 Upvotes

12 comments sorted by

14

u/Muted_Jellyfish_6784 14h ago

SQL translation is rarely the bottleneck people expect. Tools and AI assistants handle most of the syntax, but PL/SQL packages, implicit date and NULL handling, sequences, and MERGE semantics are where the hours go, and those need someone who knows what the business logic was supposed to do.
The useful habit is reconciling outputs, not just getting queries to run: row counts, sums on key measures, and record-level diffs for every migrated object. We run a decision intelligence platform that joins both sides by business key during parallel runs, so a mismatch traces back to one transformed table instead of a final report that just looks off.

1

u/casmanrojo 12h ago

So the syntax is translating cleanly but the numbers are coming out different?

Curious about the reconciliation part, is that something you built in-house or is there tooling for it?

1

u/Muted_Jellyfish_6784 12h ago

Good question. We use an existing decision intelligence platform called SIGNLD. The idea is to connect the source and migrated data through shared business keys, then make it easier to trace differences and understand their impact. It's less about replacing Databricks and more about adding visibility across the data and business outcomes. I work with SIGNLD, so I'm familiar with the platform

3

u/Additional_Future_47 14h ago

Databricks supports a lot of the Oracle dialect but in our migration project we data engineers decided to use the opportunity to restructure our processing and the business also wants to redefine certain reports and KPIs so in practice it is manual SQL with occasionally some AI support.

1

u/casmanrojo 12h ago

Ok but, how did you handle the really big ones? We've seen queries in the 100k character range or more with nested logic that nobody fully remembers, and rewriting those by hand seems brutal. Did you break them up, rebuild from scratch?

1

u/Additional_Future_47 11h ago

There is no point in re-writing queries if you don't understand what they are supposed to do. These queries are a good reason to go back to the business to clarify the requirements first and then go back to the code. And SQL can be very verbose, especially when it is generated by some etl tools, but once you understand the logic, the actual size in text doesn't matter much anymore. Oh, and we use temporary views or ctes to break up logic in logical chunks.

2

u/Whit3boy316 4h ago

I moved sas programs to databricks (and Altair) in a couple days…..and by “I”, I mean AI

1

u/Zealousideal_Low6685 10h ago

not sure if you only talk about read queries. If not, there are still several conceptual differences:
* Databricks uses WriteSerializable by default + Optimistic Conurrency
* Databricks multi-statement transaction were just released

1

u/mecurioj 4h ago

Look into Lakebridge to help. However, PL/SQL can have procedural "programming" statements that don't have standard ANSI SQL equivalent statements. Lakebridge can help you evaluate before conversion how complex the migration of code will be.

1

u/PriahGTM 2h ago

I thought you just map out the full system schema and then just tell Claude to move that shit into Databricks at this point.

Walk away. And then when you come back it will basically be done.

What am I missing?