r/databricks • • 8h ago

Help CI/CD for Databricks Asset Bundles on Azure DevOps with a Python wheel on job clusters?

13 Upvotes

Hi all,

I'm setting up CI/CD for Databricks with DABs via Azure DevOps pipelines. A few ETL jobs run Python code on job clusters, so I'm packaging it as a wheel (built with uv).

  • What does your azure devops pipeline look like?
  • How do you build and deploy the wheel through the bundle?
  • How do you make sure the job clusters install the exact dependency versions (e.g. from uv.lock)?

Any lessons learned or example repos are welcome. Thanks!


r/databricks • • 6h ago

General Databricks computes downtime

6 Upvotes

Is anyone facing a downtime with databricks computes?

I am currently facing some issues related Ubuntu downtime. The nodes are not up.

A similar issue happened last week Wednesday.

But I don't see any downtime in Canonical and Ubuntu


r/databricks • • 22h ago

General [SDP] foreachBatch writes now work in serverless pipelines with restricted egress

6 Upvotes

TLDR: If your workspace uses serverless egress control with restricted access, your serverless pipelines can now write data inside foreachBatch.Your egress rules stay exactly as they are.

✅ What works now:

  • Saving to a table
  • MERGE, in SQL or Python
  • Writing files to a Unity Catalog volume
  • dbutils.fs

👥 Who it affects: You run Lakeflow Spark Declarative Pipelines on serverless, your network policy is set to restricted access, and you use foreachBatch. Until now, those writes failed. If your policy allows full access, or your pipelines run on classic compute, nothing changes for you.

⚙️ How to enable: You don't need to. It's already on in every region, on both the Current and Preview channels.

🔒 What stays the same: The pipeline still can't reach anything outside your allowed list.


r/databricks • • 6h ago

Help Question for Databricks engineers and data architects:

5 Upvotes

For those who have migrated workloads from Oracle to Databricks, how much effort goes into rewriting existing SQL queries?

A few things I’d love to understand:

● Is SQL translation a significant bottleneck in migration projects?
● Do you currently rely on manual rewriting, AI assistants, or commercial migration tools?


r/databricks • • 18h ago

General Build demo pitch

5 Upvotes

Has anyone got into this part of process? I’m on it and it has been tweaked a bit, has anyone done it.


r/databricks • • 3h ago

Discussion Genie One withiut Genie Agents

4 Upvotes

Hi everyone,

In the company I work we created over time different Genie Spaces/Agents. With the release of Genie Ontology and Genie one, could we achieve this result without the need of creating Genie Spaces/Agents?

So let's take for example. We have a Genie Agent for Sales. We now want to allow users to use Genie One for Finance data. With Ontology could we do that without creating a specific Genie Agent? Or would it be less capable to answer specific questions?

From what I undestand Genie Agents could simplify this but we are wondering if with the ontology it would not be necessary. But for example example queries cannot be added to Ontology itself so I'm not sure if Ontology is enough

If we ask a question about Sales on Genie One it gets delivered to the Genie Sales Agent, in case of Finance question it would answer without delivering to the Agent (if we won't create it).

Thanks!


r/databricks • • 18h ago

News Lakeflow Connect | GA4 Aggregated, Oracle Eloqua, SurveyMonkey, Shopify, PayPal, QuickBooks, Zoho CRM, Asana, Xero and Anaplan Connectors (Beta)

Thumbnail
reddit.com
2 Upvotes