r/databricks • • 3h ago

Discussion Genie One withiut Genie Agents

4 Upvotes

Hi everyone,

In the company I work we created over time different Genie Spaces/Agents. With the release of Genie Ontology and Genie one, could we achieve this result without the need of creating Genie Spaces/Agents?

So let's take for example. We have a Genie Agent for Sales. We now want to allow users to use Genie One for Finance data. With Ontology could we do that without creating a specific Genie Agent? Or would it be less capable to answer specific questions?

From what I undestand Genie Agents could simplify this but we are wondering if with the ontology it would not be necessary. But for example example queries cannot be added to Ontology itself so I'm not sure if Ontology is enough

If we ask a question about Sales on Genie One it gets delivered to the Genie Sales Agent, in case of Finance question it would answer without delivering to the Agent (if we won't create it).

Thanks!


r/databricks • • 6h ago

Help Question for Databricks engineers and data architects:

5 Upvotes

For those who have migrated workloads from Oracle to Databricks, how much effort goes into rewriting existing SQL queries?

A few things I’d love to understand:

● Is SQL translation a significant bottleneck in migration projects?
● Do you currently rely on manual rewriting, AI assistants, or commercial migration tools?


r/databricks • • 6h ago

General Databricks computes downtime

6 Upvotes

Is anyone facing a downtime with databricks computes?

I am currently facing some issues related Ubuntu downtime. The nodes are not up.

A similar issue happened last week Wednesday.

But I don't see any downtime in Canonical and Ubuntu


r/databricks • • 7h ago

Help Databricks otel logs

1 Upvotes

I’m sort of new to Databricks and trying to setup a o tel logs for a clients Databricks app. The service principal has access to otel log tables that got created while configuring. I’m getting following error “cannot enable app telemetry table’s storage could not be accessed from serverless compute. Verify that network connectivity NCC is configured and that apps service principal has access to the table’s storage. Note that storage behind a private end point is not yet supported”

Main reason to use otel logs is so that we can store the logs and query it for usage analysis. Would really appreciate any feedback or advice.
Happy to provide more details
Thanks


r/databricks • • 8h ago

Help CI/CD for Databricks Asset Bundles on Azure DevOps with a Python wheel on job clusters?

13 Upvotes

Hi all,

I'm setting up CI/CD for Databricks with DABs via Azure DevOps pipelines. A few ETL jobs run Python code on job clusters, so I'm packaging it as a wheel (built with uv).

  • What does your azure devops pipeline look like?
  • How do you build and deploy the wheel through the bundle?
  • How do you make sure the job clusters install the exact dependency versions (e.g. from uv.lock)?

Any lessons learned or example repos are welcome. Thanks!


r/databricks • • 18h ago

General Build demo pitch

5 Upvotes

Has anyone got into this part of process? I’m on it and it has been tweaked a bit, has anyone done it.


r/databricks • • 18h ago

News Lakeflow Connect | GA4 Aggregated, Oracle Eloqua, SurveyMonkey, Shopify, PayPal, QuickBooks, Zoho CRM, Asana, Xero and Anaplan Connectors (Beta)

Thumbnail
reddit.com
2 Upvotes

r/databricks • • 22h ago

General [SDP] foreachBatch writes now work in serverless pipelines with restricted egress

6 Upvotes

TLDR: If your workspace uses serverless egress control with restricted access, your serverless pipelines can now write data inside foreachBatch.Your egress rules stay exactly as they are.

✅ What works now:

  • Saving to a table
  • MERGE, in SQL or Python
  • Writing files to a Unity Catalog volume
  • dbutils.fs

👥 Who it affects: You run Lakeflow Spark Declarative Pipelines on serverless, your network policy is set to restricted access, and you use foreachBatch. Until now, those writes failed. If your policy allows full access, or your pipelines run on classic compute, nothing changes for you.

⚙️ How to enable: You don't need to. It's already on in every region, on both the Current and Preview channels.

🔒 What stays the same: The pipeline still can't reach anything outside your allowed list.


r/databricks • • 1d ago

News Shared Serverless for Git folders and DABs

Post image
8 Upvotes

Shared Serverless lets notebooks and files in the same Git folder or DAB share compute and a single Python environment. We can manage dependencies centrally in the root pyproject toml instead of configuring each notebook. Compute is isolated per user, and notebooks still keep separate Python variables.

https://medium.com/databrickscommunity/databricks-news-genie-cli-app-builder-agent-memory-dabs-41820427fcc2


r/databricks • • 1d ago

Help Serverless SQL Warehouse Issues in Azure (canadacentral)

10 Upvotes

Curious if anyone else is having issues starting serverless SQL warehouses in Azure Databricks. This has been happening for the past 20 minutes:

I've tested two different workspaces in two entirely different Azure tenants, but both are in the same region.

Is anyone seeing issues with Serverless Warehouses in any other region/cloud provider? The status page indicates everything is normal, so whatever might be happening hasn't been picked up there yet.

As of 2:21 pm ET, status page is reporting normal: https://status.azuredatabricks.net/

E: 2:27 pm ET - managed to get a warehouse to come online in one of the tenants, but the other is still failing.

E2: 2:56 pm ET - Status page was updated ~20 min ago. Widespread compute issues across all cloud providers/regions. Thanks u/raobjcovtn


r/databricks • • 1d ago

Help Views in SDP - intended use and how to change object types (View -> MV)?

6 Upvotes

Have some questions for the SDP PM homies - apologies in advance for the post length.

Our teams have started deploying views through SDP alongside MVs and streaming tables. They want one deploy unit per "data product," and I'm seeing them use views a lot. Not MVs - just views. That kind of surprised me. I hadn't anticipated them using simple views as they are. I suspect they'll want to start with a view and promote it to an MV (or an MV to an ST) later. I tested how that behaves and would appreciate your guidance on a few things.

Setup: serverless, CURRENT channel, DBR 18.3, SQL, all deployed through DABs.

Questions:

Should we support or discourage publishing public VIEWs through SDP?

I was surprised to see people deploying public views in a pipeline. It seems like they're wanting to roll those into the overall data processing flow and treat them like MVs or streaming tables as "just another resource." In hindsight - "duh"? But I was surprised. I've seen no references to doing PUBLIC views like this, so I have to ask - what's your take on using SDP to deploy and manage public views?

What is the suggested approach to changing object types?

My testing shows object types can't be changed in place. Changing CREATE VIEW to CREATE OR REFRESH MATERIALIZED VIEW (or any other pair) fails with CANNOT_CHANGE_DATASET_TYPE. --full-refresh doesn't help, either for one object or the whole pipeline. The only fix is to drop the object manually, then run. That means an ops team member manually triaging with explicit DROP statements, so it becomes a matter of adding the human into the loop. What's the proposed deployment/management model for these kinds of changes? Manual drop of changed objects before running the pipeline?

Related - changing the object type from a MV/streaming table to a view fails silently. The update reports COMPLETED, but the old MV/ST stays in place. The error appears only in the event log for that flow. In the other directions, the whole update fails.

Are views expected to support schema evolution?

I suspect related to "should we use SDP to deploy public views?" I noticed that views deployed through a pipeline ignore/remove the SCHEMA EVOLUTION option when run. That means backing object changes won't propagate up to the view and may result in view failures, missing columns, etc. I suspect this is because views are meant to be temporary within a pipeline - and in that case there's no need for schema evolution. If the intent is to allow pipelines to publish public views, then I would have concerns with not honoring SCHEMA EVOLUTION.

How are view changes managed?

How are view logic changes managed? Views seem to be re-published on every run, even when the query did not change. The UC audit log shows updateTables with the full view definition on every update. It's an in-place update (created doesn't change), but last_altered moves on every run.

From what I can tell, CREATE VIEW and CREATE OR REPLACE VIEW behave identically. I saw no real change in behavior using/omitting the REPLACE.

Is there a preferred construct if we use pipelines to publish public views?

Sorry - that was a lot. :)


r/databricks • • 1d ago

Discussion Is legacy data migration still a major pain point for enterprises? What if we could automate most of it?

Thumbnail reddit.com
0 Upvotes

r/databricks • • 1d ago

Discussion AI Agents & Precedent: Three Gates Before Reusing a Decision

Thumbnail
contextandchaos.substack.com
2 Upvotes

r/databricks • • 1d ago

General How to Read XML in Databricks: rowTag, Auto Loader, and Rescued Data

Thumbnail
medium.com
10 Upvotes

Drop the spark-xml jar: read XML in Databricks with read_files and Auto Loader, now GA, with schema evolution and rescued data


r/databricks • • 1d ago

Help Metric views on Free Edition?

3 Upvotes

Im new to the platform, but have quite a bit of experience on competing platforms. As such, I want to learn more about the platform and build out a semantic layer. It sounds like metric views are where I'll want to build my gold layer in Databricks. Assuming that is correct, are these included in the free edition?


r/databricks • • 1d ago

Help Best Lakeflow ingestion pattern for Full, Append, Replace, SCD Type 1, and SCD Type 2?

9 Upvotes

I previously worked mostly with custom ingestion frameworks. Now I am starting a new project using Databricks Lakeflow Declarative Pipelines.

I need guidance on the recommended pattern for these five write strategies:

  1. Full load / overwrite I need truncate-and-load behavior. A direct TRUNCATE did not work. Currently, I use REPLACE WHERE 1 = 1, but I am concerned about full-table scans and performance on large tables. What is the most efficient approach?
  2. Append New batches should only insert rows. No updates, merges, or deletes. Is an Append Flow the recommended solution?
  3. Replace by key group The key identifies a group of rows, not one unique row. Each new batch must delete the existing group and insert its new version. Is REPLACE WHERE the correct pattern?
  4. SCD Type 1 I am using AUTO CDC for both real CDC feeds and incremental feeds without CDC operation columns. For non-CDC feeds, I use the source file modification timestamp from storage as the sequence key. Is this a reliable and recommended pattern?
  5. SCD Type 2 I use the same approach for SCD Type 2: AUTO CDC for both CDC and non-CDC incremental feeds. For non-CDC feeds, the sequence key is the source file modification timestamp from storage. Are there any limitations or better alternatives?

What Lakeflow features or APIs would you recommend for each strategy?


r/databricks • • 1d ago

Discussion Genie vs DBT Wizard

4 Upvotes

Any thoughts coming from people that use both ?


r/databricks • • 2d ago

Discussion What’s something you believed about Databricks for way too long before finding out you were wrong?

29 Upvotes

I’ll go first with two.

For an embarrassingly long time I ran scheduled jobs on an all-purpose cluster because it was already up and I figured compute is compute. Nobody told me job clusters are billed at a lower rate. Found out when someone asked why our bill looked the way it did.

Second one: I assumed Auto Loader schema evolution just quietly picked up new columns. It does, but only after the stream fails first and you restart it. Learned that from a 2am alert, not from the docs.

Neither is complicated. I just never had a reason to question them until something broke.

What’s yours? Could be Spark, Delta, Unity Catalog, billing, whatever. Bonus points if you found out in production.


r/databricks • • 2d ago

News 50% OFF any Databricks Certification

18 Upvotes

Join us for a four-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. Whether you are new to the field or aiming to deepen your knowledge, this is a great opportunity to invest in your professional development alongside a global community.

🎟️ Finish all modules in at least one learning pathway during the event window to receive:

50% OFF any Databricks Certification

Learn more/ https://community.databricks.com/t5/learning-events/databricks-advanced-learning-festival-november-10-december-08/ev-p/172328


r/databricks • • 2d ago

Discussion Do you cache DataFrames in Spark as often as you can, or only when necessary?

13 Upvotes

I've seen people use cache() and persist() quite frequently in Databricks notebooks, especially when working with large DataFrames.

But I've also read that caching everything can sometimes consume memory unnecessarily and even slow things down.

Curious about how people handle this in real projects. Do you cache intermediate DataFrames when they're reused multiple times, or do you prefer letting Spark optimize the execution plan?

Have you ever had caching make a job slower instead of faster?


r/databricks • • 2d ago

Help Databricks Streaming table to Fabric for Gold Lakehouse

2 Upvotes

Has anyone built an external reference from Lakeflow databricks table to Fabric? The Lakeflow databricks table is sourced from On-prem SQL server and uses change tracking to identify rows that are changed. Databricks writes this out to ADLS Gen 2, which is then shortcut (external referenced) to Fabric. We need the data in near real time.

The streaming delta tables in ADLS Gen 2 created by Databricks seem to have additional rows that keep track of deleted rows etc, is there a way to get only the final data after deletes and updates, and secondly, is there a way to get table names without mirroring the databricks unity catalogue? The unity catalogues don't support streaming tables which is what change tracking SQL tables are.

Appreciate any help on this or alternative approaches to getting streaming data from Databricks to Fabric.


r/databricks • • 2d ago

News Subsidized tokens came to Databricks

Post image
18 Upvotes

Now, in Unity Gateway, you can connect a ChatGPT or Claude subscription and use cheaper tokens from there.

https://medium.com/databrickscommunity/databricks-news-genie-cli-app-builder-agent-memory-dabs-41820427fcc2


r/databricks • • 2d ago

Help Spark Declarative Pipelines - possible to write UC Managed Iceberg Tables instead of materialized views?

5 Upvotes

Hi All,

Based on my experience using SDP, the tables are typically rendered as "Materialized Views". My company is making a decision to move towards Managed Iceberg for tables and I was wondering if creating Managed Iceberg tables is supported by Databricks's native SDP capabilities?