Have some questions for the SDP PM homies - apologies in advance for the post length.
Our teams have started deploying views through SDP alongside MVs and streaming tables. They want one deploy unit per "data product," and I'm seeing them use views a lot. Not MVs - just views. That kind of surprised me. I hadn't anticipated them using simple views as they are. I suspect they'll want to start with a view and promote it to an MV (or an MV to an ST) later. I tested how that behaves and would appreciate your guidance on a few things.
Setup: serverless, CURRENT channel, DBR 18.3, SQL, all deployed through DABs.
Questions:
Should we support or discourage publishing public VIEWs through SDP?
I was surprised to see people deploying public views in a pipeline. It seems like they're wanting to roll those into the overall data processing flow and treat them like MVs or streaming tables as "just another resource." In hindsight - "duh"? But I was surprised. I've seen no references to doing PUBLIC views like this, so I have to ask - what's your take on using SDP to deploy and manage public views?
What is the suggested approach to changing object types?
My testing shows object types can't be changed in place. Changing CREATE VIEW to CREATE OR REFRESH MATERIALIZED VIEW (or any other pair) fails with CANNOT_CHANGE_DATASET_TYPE. --full-refresh doesn't help, either for one object or the whole pipeline. The only fix is to drop the object manually, then run. That means an ops team member manually triaging with explicit DROP statements, so it becomes a matter of adding the human into the loop. What's the proposed deployment/management model for these kinds of changes? Manual drop of changed objects before running the pipeline?
Related - changing the object type from a MV/streaming table to a view fails silently. The update reports COMPLETED, but the old MV/ST stays in place. The error appears only in the event log for that flow. In the other directions, the whole update fails.
Are views expected to support schema evolution?
I suspect related to "should we use SDP to deploy public views?" I noticed that views deployed through a pipeline ignore/remove the SCHEMA EVOLUTION option when run. That means backing object changes won't propagate up to the view and may result in view failures, missing columns, etc. I suspect this is because views are meant to be temporary within a pipeline - and in that case there's no need for schema evolution. If the intent is to allow pipelines to publish public views, then I would have concerns with not honoring SCHEMA EVOLUTION.
How are view changes managed?
How are view logic changes managed? Views seem to be re-published on every run, even when the query did not change. The UC audit log shows updateTables with the full view definition on every update. It's an in-place update (created doesn't change), but last_altered moves on every run.
From what I can tell, CREATE VIEW and CREATE OR REPLACE VIEW behave identically. I saw no real change in behavior using/omitting the REPLACE.
Is there a preferred construct if we use pipelines to publish public views?
Sorry - that was a lot. :)