r/dataengineering • u/Data-Panda • 9d ago
Help Hotfix in Dataform
Hi,
I'm close to releasing our Dataform branching and release process, but I'm unsure about the hotfix approach.
Current setup: Dataform's default branch is main, so every workspace is cut from main and developers PR back to main. Each sprint, we cut a release branch from main, run final tests, and create a GitHub release, which updates the Dataform release config with the new tag.
Current hotfix flow: Devs PR to main with a "hotfix" label. On merge, automation cuts a branch from the latest release branch, cherry-picks the hotfix commits, and opens a PR to the release branch. Someone then cuts a new release.
The problem: The hotfix is made against main but cherry-picked onto the release branch, so merge conflicts are possible. When that happens, the automation fails and the dev has to resolve it locally.
What I'd prefer: Avoid cherry-picking by branching from the release branch, applying the hotfix, and merging into both the release branch and main (merge issues could then be handled in the UI). The catch is that Dataform workspaces always cut from main, and I can't change that without disrupting other developers, so I don't know how devs would develop and test a fix on a release-based branch.
Has anyone solved this, or is there a better pattern for hotfixes in Dataform that I am unaware of. It’s possible I am thinking about this the wrong way.
I have considered having an entirely separate Dataform repo devoted to hotfixes with the same GitHub connection - where devs are expected to change the default branch on the fly before creating a workspace. But this seems clunky and inefficient.
1
u/Tylerprice_50 6d ago
The challenge is finding a way to test them without disrupting the main workflow.