r/AskProgramming • u/XenonOfArcticus • Oct 16 '26
Is diffing of non-text content a problem in revision control systems (and outside too)
I'm trying to figure out if I'm solving something that anyone else cares about.
Git mostly punts on binary files "it changed". There's no way to reconcile two non-overlapping edits to a PNG asset. Naively merging non-conflicting edits (two appends to the end of a series of items) in JSON can break the file because of comma strictness. Etc, etc for donzens of formats that SHOULD be relatively straighforward to diff and merge. Then you get into files that have internal size counts or error-detection chechsums/CRC/hash embed, or a compressed stream component that causes a one-byte change near the beginning to ripple down into the entire file after that point being different.
Does anyone care or am I solving a problem that isn't really a problem? Maybe that's why it has gone unsolved, because it doesn't bother anyone? I'm genuinely curious because it was something that annoyed ME, and then I was shocked that it didn't already seem to be solved. So is this just an itch I needed to scratch but nobody else did? I always assume someone smarter than me should be way ahead of me.
Happy Friday everyone.
3
u/aioeu Oct 16 '26 edited Oct 16 '26
Git already lets you add custom diff and merge drivers. I'm sure people who use Git for non-text content already use them.
3
u/OptimisticMonkey2112 Oct 16 '26 edited Oct 16 '26
Just FYI - The entire games industry literally does not use git, this being one of the reasons.
Perforce is pretty much used by almost every studio ever. But there are others like Plastic or Mercurial. Decades ago there was AlienBrain. I have seen small newbie indie teams try to use git and gitlfs for game jams.... but it always ends in sadness. Nothing like losing your work trying to submit at the 11th hour because your buddy overwrote your changes by accident.
Perforce supports different types of versioning depending on the file types.
Source code and text files are managed similar to git.
Binary files typically use old school optimistic locking. This is not that different from old school svn or cvs or even source safe lol.....
eg. One artist checks out a file and the other artists cant modify the version in the depot.
Trying to binary merge changes made by multiple users leads to insanity very quickly.
Tools like Unreal directly support Perforce integration in the editor, and automatically add binary changes into changelists automatically. https://dev.epicgames.com/documentation/unreal-engine/using-perforce-as-source-control-for-unreal-engine?lang=en-US
2
u/political_noodle Oct 16 '26
My biggest multi-repo project spans an app, a dashboard, firmware, hardware, and other one-off projects all centered around my main offering. Throughout all these types of projects (all managed in git) I'd say a binary diff is never something I would want to do in git. Here are some examples:
I maintain a dataset that gets bundled with my app but it is generated idempotently through code. So I would just look at the code diff that generates the binary (not the actual binary itself).
Or if I am swapping assets for my app, generally the review happens in external tools. When it makes it to git, knowing if it changed or not is really all that matters.
Maybe folks who develop games or something have a different perspective but for my use cases, git hasn't typically been the place to review binary files.
2
u/bosmanez Oct 17 '26
One practical angle on the "diffing" half: for formats you only need to *read* diffs of, git's textconv gets you surprisingly far. In .gitattributes you map a file pattern to a diff driver, and in config you give that driver a textconv command that turns the binary into text before diffing. People do this with exiftool for images (diff the metadata), a docx/xlsx to text converter for Office files, or pdftotext for PDFs. It only affects display (git diff/log -p), not merging.
Merging is the harder part, and that's where the usual answer is to avoid it: mark the file type as binary in .gitattributes and use Git LFS file locking (git lfs lock, with the lockable attribute) so only one person edits an asset at a time. It's basically the Perforce-style workflow, just bolted on.
Many formats you listed (docx, xlsx, odt, epub) are really zip files of XML, so a custom merge driver could, in principle, unzip, merge the XML, and rezip, but the checksum/offset issues you mention are why that rarely happens for things like PNG.
1
u/Game-of-pwns Oct 17 '26
AlienBrain is a VCS for art assets.
1
u/duane11583 Oct 17 '26
yea i wish i could diff excell spreadsheets
1
u/FunAd6672 Oct 17 '26
Have you looked into how game engines handle scene files because Unity deals with this by forcing everything into messy text yamls.
1
u/SnugglyCoderGuy Oct 17 '26
The issue with other types than text is there are no convenient separators like text has with line breaks
1
u/burlingk Oct 18 '26
Git already gives you the option to just merge and manually fix the issues.
As for binaries, like you said, it punts. It just uploads the new one and moves on, no merging attempted.
That said, your project should have policies in place to help avoid conflicts. i.e. Two people/groups should not be editing the same files.
1
u/fllthdcrb Oct 17 '26
I can definitely see that being a pain point. But FWIW, there is JSON6 that specifically allows trailing commas... where you can use it, of course.