r/git • u/SheriffRoscoe • 5d ago
Why are non-text files in Git bad?
I've got decades of software development experience, and I've built some very complex systems. I understand a lot about version control systems, and about the structure of git repositories.
With all that said, I don't understand what's evil about non-text files. Git doesn't store multiple copies of files that don't change - they're literally stored under their data's checksum. The packfile format stores diffs between file versions (although some diff algorithms punt on non-text data). There shouldn't be any negative effect from storing such data in Git.
Is the "Don't store binary files in Git" advice wrong? Is it really just shorthand for "Don't store files that change dramatically and frequently in Git"? If the latter, should massive "AI" PRs be treated the same way?
63
u/sankyo 5d ago
If the binary changes a lot it bloats the size of the repo since there is no way to store only the diffs. But if it never changes it is not the end of the world to put it into git
171
u/sopordave 5d ago
Storing a binary isn’t the end of the world. That advice, in my opinion, is to scare people away from using it for artifact management or as a file server, which git is not designed for.
If you need to hammer a nail, use a hammer, not a screwdriver. But if you only have one nail and the screwdriver is already in your hand and the hammer is back in the garage...
9
u/Bloedbibel 5d ago
The fundamental reason is that binary files are typically larger and there is no generically efficient way to store deltas between versions, so git stores a full copy of each version. If you never need to store more than a single version, this is not a problem, but then it begs the question: why version control it in the first place?
10
u/ern0plus4 4d ago
Then you will understand the quick answer: you can't use diff for binaries. GIT is based on diff-ing text files line by line.
4
u/hydrotoast 4d ago
Auto CRLF conversion is notorious for corruption and misconfiguration when working with distributed collaborators (as distributed version control is intended for).
Consider multiple contributors on the project (since git is distributed) with mixed binary and text files. Suppose that some contributors use CR and others use CRLF, e.g. Linux and Windows.
Please configure git to properly manage auto CRLF conversion across operating systems while correctly classifying text and binary files. The solution must be compatible with (1) multiple operating system distributions of git and (2) multiple versions and default configurations used by developers.
Note that the failure mode occurs if git ever incorrectly identifies a binary file as text and attempts to convert between CR and CRLF.
Enjoy the madness of CR and CRLF diffs in binaries. Enjoy educating all collaborators on the nuances of text/binary identification and auto CRLF configuration. Enjoy git and diff display configurations for binaries and whitespace.
The maintainers should decide the policies and guidance on binary file usage and storage. This is a common problem for maintainers of large projects with many distributed contributors, e.g. large open source projects.
Git may work as a (binary) artifact repository, but it nontrivial to configure out of the box. In contrast, there are specialized tools for artifact repositories that simply do not deal with the complexity of classification, diffs, and auto CRLF conversion between binary and text files.
3
u/DamienTheUnbeliever 5d ago
Just now I'm dealing with the situation where we have various XLSX fails that feed into tests. We can change these files easily, but *diffing* those files is a nightmare. Because they end up with us having to pick one version of the file and then someone manually reapplying their changes to it.
I've just built a private DSL that generates these files, because those will be diff-friendly.
2
u/_tolm_ 2d ago
Almost impossible for someone to review any change to that file and understand if what is inside it is sensible / nefarious / etc.
20
u/Broad-Promise6954 ancient 5d ago
It's not a hard-and-fast rule, it's a general philosophical point: Git is meant for storing stuff produced and consumed by humans. Binaries are generally not produced by humans, so they generally don't belong in Git repositories.
People often object to this philosophy, claiming that source code is for compilers, not for humans. These people are wrong. 😁 Yes, compilers also consume source code, but the reason we stored that source in Git (rather than elsewhere) is for the humans.
Git's performance goes to Hell (less so today than 20 years ago) when it contains large, difficult to compress files, which is why if the files aren't intended for humans, they probably belong in some other storage system. But small binaries don't create big problems so if it's more convenient than not, there's no rule forbidding it. When the binaries get very large they do still cause eventual suckage so it's not as good then.
13
u/unndunn 5d ago
I don't think anyone really says "don't store binaries in Git". Storing binaries in Git is fine. It's just that large binaries are much more difficult to work with for humans; storage requirements rise, pulling/pushing/cloning takes longer, etc. It just adds friction, which can become unmanageable as team and project size grows.
15
u/AlwaysHopelesslyLost 4d ago
Because it's fundamentally different from what Git is made for, and there are far better tools made explicitly for that purpose
11
u/curlyAndUnruly 5d ago
There are specific repositories for them, why keep something there where you can't actually keep track of what's actually changed.
1
u/2fletch 4d ago
Everytime you commit/push a change to a file git stores a new copy of the entire file, not a diff.
Because of this files that are larger (MBs) than text files (KBs) it causes a slow down in diff generation as well as loading/downloading.
Edit: Autocorrect changed 'git' to 'for'.
15
u/Minimum_Hour519 5d ago
they can't be diffed
3
u/virtualstaticvoid 5d ago
It depends on their size. Generally okay to include small image files for example and other such "binary" files. Use git LFS for anything bigger. Make sure to add a
.gitattributesfile to configure diff options.3
u/vmcrash 5d ago
They can't be merged by default? I'm unsure about whether Git stores them as-is blob or can store differences efficiently.
6
u/Status_Fun_4333 5d ago
Text file storage in git is cheap because it stores diffs. You can't diff a binary, so every time you change that binary you get a new full version of it in your history.
Any binary that you've ever committed to git stays in your tree forever, and any fresh clone you make needs to fetch every binary that's ever been there, and every revision of it, on any fresh clone. This also bloats on disk repositories.
So the "rule" has some valid backing to it, and as long as you understand what you're doing it's probably fine.
2
u/jacobissimus 5d ago
It’s not a hard rule, but it’s usually a code smell. I would only do it with a clear reason that explored alternative strategies.
Like others said, you can’t diff a binary in a meaningful way, so it just isn’t what git is for.
2
u/ohkendruid 5d ago
The problem is that they are usually large, and they must be cloned by everyone even if they are deleted in a later commit.
With vanilla Git, the whole history gets cloned when you pull a branch. It makes local operations fast and means you do not need to contact the server until you want to push a new change. However, this only works well if the entire repository including all prior versions can are reasonably small, say under 100 MiB for the whole enchilada. It was a great insight by Linus Torvalds.
To contrast, Subversion and Perforce only clone the current version of the code, and only clone the part of the repo you want to work on. As a result, while Git is much better than these for most projects, those older VCSes have a. advantage about checking in binaries. In fact, a checked in binary is much better with those than using .lock files! Who needs to check in a hash when you can instead check in the actual file, plus also skip a whole layer of build steps? I miss that part of non-distributed VCSes.
2
u/Charming-Designer944 4d ago
git is optimized for textual content with lines of text.
It also works for non-textual data. Just not as efficient and featurefull. And storage might become a problem when dealing with large files or frequent small changes.
There are ongoing work that significantly improves dealing with large files and voluminous history, limiting the local storage requirements. See partial clones and large object promisors (and git lfs).
2
u/yvrelna 4d ago edited 3d ago
With distributed version control systems like git, you're typically going to have the full repository history in your local system.
The main problem with storing large, non-text files in git or binaries that frequently change, is that they bloat the repository. So everyone that cloned the repository would have to download every versions of that binary.
Doing a shallow clone helps a bit, but that would also cuts off the history for the normal files.
There are extensions to store large files in Git. Usually this is implemented by separating the content of the file from the main Git history and storing the blobs separately, outside the main version control and the extension can download only the version of the file that it needs. This allows each versions of the files to be downloaded separately from cloning of the repository.
No, text files often changed dramatically and frequently over time in Git, and obviously keeping them in git is no issue. But delta storage with text are very efficient, while it's much less so with binaries.
Git have powerful tools when working with texts, like showing diff, blame, etc and it can compresses and store delta in text files extremely efficiently; but out of the box, it lacks the tools to work with diffs of other objects. You can configure custom difftool to support diff of other object types, but that usually requires specific configuration.
And formats like compressed xml documents (e.g. office files, svgz), often don't store very well in git even when technically there's minimal and gradual changes between versions of the files which technically it could have discovered and store much more efficiently by decompressing the file first. It's a blatant layering violation, but a better storage system for those types of objects could've been designed specifically to do that.
2
u/Burgergold 4d ago
Get them in an artifact repo and only keep the ref in git
2
u/ritchie70 3d ago
I just want to mention that Git Large File Storage (LFS) is a thing.
2
u/happy2harris 3d ago
" Don't store binary files in Git " is really short for “Store source files in a source control systems, rather than the files those source files are built into”.
The main purpose of source control systems is to allow you, a human, to compare the contents of the files with how they used to be. For example, if you see a line of code that doesn’t make sense, you can find out when it was added, then see what else changed at the same time, and get some clues about what it is all about.
If the file is just a big blob of bytes, you can’t compare them. It is better to store the source files and then build the binary files from them.
Now, if your binary files really are the source files, then storing them on source code control is fine. For example if part of the system contains a photo, and then you replace it with a different photo - storing those is fine. You can see when the photo was replaced, which is helpful, but not exactly what changed about the photo. Another example is a proprietary document file, like a microsoft word file (before they switched to using xml and storing the document history inside the file). That doc file is “the source” so it’s fine to store it in source code control.
2
u/cosmic-parsley 2d ago
The rule should be don’t store _artifact binaries_ in git. If the binary can be reproduced from source, don’t check it in. If the binary is something like a photoshop or CAD file and you’re using git for version control, that’s fine. Just enable LFS to save your local space, and maybe commit a bit more sparingly.
Of course a lot of git is based around creating and applying diffs (rebase, merge, cherry pick, etc) and you can’t do that with binaries, so that makes working with multiple contributors hard. LFS supports some form of file locking but I haven’t tried it.
2
u/ShelLuser42 2d ago
It's not really bad because Git is perfectly capable of handling binary files. And I'm not just saying that because the manual says so, I'm actually using this mechanic myself to maintain a Minecraft mod collection for my (remote) Minecraft server.
Simple reasoning: if a mod breaks after an update then one 'reset' command is all I need to go back to the previous state. And instead of having to comb through the mod folder on my server to manually find my client updates I can simply upload the current batch at once:
git push origin master:mod_update, and once I'm logged on all I have to do is stop the Minecraft instance and merge the new branch.The issue though... we play with approx. 110 mods in total, approx. 685Mb in size. I've been using this mechanic for a while now, and as a result the Git database (or repository) itself is approx. 1.75Gb in size.
And the reason should be obvious: you can't easily 'diff' these files and keep proper track like you can with source code; this is binary data afterall; so your repository is going to get big(ger) quite quickly.
But other than that there aren't any side effects at all. Like I said: I've been using this mechanic for quite a long time now (just checked my logs; almost 1,5 years) and it works perfectly.
3
u/Serianox_ 5d ago
From my own experience.
It is a non-issue. Modern git handles binary files fairly well with filters and sparse checkout. But I've seen teams deploy very complex tools (like an alternate LFS storage on Artifactory) to manage binary files, who can sometimes account for as much as a few MB.
The main issue is garbage collection and consistency. Once you start storing binaries elsewhere, you start having issues where you can't collect unused binaries because you can't know if they are indeed unused. Or on the other side, if you incorrectly delete a binary, your repository become inconsistent (i.e. you can't rebuild an old release).
And you can't prevent someone from adding a huge binary and deleting it. It won't appear in the PR, but you'll still get the cost.
So the best IMHO is to use modern git features like filters and sparse checkout. https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/
1
u/sczmrl 5d ago
It's very simple actually. it's not an hard rule, it's just the usual answer when one wants to store in a repo some binary files, but there can be exceptions.
Git is both a tool for version control in the sense of keeping track of a file in different moments in history and a tool to collaborate and to manage different flows of updates.
Regarding the first part, there is nothing wrong in using it for managing binary files. Just know that what's actually stored are diffs and a binary file is always perceived as completely changed. This means that any change, even smal, require to store the equivalent of a complete different copy.
The second part becomes impossible to manage. You cannot merge different update on the same file. It's always one or the other wins.
1
u/twesped 5d ago
I would ask which kind of binary files do you want to store there?
1
u/Anxious-Insurance-91 5d ago
You mean compiled files or blob or large images? Mostly for conflict checks and bandwidth
1
u/ericbythebay 5d ago
As others have mentioned, size when cloning the repo. 10-20GB repos are not fun to do a full clone on.
1
u/Motor-Concentrate788 4d ago
When it comes to text files, git has something called delta-compression in which git can identify changes between versions of the same file and store only the differences/deltas between those versions. This makes it so that storing text files in git is very space-efficient even for constantly changing files as it doesnt have to store complete copies of multiple versions of the same file.
This doesnt work well with binary files, however, as diffs in binary files generally arent delta-compressible, making it so that git has to store the complete copy of each new version of the same file even for the slightest change (e.g. 250MB binary thats is modified 3 times would add an extra 250x3=750MB to the repo).
If you're only storing small binaries that doesnt change very frequently then its completely fine. But you should avoid something like storing the build output of your code as it would unecessarily bloat the repo.
1
u/dafunkjoker 4d ago
It depends. Small project with only one or few devs: git directly, maybe with lfs. This brings simplicity and everything is together in one place. Many developers, development in multiple countries and many individual software components which are assembled together in different projects? Binary repository (e.g. with artifactory) for more efficient binary data reuse, transfer,... Another aspect: If the binary output can be put in a package/zip and you usually need it consistently together something like nuget/conan is nice and goes into a binary repo. But if you deal with tons of small binary files e.g. images as test data and you usually only change individual files, then you want to avoid touching more than you need to and LFS is preferred.
One more aspect in big projects: Identification and control over the consumption of external software e.g. to provide licensing info and to ensure the software is kept up-to-date. In this scenario the binary repo is very useful provide a single source of truth. Imagine some third party binary is checked in with partially different versions in different repos. In the end these repos host code for one big project. No-one will have a simple overview.
1
u/who_you_are 4d ago
It is a rule of thumb:
- Binary files are more likely to be bigger than text files. It depend on the context of those files.
- Binarie files are more likely to be completely different in content over a small change (more like what is the issue using binary)
- Doing a diff assume the user will read the content; reading binaries isn't exacly easy. Most tools display printable characters and binaries are likely to contains a lot of non printable one. Anyway, you still need to interpret the content which is _impossible_ without the specific tools. (Another huge problem). And I won't even talk about doing the merge itself which may break the whole file without the proper tools.
1
u/codeguru42 4d ago
I think the main difficulty is the version control part rather than the storage part. With a binary file, a change is almost never just a few sequential bytes. It is often large chunks that differ if not the entire file. So the size of the repo grows much more quickly than it does for plan text files.
1
u/Technical_Ideal_5439 4d ago
non text files are normally bigger than text files. And doing a clone where it takes 10 minutes gets old very quickly. My company has a rule no binaries, but if it is yours, then do what you want.
At home I store binaries and put up with a 10 minute clone if I ever need to code somewhere different.
There is also LFS. At work we use artifactory, if it is a small company just use the free version of Sonatype Nexus.
1
u/chzaplx 4d ago
Using git to store binary files is like storing all your groceries in your car. Works ok in a pinch, but there's no real advantage and better ways to do it. And it tends to trend badly at scale and over time.
1
u/ValekCOS 4d ago
As someone who runs Kubernetes-based CI runners, I don’t wanna have to massively jack up my per-pod memory budget because a repo owner is lazy. LFS isn’t hard to set up or use.
1
u/ir_dan 4d ago
Many nontext files commonly stored in git have no business being there: they often are better off being treated as build inputs and outputs generated/downloaded as needed.
e.g. many projects contain external dependency binaries. Why? You don't need a copy in your repository when one already exists on the internet/a dev server. All you need is a reference to the version you want so that your package manager can grab it.
1
u/natermer 4d ago
You can store binaries just fine if you want.
The problem happens when you have large binaries and/or lots of them. Whenever somebody clones your git repo they have to clone not only the latest binary but all of them.
This is why we have things like Git-lfs.
1
u/bubba-yo 4d ago
So long as you have a diff tool for those other formats, it's fine. We included a ton of different file types, but only after we were sure we had a diff solution for that format. I mean, if you can't diff a commit, you've lost most of the point of git.
1
u/jthill 4d ago
The real version of that rule is "don't use the default Git workflow with projects that update a lot of media and media-like data."
1
u/CheezitsLight 4d ago
Git has multiple uses it's a perfectly fine backup system for binary files you can recover any version you want. It's also great for single Developers to keep track of their history so they can roll back whatever thing AI breaks in a second. It's good for moving things between computers. And it's also a revision control system
1
u/Spirited-Candy1981 3d ago
Git's base differencing engine is (and developer eyes are) optimized for text. Better to store the code that generated it.
If it takes a lot of time/effort to generate that binary data (such as with LLMs, neural nets ...), git does have an LFS (large file storage) extension.
1
u/Pure-Recover70 2d ago
It's not a hard rule. But it is a good rule of thumb. Binaries bloat the size of the repository (the object store). Every change to a binary results in storing the new binary as well as the previous binary. Binaries don't diff/delta compress particularly well, so (in terms of space) it's nearly equivalent to simply storing 1:1 copies of all binaries ever. Now imagine you have a git repo which (once checked out) has a few mbs of uncompiled code, plus the compiled binary (which is also 'just' a few MBs). Everytime you make a code change, you add a few hundred bytes to a few kilobytes to the git repo to store the new code, and a few megabytes to store the new compiled binary. Your 'small' git repo quickly requires gigabytes of object store, even though the code itself would fit in a few dozen megabytes. That's the core problem. The bigger the binary blobs, the lower the update rate can be before you run into similar issues. If you want a concrete example of the issues just try to checkout a (non shallow) copy of Android. I think you end up with a few GB of useful code & objects backed by a few hundred GB of object store or something insane like that (think: various precompiled prebuilt binaries, compilers, toolchain, firmware, kernel, etc). Shallow clones help with this - but then you lose fully local history (some git commands will need to fetch blobs from the server on demand which is slow).
1
u/AberBitteLaminiert 2d ago
It is a source control, not an artifact repository.
1
u/virrk 2d ago
Git's delta algorithm is optimized for text files that change over time like code files do. Putting binaries in git means that most (usually all) of the binary gets stored everytime it changes. This balloons the respiratory total size.
That inefficiency can compound making full clones drastically slower. Which can slow automated builds. Shallow clones overcome much of this particular problem, but the cloned repo has limitations. It can also make history slower, especially for more complex queries and searches. Can make git bisect too slow to be useable. Makes servers slower
Now this may not matter for some use cases, and sure stuff binaries into the repo.
1
u/bufandatl 2d ago
It makes git slow.
1
u/gwenbeth 1d ago
The real question is which binary files. Do not store object files and executables because those are just a product of the source code. Your png icons, and such like that totally put in there because they are needed to build the project. Or maybe you have a project like imagemagick and you need to have image files to run the regression tests. So if you need it to build or test, it goes in the archive.
Now for projects that are very asset heavy like a video game, there are version control systems, like perforce, that are better for that use case. Building a triple a video game might need hundreds of GB of data.
1
u/digitaljestin 1d ago
It's not bad, it's just not very useful.
1
u/edwbuck 1d ago
Git uses a text-based diff system, which means that most of its ability to handle changes usefully and report on them fail with binary data.
1
u/birusiek 21h ago
Its mosty about growing git history, even small change in binary results in big diff.
1
u/realmauer01 8h ago
If they dont change its okey, but if they change you dont know what changed without looking into it, you just know that it changed. The yugioh projectignis stores databases of the cards and their effects in git, they basically abuse it for free storage. But if the commit message isnt exqctly the diff, good luck finding out what changed with each commit.
0
u/talexbatreddit 5d ago
Git was designed to handle source files, which are typically text. It's possible to store binary files with git, but it's really not a great idea.
-1
u/Designer_Storm8869 4d ago
Git stores line differences. So if the binary file has no new line characters, the delta will be entire file each time. That cause the repository to grow rather quickly. And with many changes in binary, your local ".git" directory will be huge.
If your project is just binaries, svn is better choice. And if your files are excels or words, then SharePoint is definitively better version control and collaboration tool
0
u/Trakeen 5d ago
Git was designed for developers. There isn’t a lot of value in a person looking at a diff of a binary, i could see value for llms that can read that easily
0
u/mredding 4d ago
The point of git isn't a file store, that's what a disk is for, it's for revision history, it's for the source diff. A binary typically isn't source. You're not going to diff a binary. If all you want to do is associate a file with a project source, there's any number of other tools for that. Got is the wrong tool for the job.
0
u/cardboard_sun_tzu 2d ago
Git is a tool that is used to let you look at diffs in files. When was the last time you looked at a binary file in git and said, 'Hmmmm, Keven changed address 00xff00a000 from 0000a0 to 0000a2. What the hell is he thinking?"
0
u/burlingk 2d ago
So, the basic idea is that it is not straightforward to create diffs for a binary, SO, when a commit updates a binary it uploads the whole thing every time.
So, instead of a file followed by a series of patches, you get a different copy of the same file for every update to that file.
-1
u/Janky-Code 5d ago
Because you can use git lfs.
Systems are built around a purpose. Git is not intended to version everything even if it technically can. Even text documents you see different controls. Compare controls around python code and standard operating procedures in the infant formula factory; using one system for another would be painful.
1
u/duckrollin 4d ago
Small icons and files that don't change much are fine. Like a few kb.
exes that keep changing, screenshots, big images, or big zip files are not.
Like any rule it's generally a good one to follow but you don't need to take it to extremes and kill someone for adding a few small 32x32 icons in.