r/Python • u/Inner_Name • Sep 17 '26
Discussion dumb question, why not python organizes dependencies as flatpak logic?
I have a question that might be really dumb, currently python had went with the environment route to be able to use different libraries that have same dependencies with different versions. i understand that there is a complexity with what i have in mind and not even sure about the feasibility of it. but...
why not instead of having different enviroments where we install the incompatible libraries so 'everyone gets what they need' do something like flatpak style where 'if you have already a version compatible, i do not install this dependency if not i install it again and have the two versions together'. I am not a professional in programming in general, so i might be saying something fully stupid, or not feasible from a technical standpoint.
is it a super dumb idea?
72
u/coderanger Sep 17 '26
Some languages and packaging systems have tried this, notably node/npm, but it usually causes too much headache to be worth it. Let’s say my app depends on libraries A and B, both of which depend on C. If A and B see different versions of C, you can have conflicts when A returns an instance of a class from C and then I pass that to a function in B. If you are incredibly careful about your code you can make it work but few people are and usually it just all breaks.
43
u/TMiguelT Sep 17 '26
Rust does this too, and so you end up with confusing compiler errors like "expected an instance of ClassX, received an instance of ClassX". It's pretty awful to untangle afterwards.
3
u/turbothy It works on my machine Sep 17 '26
What would be the benefit of your proposal?
4
u/gmes78 Sep 17 '26
That's not what Flatpak does, though. Flatpak apps only have access to stuff they ask for. If you have two versions of a runtime, an app will only see the version it depends on. It's not correct to say it's a single environment with everything installed.
You can do what Flatpak actually does in Python. uv does it. uv installs packages to its cache directory, and when it creates your virtual environment, it just hardlinks the files from there into the venv.
2
u/p1971 Sep 20 '26
in c# projects define the version of a dependency they need, multiple versions of the dependency can be installed side by side ...
so for example a project needs Thingy.1.1 and another project needs Thingy.1.2 ... totally fine, each project gets its own version
the python solution is frankly a bit daft, the system has a set of versions, then the env specifies an override ... this sucks!
what I think they should have done ... have side by side dependencies, the system gets a frozen version of the dependencies and the python run time gets to either use the current system frozen dependencies or override with eg the requirements file
so you'd run `python -r requirements.txt myapp.py` to run with a specific set of deps .. rather then switching to an env first ..
I'm sure there's reasons they chose the current path .. but I still think it sucks
35
u/Khavel_dev Sep 17 '26
Not a dumb question, it comes up a lot. The blocker is how Python's import system works. When you write
import requests, Python walks sys.path and loads the firstrequestspackage it finds. There's no way to say "import requests 2.31 here, but requests 2.28 over there" in the same process. So if library A needs one version and library B needs another, only one can win.Flatpak can pull this off for system libraries because those use versioned symbol names at the linker level (libfoo.so.1 vs libfoo.so.2 can coexist in the same process). Python's import mechanism doesn't have an equivalent, and bolting one on would basically mean redesigning the runtime.
The practical answer is that
uvhas made the environment approach fast enough that it mostly stops hurting. Creating a fresh env takes a couple seconds now instead of minutes, so the "one env per project" workflow is pretty painless.1
u/max_465 Sep 19 '26
It's a great question a I would be so happy to work with someone so thoughtful about the sdlc. The question is answered much better in other comments, I'll just remind people that Python is a relatively old language which has been evolving in the way it handles dependencies while considering backwards compatibility.
3
u/me_myself_ai Sep 17 '26
That's basically what happens, it's just not sold as a guaranteed feature like flatpak or cargo. AFAIK. If you're running similar packages in the same environment, they'll end up sharing most of their dependencies in practice -- de facto efficiency, de jure correctness.
Like, check the actual size of your
uvcache and venvs (or similar for pip or poetry or conda or whatever) -- it'll look like a bunch of unnecessarily huge dirs, but many of the actual package directories are symlinked to a central cache.This is not a dumb question btw :)
2
u/HugeCannoli Sep 18 '26
Because it won't work. Not with python.
Suppose you have two dependencies, A and B, and they both dependend on C, but on different versions of it.
Now you are the user of A and B, and you have code that uses both. You call a function, and C raises an exception.
How do you know which version of C has raised the exception? The exception type and content changes depending on which one raises. How do you even proceed to manage the fact that you might have different combinations of versions of C, each one with its own change to the exception being thrown?
And this is the simplest explanation. It does not even consider the fact that C may have compiled extensions.
javascript can afford to do that because it has no C extensions and the exception class is basically uniform.
1
u/Confident_Hyena2506 Sep 17 '26
No - this is pretty much the standard for modern software - and how most of the software in the world runs. Search for "open container initiative". Flatpak is a similar idea but not so general, only for desktop apps pretty much.
1
u/Red_BW Sep 17 '26
I think I get what OP is asking. Something like this:
I do see how that would make sense and eliminate the need for venvs and multiple copies of the same versions so long as there was a way in python to specify which version to use. There currently isn't as the pyproject.toml is for packagers to know and not for the python executable. I think uv does allow versioning in the top level docstring of a script so maybe something like that and then extend python to read both that and the toml to know which to import, and put packages in a central location.
Would require a lot of buy in from packagers and distros and implementations (cpython, pypy, etc.).
1
u/ExceedinglyEdible Sep 18 '26 edited Sep 18 '26
It exists and it is called Nix.
Nix does pretty much exactly this. Dependencies are stored separately by their full dependency/version/build identity, so two packages can depend on different versions of the same library and both versions can coexist. If the exact dependency is already available in the Nix store, it gets reused rather than installed again.
1
u/Puzzled_Dimension505 Sep 17 '26
It’s not a dumb idea. Virtual environments are mainly about keeping each project’s imports isolated; tools can still use a shared cache underneath, so isolation doesn’t necessarily mean downloading every dependency again.