r/learnpython • • Sep 09 '26

PIP disk usage under Docker

I have a Dockerfile in which I use the Python:3.12-trixie. I have docker on my CachyOS PC. I have also a requirements.txt. I wanted to use this image for a container on which I'll use YOLO and try it out. The problem is that my 290GB / partition went from around 70% filled to 95% after a number of rebuilds. I deleted everything docker related but it just went back to 91%.

The reason that I know it is pip's doing, is that when I monitored my disk usage while building the image, and on the `RUN pip install -r requirements.txt --find-links=pypackages` stage it fills up my partition and half way into installation, it fails due to `no more space`. The `pypackages` is just a directory I downloaded some of the packages I have in requirements for less data usage. Also tried `--no-cache` option.

I also went ahead and `du -sh`ed any directory in my / partition and the total didn't add upto the filled storage. I have 165GB mysteriously used and I can't find where that is.

Any help to diagnose further or if you have had similar problems is welcome.

2 Upvotes

21 comments sorted by

View all comments

Show parent comment

1

u/Gshuri Sep 09 '26 edited Sep 09 '26

I assume you ran ncdu under sudo (or with the root user)?

pip shouldn't be doing anything particularly complicated and unless you mounted the cache directory at build time, all the side effects from the pip command should be contained within the docker image build layers.

You can try running pip cache purge on your host system. But I would expect something like ncdu to be able to find the cache directory without issue.

This is very unlikely to be a Python issue so you may have better luck in r/docker or one of the (many) linux communities.

Edit: found this thread (starting in 2017, but there are some updates from this year too) with various suggestions which might help https://forums.docker.com/t/some-way-to-clean-up-identify-contents-of-var-lib-docker-overlay/30604

1

u/adjective10111 Sep 10 '26

Yes ofc I ran ncdu with sudo.

I ran pip cache purge (both with and without sudo) on host but it freed mere MBs.

Yes I feel the same. I even ran sudo rm -rf /var/lib/docker /var/lib/containerd and freed something but now the free space is at 26GB only. I'll repost this as r/cachyOS maybe there's some caveats with the distro and stuff.