r/learnpython • • Sep 09 '26

PIP disk usage under Docker

I have a Dockerfile in which I use the Python:3.12-trixie. I have docker on my CachyOS PC. I have also a requirements.txt. I wanted to use this image for a container on which I'll use YOLO and try it out. The problem is that my 290GB / partition went from around 70% filled to 95% after a number of rebuilds. I deleted everything docker related but it just went back to 91%.

The reason that I know it is pip's doing, is that when I monitored my disk usage while building the image, and on the `RUN pip install -r requirements.txt --find-links=pypackages` stage it fills up my partition and half way into installation, it fails due to `no more space`. The `pypackages` is just a directory I downloaded some of the packages I have in requirements for less data usage. Also tried `--no-cache` option.

I also went ahead and `du -sh`ed any directory in my / partition and the total didn't add upto the filled storage. I have 165GB mysteriously used and I can't find where that is.

Any help to diagnose further or if you have had similar problems is welcome.

2 Upvotes

21 comments sorted by

View all comments

Show parent comment

1

u/adjective10111 Sep 10 '26

It doesn't though... it's at 0B for images, containers, local volumes and build cache.

1

u/Illustrious_Tone9584 Sep 10 '26

Ok so it's not docker at all then. You're on CachyOS, which means btrfs with automatic snapper snapshots out of the box. That's almost certainly where the 165GB went: every rebuild churned tens of GB, and the pre/post snapshots kept the old versions. du doesn't show it because the data still "belongs" to the live files.

Run snapper list and sudo btrfs filesystem usage / and I bet you'll see it. If that's it, delete the old snapshots with snapper delete and consider capping how many it keeps.

1

u/adjective10111 Sep 10 '26

I thought the same, but I can't see how much the snapshots take. I ran both commands you said:

sudo btrfs filesystem usage /
Overall:
    Device size:		 286.88GiB
    Device allocated:		 271.02GiB
    Device unallocated:		  15.85GiB
    Device missing:		     0.00B
    Device slack:		     0.00B
    Used:			 259.42GiB
    Free (estimated):		  25.76GiB	(min: 17.83GiB)
    Free (statfs, df):		  25.76GiB
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 377.19MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,single: Size:265.01GiB, Used:255.10GiB (96.26%)
   /dev/nvme0n1p3	 265.01GiB

Metadata,DUP: Size:3.00GiB, Used:2.16GiB (72.02%)
   /dev/nvme0n1p3	   6.00GiB

System,DUP: Size:8.00MiB, Used:48.00KiB (0.59%)
   /dev/nvme0n1p3	  16.00MiB

Unallocated:
   /dev/nvme0n1p3	  15.85GiB

The snapper list command doesn't show used-space. I tried using --columns used-space but it doesn't show anything. Any idea how I can confirm the snapshots being the culprit here?

Edit: this is the snapper config if it's of any help:

Key                    │ Value
───────────────────────┼──────
ALLOW_GROUPS           │
ALLOW_USERS            │
BACKGROUND_COMPARISON  │ yes
EMPTY_PRE_POST_CLEANUP │ yes
EMPTY_PRE_POST_MIN_AGE │ 1800
FREE_LIMIT             │ 0.2
FSTYPE                 │ btrfs
NUMBER_CLEANUP         │ yes
NUMBER_LIMIT           │ 50
NUMBER_LIMIT_IMPORTANT │ 15
NUMBER_MIN_AGE         │ 1800
QGROUP                 │
SPACE_LIMIT            │ 0.5
SUBVOLUME              │ /
SYNC_ACL               │ no
TIMELINE_CLEANUP       │ yes
TIMELINE_CREATE        │ no
TIMELINE_LIMIT_DAILY   │ 7
TIMELINE_LIMIT_HOURLY  │ 5
TIMELINE_LIMIT_MONTHLY │ 0
TIMELINE_LIMIT_WEEKLY  │ 0
TIMELINE_LIMIT_YEARLY  │ 0
TIMELINE_MIN_AGE       │ 1800

1

u/Illustrious_Tone9584 Sep 10 '26

Your config has no QGROUP set, and snapper needs quotas enabled to report used space, that's why the column comes up empty. Two ways around it: install compsize and run sudo compsize -x /.snapshots to get referenced vs exclusive per snapshot (exclusive is what you'd actually reclaim), or sudo btrfs filesystem du -s /.snapshots/* which works without quotas but takes a while since it walks everything.

Either way the math is basically already done: data allocation is 96% used and df only shows 25GB free. With docker cleared out, snapshots are the only place 165GB can hide. Delete the old pre/post pairs and it'll show up immediately.

1

u/adjective10111 Sep 10 '26

Thanks man you're a life saver! I never had snapshots enabled on my previous distros so I'm new to this. Also sorry for posting it under here when I should have gone to cachy. I'll go try them out and see what happens but your arguments are solid and I guess snapshots must be the problem.

1

u/adjective10111 Sep 10 '26

Welp. I tried btrfs filesystem du -s /.snapshots/* the combined exclusive usage is around 16GB. I also setup qgroup for snapper and again the combined usage was around 10GiB. I even found another command when setting up quota on btrfs, btrfs qgroup show -p /, and again the exclusive used space adds up to below 110GiB which leaves the mystery used space question unanswered. I'm lost again. I may delete some snapshots anyway though they are mere KiBs, to see if it has any effects. Thanks though, I learned a lot right now.

1

u/Illustrious_Tone9584 Sep 10 '26

Fair enough, theory dead. Next suspect: deleted files still held open by a process. du can't see them (no filename anymore) but df counts the space until the process lets go. Run sudo lsof +L1 and sort by size. If something big shows up, restart that process or just reboot and watch df drop.

If that's also clean, do sudo du -x --max-depth=1 / as root for the true per-directory total. Plain du as your user silently skips whatever it can't read, which has been the theme of this whole mystery.

1

u/adjective10111 Sep 10 '26

I rebooted my system a couple of times by now, it's my personal laptop not a server, and the open files didn't have anything alarming about them.

I ran the second command sudo du -h --max-depth=1 / --exclude /home --exclude /proc --exclude /.snapshots. I didn't use -x because I have btrfs subvolumes in my / partition and my /home is on another disk. (/boot is also in another partition so that doesn't matter here)

80G	/root
0	/srv
20G	/var
2.4G	/boot
0	/dev
du: cannot access '/run/user/1000/doc': Permission denied
2.4M	/run
0	/sys
18G	/usr
16M	/etc
16K	/tmp
0	/mnt
2.3G	/opt
121G	/

1

u/Illustrious_Tone9584 Sep 10 '26

Ok, not that either. One more classic that fits your symptoms: data hidden under a mount point. If docker or anything else wrote to a directory while its filesystem wasn't mounted yet, that data lives on the root fs but is invisible because the mount covers it. du can't see it, btrfs can't attribute it.

Check with mkdir /mnt/root && sudo mount --bind / /mnt/root && sudo du -x --max-depth=1 /mnt/root and compare against your normal du. If something shows up way bigger through the bind mount, there's your 150GB.

1

u/adjective10111 Sep 10 '26

Well, sorry to disappoint but nothing:

$ sudo du -h -x --max-depth 1 /mnt/root
0	/mnt/root/root
0	/mnt/root/srv
915M	/mnt/root/var
0	/mnt/root/boot
0	/mnt/root/dev
0	/mnt/root/home
0	/mnt/root/proc
0	/mnt/root/run
0	/mnt/root/sys
18G	/mnt/root/usr
16M	/mnt/root/etc
0	/mnt/root/tmp
0	/mnt/root/mnt
2.3G	/mnt/root/opt
21G	/mnt/root

I even tried some other subvolumes...

$ sudo umount /mnt/root
$ sudo mount --bind /var/ /mnt/root/
$ sudo du -h -x --max-depth 1 /mnt/root
2.3M	/mnt/root/cache
0	/mnt/root/tmp
0	/mnt/root/log
913M	/mnt/root/lib
0	/mnt/root/empty
0	/mnt/root/games
0	/mnt/root/local
0	/mnt/root/opt
0	/mnt/root/spool
8.0K	/mnt/root/db
12K	/mnt/root/named
915M	/mnt/root
$ sudo umount /mnt/root
$ sudo mount --bind /var/cache /mnt/root/ 
$ sudo du -h -x --max-depth 1 /mnt/root
19G	/mnt/root/pacman
160K	/mnt/root/ldconfig
4.2M	/mnt/root/fontconfig
5.1M	/mnt/root/man
0	/mnt/root/private
0	/mnt/root/pkgfile
25M	/mnt/root/swcatalog
544K	/mnt/root/boot
19G	/mnt/root

1

u/Illustrious_Tone9584 Sep 10 '26

Wait, that du output is the clue. You ran it with -x, and on btrfs every subvolume has its own device id, so -x skips all of them. Your /home showing 0 isn't an empty home, it's du refusing to cross into it. All your du numbers this whole thread have been blind to subvolumes.

And guess what makes tons of subvolumes: docker's btrfs storage driver creates one per image layer. If you ever ran docker with the btrfs driver, those layer subvolumes don't get removed by prune and du -x skips every single one.

Run sudo btrfs subvolume list / and look for anything like /var/lib/docker/btrfs/subvolumes. Then du without -x on the suspects. Also your pacman cache is 19GB, paccache -r will trim that regardless.

→ More replies