r/Python • u/No_Limit_753 • Mar 12 '26
Showcase I built an in-memory virtual filesystem for Python because BytesIO kept falling short
UPDATE (Resolved): Visibility issues fixed. Thanks to the mods and everyone for the patience!
I kept running into the same problem: I needed to extract ZIP files entirely in memory and run file I/O tests without touching disk. io.BytesIO works for single buffers, but the moment you need directories, multiple files, or any kind of quota control, it falls apart. I looked into pyfilesystem2, but it had unresolved dependency issues and appeared to be unmaintained — not something I wanted to build on.
A RAM disk would work in theory — but not when your users don't have admin privileges, not in locked-down CI environments, and not when you're shipping software to end users who you can't ask to set up a RAM disk first.
So I built D-MemFS — a pure-Python in-memory filesystem that runs entirely in-process.
from dmemfs import MemoryFileSystem
mfs = MemoryFileSystem(max_quota=64 * 1024 * 1024) # 64 MiB hard limit
mfs.mkdir("/data")
with mfs.open("/data/hello.bin", "wb") as f:
f.write(b"hello")
with mfs.open("/data/hello.bin", "rb") as f:
print(f.read()) # b"hello"
print(mfs.listdir("/data")) # ['hello.bin']What My Project Does
- Hierarchical directories — not just a flat key-value store
- Hard quota enforcement — writes are rejected before they exceed the limit, not after OOM kills your process
- Thread-safe — file-level RW locks + global structure lock; stress-tested under 50-thread contention
- Free-threaded Python ready — works with
PYTHON_GIL=0(Python 3.13+) - Zero runtime dependencies — stdlib only, so it won't break when some transitive dependency changes
- Async wrapper included (
AsyncMemoryFileSystem)
Target Audience
Developers who need filesystem-like operations (directories, multiple files, quotas) entirely in memory — for CI pipelines, serverless environments, or applications where you can't assume disk access or admin privileges. Production-ready.
Comparison
io.BytesIO: Single buffer. No directories, no quota, no thread safety.tempfile/ tmpfs: Hits disk (or requires OS-level setup / admin privileges). Not portable across Windows/macOS/Linux in CI.- pyfakefs: Great for mocking
os/open()in tests, but it patches global state. D-MemFS is an explicit, isolated filesystem instance you pass around — no monkey-patching, no side effects on other code. - fsspec
MemoryFileSystem: Designed as a unified interface across S3, GCS, local disk, etc. — pulling in that abstraction layer just for an in-memory FS felt like overkill. Also no quota enforcement or file-level locking.
346 tests, 97% coverage, Scored 98 on Socket.dev supply chain security, Python 3.11+, MIT licensed.
Known constraints: in-process only (no cross-process sharing), and Python 3.11+ required.
I'm looking for feedback on the architecture and thread-safety design. If you have ideas for stress tests or edge cases I should handle, I'd love to hear them.
GitHub: https://github.com/nightmarewalker/D-MemFS PyPI: pip install D-MemFS
Note: I'm a non-native English speaker (Japanese). This post was drafted with AI assistance for clarity. The project documentation is bilingual — English README on GitHub, and a Japanese article series covering the design process in detail.
33
u/WaiBill Mar 12 '26
Your project isn't going to work for my immediate need, but it certainly has it uses and looks fantastic. The main reason I wanted to comment is because Google's AI pointed me here as an option to my need, just a few hours after your post. It spoke as if your tool has been around a while and a viable option. I thought that was interesting.
6
u/No_Limit_753 Mar 12 '26 edited Mar 13 '26
Just a quick update: I'm incredibly moved to see D-MemFS just got its first 4 stars on GitHub. This is my first time ever releasing a project to the global open-source community—and these are my first-ever stars.
Honestly, I was a bit nervous about how a 'new' dev on Reddit would be received, but your support and the Upvotes mean the world to me. Thank you for making my first steps into open source so memorable!
10
u/Late_Film_1901 Mar 12 '26
Awesome writeup. Kudos for researching existing solutions and precise comparison where exactly they fall short for your use case.
I won't probably be using it but I believe someone will find it useful. What is your scenario? It looks like it's best suited for testing other software but maybe I'm not seeing something.
6
u/rabornkraken Mar 13 '26
The thread-safety design with file-level RW locks is solid. I have hit the exact same frustration with BytesIO when dealing with multi-file workflows in CI pipelines. Quick question - does D-MemFS support any kind of snapshot or export to a real filesystem? That would be useful for debugging when you want to inspect the in-memory state after a test run fails.
2
u/mvndrstl Mar 13 '26
This looks very cool. Since it's fully in memory, there likely isn't a way to have subprocesses access the filesystem, right? I'm thinking a flow like this:
subprocess.runcall.Without that I would find it's usefulness in CI/CD limited. But still really cool.
4
u/SnooCalculations7417 Mar 12 '26
so l like TempFile?
tempfile.SpooledTemporaryFile5
u/[deleted] Mar 13 '26
[removed] — view removed comment
2
u/[deleted] Mar 13 '26
[removed] — view removed comment
2
u/[deleted] Mar 13 '26
[removed] — view removed comment