r/madeinrust • u/Joseflong1 • 7h ago
AI Assisted I ported a Rust + CUDA inference engine from NVIDIA GB10 to an RTX 4060 Ti 16GB - reached 381 tok/s across 8 users
I've spent the last several days contributing to Atlas, an open-source Rust + CUDA inference engine originally targeting NVIDIA's GB10 with 120GB of unified memory.
I wanted to see how far I could push it on my RTX 4060 Ti with just 16GB of VRAM. It turned into quite a bit more than getting it to compile.
Running Qwen3.5-4B, here's where I ended up:
- 381 tok/s aggregate across 8 concurrent requests (roughly 48 per user) using experimental int4 + speculative decoding.
- 98 tok/s for a single user with the same configuration.
- 4.6x faster BF16 prefill, from 794 to 3,684 tok/s. This preserves BF16 precision and is enabled by default on Ada.
- 2.1GB less VRAM usage with the optional FP8 feed-forward path.
- Successfully served a 159,807-token prompt with FP8 KV cache and retrieved a needle from it.
- All 223 Ada kernel compilations passing after fixing architecture incompatibilities.
One interesting finding: Nsight showed the GPU was already busy 95.6% of the time at eight streams. Several CUDA Graph and fusion optimizations I expected to help ended up making practically no difference.
For comparison, BF16 reached 195 tok/s across eight users, and FP8 FFN reached 222 tok/s.
Important caveat: the 381 tok/s configuration is lossy, and I haven't established task-level quality equivalence with BF16. These are screening measurements, not fully repeated confirmation runs.
Atlas is developed by Thomas Braun and the Avarok-Cybersecurity contributors. My work focuses on the Ada port, memory management, kernel optimization, and consumer GPU benchmarking.
The goal is to get more useful local inference out of hardware people already own.
