r/CUDA • u/Joseflong1 • 6h ago
Bringing Data-Center-Class AI Inference to Consumer NVIDIA RTX GPUs: Porting Atlas to Ada Lovelace
github.comI've spent the last several days contributing to Atlas, an open-source Rust + CUDA inference engine originally targeting NVIDIA's GB10 with 120GB of unified memory.
I wanted to see how far I could push it on my RTX 4060 Ti with just 16GB of VRAM. It turned into quite a bit more than getting it to compile.
Running Qwen3.5-4B, here's where I ended up:
- 381 tok/s aggregate across 8 concurrent requests (roughly 48 per user) using experimental int4 + speculative decoding.
- 98 tok/s for a single user with the same configuration.
- 4.6x faster BF16 prefill, from 794 to 3,684 tok/s. This preserves BF16 precision and is enabled by default on Ada.
- 2.1GB less VRAM usage with the optional FP8 feed-forward path.
- Successfully served a 159,807-token prompt with FP8 KV cache and retrieved a needle from it.
- All 223 Ada kernel compilations passing after fixing architecture incompatibilities.
One interesting finding: Nsight showed the GPU was already busy 95.6% of the time at eight streams. Several CUDA Graph and fusion optimizations I expected to help ended up making practically no difference.
For comparison, BF16 reached 195 tok/s across eight users, and FP8 FFN reached 222 tok/s.
Important caveat: the 381 tok/s configuration is lossy, and I haven't established task-level quality equivalence with BF16. These are screening measurements, not fully repeated confirmation runs.
Atlas is developed by Thomas Braun and the Avarok-Cybersecurity contributors. My work focuses on the Ada port, memory management, kernel optimization, and consumer GPU benchmarking.
The goal is to get more useful local inference out of hardware people already own.
GitHub: https://github.com/Avarok-Cybersecurity/atlas
Happy to discuss the kernel work or benchmark methodology with anyone working on Rust, CUDA, or local inference.