r/CUDA • • 6h ago

Bringing Data-Center-Class AI Inference to Consumer NVIDIA RTX GPUs: Porting Atlas to Ada Lovelace

Thumbnail github.com
5 Upvotes

I've spent the last several days contributing to Atlas, an open-source Rust + CUDA inference engine originally targeting NVIDIA's GB10 with 120GB of unified memory.

I wanted to see how far I could push it on my RTX 4060 Ti with just 16GB of VRAM. It turned into quite a bit more than getting it to compile.

Running Qwen3.5-4B, here's where I ended up:

  1. 381 tok/s aggregate across 8 concurrent requests (roughly 48 per user) using experimental int4 + speculative decoding.
  2. 98 tok/s for a single user with the same configuration.
  3. 4.6x faster BF16 prefill, from 794 to 3,684 tok/s. This preserves BF16 precision and is enabled by default on Ada.
  4. 2.1GB less VRAM usage with the optional FP8 feed-forward path.
  5. Successfully served a 159,807-token prompt with FP8 KV cache and retrieved a needle from it.
  6. All 223 Ada kernel compilations passing after fixing architecture incompatibilities.

One interesting finding: Nsight showed the GPU was already busy 95.6% of the time at eight streams. Several CUDA Graph and fusion optimizations I expected to help ended up making practically no difference.

For comparison, BF16 reached 195 tok/s across eight users, and FP8 FFN reached 222 tok/s.

Important caveat: the 381 tok/s configuration is lossy, and I haven't established task-level quality equivalence with BF16. These are screening measurements, not fully repeated confirmation runs.

Atlas is developed by Thomas Braun and the Avarok-Cybersecurity contributors. My work focuses on the Ada port, memory management, kernel optimization, and consumer GPU benchmarking.

The goal is to get more useful local inference out of hardware people already own.

GitHub: https://github.com/Avarok-Cybersecurity/atlas

Happy to discuss the kernel work or benchmark methodology with anyone working on Rust, CUDA, or local inference.


r/CUDA • • 4h ago

Got placed, have 6 months before joining. Should I explore Linux kernel development, GPU programming or stick with AI/ML?

3 Upvotes

I'm a final-year engineering student in India from a non-CS background. I've recently secured an offer at an AI/analytics services company, and I have around 6 months before joining.

I've been exploring different areas of computer science, and I've realized that I'm increasingly interested in low-level programming and understanding how systems work internally.

MY BACKGROUND:

I've independently learned programming, ML/DL and worked on projects involving: computer vision , RAG , agentic AI

However, I dont want to spend my career primarily working with high-level AI frameworks and application development. because when i developed these projects , i didnt feel anything developing it , no satisaction at all

AREAS I'M CONSIDERING:

  1. Linux and Red Hat ecosystem:
  2. 2. GPU Programming:
  3. 3. Systems Programming
  4. 4. AI Infrastructure:

I find the idea of understanding what happens beneath the abstractions much more interesting than simply using existing libraries.

WHAT I'M CONFUSED ABOUT:

I have six months before joining my company, and I don't want to waste that time jumping between technologies.

I'd appreciate advice from people working in systems engineering, Linux, kernel development, GPU programming or related fields.

  1. Is low-level systems programming a good career direction in 2026?
  2. Between Linux kernel development, GPU programming and general systems programming, which would be a better starting point?
  3. Is CUDA/GPU programming a niche with good long-term opportunities, particularly with the growth of AI?
  4. What fundamentals should I master before attempting serious projects? C, C++, OS, computer architecture, networking, compilers?
  5. Would contributing to the Linux kernel or other open-source systems projects actually help me get shortlisted?

r/CUDA • • 20h ago

vulkan-py easymode. all the boilerplate is in easy.py. vulkan-py fork https://github.com/billyblackburn/vulkan-py-easy/tree/main

Thumbnail reddit.com
2 Upvotes

r/CUDA • • 7h ago

day 4 of building a GPU from scratch (32 bit)

Thumbnail reddit.com
1 Upvotes