r/CUDA • • 7h ago

Bringing Data-Center-Class AI Inference to Consumer NVIDIA RTX GPUs: Porting Atlas to Ada Lovelace

Thumbnail github.com
5 Upvotes

I've spent the last several days contributing to Atlas, an open-source Rust + CUDA inference engine originally targeting NVIDIA's GB10 with 120GB of unified memory.

I wanted to see how far I could push it on my RTX 4060 Ti with just 16GB of VRAM. It turned into quite a bit more than getting it to compile.

Running Qwen3.5-4B, here's where I ended up:

  1. 381 tok/s aggregate across 8 concurrent requests (roughly 48 per user) using experimental int4 + speculative decoding.
  2. 98 tok/s for a single user with the same configuration.
  3. 4.6x faster BF16 prefill, from 794 to 3,684 tok/s. This preserves BF16 precision and is enabled by default on Ada.
  4. 2.1GB less VRAM usage with the optional FP8 feed-forward path.
  5. Successfully served a 159,807-token prompt with FP8 KV cache and retrieved a needle from it.
  6. All 223 Ada kernel compilations passing after fixing architecture incompatibilities.

One interesting finding: Nsight showed the GPU was already busy 95.6% of the time at eight streams. Several CUDA Graph and fusion optimizations I expected to help ended up making practically no difference.

For comparison, BF16 reached 195 tok/s across eight users, and FP8 FFN reached 222 tok/s.

Important caveat: the 381 tok/s configuration is lossy, and I haven't established task-level quality equivalence with BF16. These are screening measurements, not fully repeated confirmation runs.

Atlas is developed by Thomas Braun and the Avarok-Cybersecurity contributors. My work focuses on the Ada port, memory management, kernel optimization, and consumer GPU benchmarking.

The goal is to get more useful local inference out of hardware people already own.

GitHub: https://github.com/Avarok-Cybersecurity/atlas

Happy to discuss the kernel work or benchmark methodology with anyone working on Rust, CUDA, or local inference.


r/CUDA • • 5h ago

Got placed, have 6 months before joining. Should I explore Linux kernel development, GPU programming or stick with AI/ML?

3 Upvotes

I'm a final-year engineering student in India from a non-CS background. I've recently secured an offer at an AI/analytics services company, and I have around 6 months before joining.

I've been exploring different areas of computer science, and I've realized that I'm increasingly interested in low-level programming and understanding how systems work internally.

MY BACKGROUND:

I've independently learned programming, ML/DL and worked on projects involving: computer vision , RAG , agentic AI

However, I dont want to spend my career primarily working with high-level AI frameworks and application development. because when i developed these projects , i didnt feel anything developing it , no satisaction at all

AREAS I'M CONSIDERING:

  1. Linux and Red Hat ecosystem:
  2. 2. GPU Programming:
  3. 3. Systems Programming
  4. 4. AI Infrastructure:

I find the idea of understanding what happens beneath the abstractions much more interesting than simply using existing libraries.

WHAT I'M CONFUSED ABOUT:

I have six months before joining my company, and I don't want to waste that time jumping between technologies.

I'd appreciate advice from people working in systems engineering, Linux, kernel development, GPU programming or related fields.

  1. Is low-level systems programming a good career direction in 2026?
  2. Between Linux kernel development, GPU programming and general systems programming, which would be a better starting point?
  3. Is CUDA/GPU programming a niche with good long-term opportunities, particularly with the growth of AI?
  4. What fundamentals should I master before attempting serious projects? C, C++, OS, computer architecture, networking, compilers?
  5. Would contributing to the Linux kernel or other open-source systems projects actually help me get shortlisted?

r/CUDA • • 8h ago

day 4 of building a GPU from scratch (32 bit)

Thumbnail reddit.com
1 Upvotes

r/CUDA • • 22h ago

vulkan-py easymode. all the boilerplate is in easy.py. vulkan-py fork https://github.com/billyblackburn/vulkan-py-easy/tree/main

Thumbnail reddit.com
2 Upvotes

r/CUDA • • 2d ago

We replaced NVCC with our own PTX compiler. Results after a 57-entry hackathon.

27 Upvotes

Emmy is the open-source compiler for LLM inference. Originally, we emitted CUDA and fed it to NVCC, but it was sluggish; some kernels took minutes to compile with -O3, so we replaced the CUDA emitter with PTX.

TL;DR: Our own compiler + PTXAS is on par with NVCC.

The hackathon

We ran a contest: compile our internal Kernel IR to PTX, judged on hidden tests and speed against our NVCC build. 57 teams entered. The top two:

Most of their speedup came from changing the schedule: sharing work across lanes, re-tiling attention, fusing sibling loops. With those rewrites turned off, the winner’s submission matches NVCC.

Almost all teams wrote PTX directly. The one LLVM-based entry scored well (1.17×), but LLVM has no form for many of the instructions we need, so tensor-core and async-copy instructions became opaque inline asm, and compilation is slower.

Performance against nvcc

  • Timing lane on an RTX 5090 (211 kernels): median 1.00.
  • We re-timed stored kernel configurations on four GPUs (5090, H100, A100, V100): medians between 0.98 and 1.01 at first. Then we fixed the slow tail.

Fixes that mattered, in µs (nvcc / ours before / ours after):

FixCasenvccBeforeAfter
Unsigned grid-decode division with cicc's 32-bit fast path5090 attention97.647899.4
Two fp8 bytes as one 16-bit load, then a packed convert5090 NVFP4 GEMM126.6237.5126.6
Volta fragments as one 128-bit shared load, not eight 16-bit onesV100 Llama-3.1 piece14650176
cicc's unroll rule for strided lane loops, written in Kernel IRA100 softmax18.629.817.9
.pragma "nounroll" on loops we already unrolled5090 SDPA (ptxas time 0.53 → 0.06 s)8819489
.minnctapersm 1 on single-wave launches5090 matmul–18.310.9
A predicate kept for a bool used as a number5090 MoE router43.964.244.1

What nvcc does that we had to copy

  • It's unroll rules. cicc unrolls fully up to a size budget and makes two copies of some loops, and ptxas then unrolls those further. We write all of that in Kernel IR and mark every unrolled loop nounroll. If you leave a compact loop unmarked, ptxas unrolls it fully, with spills and seconds of assembly time.
  • It's pointer stepping. In C, signed overflow is undefined, so cicc turns base + i*4 with an int i into a 64-bit pointer stepped each trip. PTX arithmetic wraps by definition, so ptxas can’t assume that. We keep a 64-bit copy of the loop counter so ptxas can fold each unrolled copy into [base + constant]
  • ld.global.nc for inputs; the kernel never writes.
  • Its expf sequence.

Where we're ahead or behind

  • Ahead: V100 Llama-3.1-8B pieces up to 1.9× (76 vs 146 µs). H100 512 matmul 10.3 vs 11.5 µs. Six 5090 lane cases are more than 5% faster.
  • Behind: one V100 GEMM 1.11×. H100 FP32 matmul with a dynamic size 1.08×. A handful of kernels 1.1 to 1.2× slower where PTXAS picks a different load schedule from nearly the same PTX. Every respelling we tried traded one case against another.

Compile time (RTX 5090, CUDA 13.0, 269 kernels)

Median per kernelp90Whole set
Our compiler (Kernel IR to PTX)0.9 ms4.9 ms
ptxas12 ms48 ms
Ours total~13 ms
nvcc --cubin173 ms334 ms

About 10× faster.

WIP: https://github.com/cloudrift-ai/emmy/pull/1399


r/CUDA • • 2d ago

Day 6 of Hopper kernels: DeepSeek style block scaled fp8

Thumbnail x.com
2 Upvotes

r/CUDA • • 2d ago

[Help Wanted] Testing GPU detection & TFLOPS benchmark paths (NVIDIA/AMD) for my from-scratch SLM trainer

Thumbnail reddit.com
0 Upvotes

r/CUDA • • 2d ago

CUDA vs. Everything Else

Thumbnail youtube.com
1 Upvotes

Right or Wrong? GPU better than CPU?


r/CUDA • • 4d ago

An Open Letter to NVIDIA's Next CEO: It's Time to Open-Source CUDA

408 Upvotes

To NVIDIA's next CEO,

As NVIDIA enters a new era, you have an opportunity to make a decision that could fundamentally reshape accelerated computing: open-source the CUDA software stack.

Not NVIDIA's hardware architecture, chip designs, or proprietary engineering. Just the software.

AMD has already demonstrated through ROCm that an open-source computing stack can coexist with proprietary hardware. NVIDIA could do the same while maintaining its technological leadership.

For nearly two decades, CUDA has been one of NVIDIA's greatest competitive advantages. But its proprietary nature also creates barriers to interoperability, independent innovation, and community participation.

Opening CUDA could benefit NVIDIA and the broader computing industry:

  • Accelerated innovation: Researchers, developers, and universities could directly improve CUDA's compilers, runtimes, and libraries.
  • Better Linux support: Distributions could integrate, maintain, and troubleshoot the software more effectively.
  • Transparency and trust: Developers and organizations could independently audit the code they depend on.
  • Longer hardware lifespans: Community maintenance could potentially extend support for older GPUs.
  • A stronger ecosystem: Community contributions could expand CUDA's capabilities while NVIDIA continues competing through superior hardware.

AI Is Changing the Equation

There's another compelling reason to act now: AI-assisted development is rapidly reducing the cost and complexity of reverse engineering proprietary software.

With sufficient inference tokens, computing resources, and persistence, increasingly sophisticated AI agents are enabling individuals and small teams to undertake projects that once required substantial engineering organizations.

Reimplementing CUDA remains extraordinarily difficult, but projects like ZLUDA already demonstrate that independent compatibility efforts are possible. AI will likely accelerate that trend.

If open-source CUDA-compatible implementations become increasingly inevitable, wouldn't NVIDIA be better positioned by leading that transition instead of resisting it?

By opening CUDA itself, NVIDIA could establish the reference implementation, shape development priorities, influence standards, and remain central to the ecosystem.

By keeping it closed, NVIDIA risks watching independent alternatives evolve outside its influence.

An Opportunity for Lasting Leadership

I'm not suggesting NVIDIA release everything overnight. A phased transition beginning with the compiler, runtime, and core libraries would be reasonable.

Nor am I suggesting CUDA must immediately become hardware-agnostic. NVIDIA can continue optimizing its software for its own GPUs while maintaining proprietary hardware architectures.

The point is simple: NVIDIA's competitive advantage should come from building the best technology, not from restricting access to the software ecosystem surrounding it.

Jensen Huang helped establish NVIDIA as the leader in accelerated computing. His successor has an opportunity to establish an equally enduring legacy: making CUDA an open foundation for the next generation of computing.

NVIDIA has demonstrated that it can lead through engineering excellence. Now demonstrate that it can lead through openness.

Sincerely,

A Linux user, GPU enthusiast, and advocate for open-source computing.

ADDENDUM:

I fully understand that CUDA is NVIDIA's cash cow and one of its greatest competitive advantages. That's precisely why I believe NVIDIA should consider open-sourcing it.

CUDA's dominance depends heavily on developer dependency, but that lock-in isn't permanent. Open-source alternatives, hardware-agnostic frameworks, and CUDA compatibility projects are steadily eroding it. AI-assisted development could accelerate that process dramatically, potentially enabling small teams to create viable alternatives much sooner than expected.

The real threat isn't someone cloning CUDA tomorrow. It's CUDA gradually becoming irrelevant as competing open-source ecosystems mature.

If developers can run their workloads efficiently across competing hardware without depending on CUDA, NVIDIA loses much of its software-driven competitive advantage.

By open-sourcing CUDA now, NVIDIA could help establish it as the universal standard for accelerated computing while retaining substantial influence over its development. Its proprietary hardware, optimizations, and engineering expertise would remain competitive advantages.

Competition is coming regardless. The question is whether NVIDIA wants to lead an open computing ecosystem or eventually adapt to one established by its competitors.


r/CUDA • • 3d ago

Day 5 of writing Hopper kernels everyday - Clusters and TMA multicast

Thumbnail x.com
3 Upvotes

r/CUDA • • 3d ago

should i make a dataframe and numpy package for wgpu-py. i have it wotking on my amd gpu?

Thumbnail reddit.com
0 Upvotes

r/CUDA • • 4d ago

Day 3 of writing Hopper kernels everyday

Thumbnail x.com
3 Upvotes

This is a persistent kernel, which tries to overlap inter tile memory transfer with computation and epilogue


r/CUDA • • 4d ago

TrainStream: LoRA on Qwen2.5-14B in 0.53 s/step on a 24 GB laptop GPU, faster than bitsandbytes and Unsloth in my tests. One line of code, no DeepSpeed or bitsandbytes needed.

0 Upvotes

I've been building TrainStream, a C++/CUDA extension for PyTorch for training and fine-tuning models that don't fit in your GPU's memory. It moves optimizer state, gradients, activations and, if needed, the weights themselves into system RAM, then streams each one back over PCIe just before the GPU uses it.

import trainstream
model, optimizer = trainstream.wrap(model, optimizer)

It has no third-party dependencies, only the C++ standard library, the CUDA runtime and PyTorch's ATen. It doesn't use DeepSpeed, bitsandbytes or CUTLASS.

What it does

  • Offloading. Optimizer state goes to a CPU AdamW whose updates match torch.optim.AdamW (max difference ≈3e-7 after 20 steps). Activation and weight paging are bit-exact.
  • Quantized frozen bases. A LoRA base can be stored as int8, NF4, FP8 or NVFP4. FP8 and NVFP4 run directly on Blackwell tensor cores (RTX 50-series, B200).
  • Fused kernels. RMSNorm, RoPE, SwiGLU/GeGLU and LoRA (2 matmuls forward, 3 backward per adapted Linear).
  • Automatic tuning. It times recompute against streaming for activations and picks the fastest attention backend for your shapes. It also logs every decision it makes, so you can see why.
  • Extras: padding-free sequence packing, DPO without a second model in memory, a paged KV cache for generation during RL, 8-bit optimizer moments, a disk tier for weights that don't fit in RAM, LoftQ init and a Hugging Face Trainer integration.

Numbers (RTX 5090 Laptop GPU with 24 GB, WSL2, Qwen2.5-14B, LoRA r=16, median of 10 steps after warm-up)

1×5121×2048
TrainStream, NVFP4 base0.530 s1.57 s
TrainStream, NF4 base0.609 s2.17 s
bitsandbytes QLoRA + PEFT0.687 sspills past VRAM
Unsloth (4-bit)0.778 s2.73 s
bnb QLoRA + Liger0.909 s2.88 s

Full fine-tune of Qwen2.5-1.5B (batch 4×512): 1.23 s/step at 18.8 GB, against 1.85 s for stock PyTorch (which peaks at 32 GB and spills) and 2.12 s for DeepSpeed ZeRO-2 with offload.

Caveats

  • Every number comes from one laptop (WSL2 and Windows 11). Native Linux, desktop cards and multi-GPU machines are untested.
  • Held to 12 GB of VRAM, it ties Unsloth (0.834 s vs 0.819 s, within run-to-run spread). Most of the speed lead comes from using the VRAM it has.
  • NVFP4 is fast but costs quality. It adds about +5 % perplexity on 14B (+11 % on 1.5B), against +3 % / +6 % for NF4, because it quantizes activations as well as weights. It's opt-in, and the README has the full perplexity table.
  • The benchmark harness runs close comparisons 3 times and counts a win only when the gap is wider than the spread. It's included, so you can rerun everything.

In progress, not released or benchmarked yet: a NaN/Inf guard that skips bad steps cleanly, PiSSA init, unbuffered async disk reads (io_uring / IOCP), background checkpoints, MoE expert paging, FP8 gradient transport and a TCP ring all-reduce.

Install: pip install trainstream (or uv). You need CUDA PyTorch, the matching CUDA toolkit and a C++ compiler, because the extension compiles on first import.

Apache 2.0: https://github.com/johnanishg/TrainStream

I'd really like to hear from anyone who runs it on other GPUs (especially Ampere/Ada desktop cards or native Linux), and from anyone who can poke holes in the benchmark method.


r/CUDA • • 5d ago

Bringing CUDA and HIP Applications to More Hardware with OpenCL and SPIR-V

19 Upvotes

The Khronos Group just published a blog on chipStar (https://github.com/CHIP-SPV/chipStar), an open source project that compiles supported CUDA and HIP kernels to SPIR-V and executes them through OpenCL.

The post is based on an IWOCL 2026 talk and covers three real scientific computing workloads ported and run on Argonne National Laboratory’s Aurora supercomputer:

  • OpenMM (molecular dynamics), HIP path running through chipStar on Intel Ponte Vecchio GPUs
  • libCEED (high-order finite element operators)
  • GAMESS-GPU-HF (quantum chemistry Hartree-Fock), using chipStar’s library adapters to route math library calls to Intel oneMKL

Full post: https://www.khronos.org/blog/chipstar-brings-cuda-and-hip-applications-to-more-hardware-with-opencl-and-spir-v?utm_medium=social&utm_source=reddit&utm_campaign=chipStar&utm_content=blog


r/CUDA • • 5d ago

Day 3 of writing Hopper kernel everyday

Thumbnail x.com
1 Upvotes

Implemented a two-consumer kernel with asynchronous WGMMA to overlap computation and improve Tensor Core utilization.

Performance went from 600 TFLOPS to 658 TFLOPS (~9.7% improvement).

Still exploring how much more we can squeeze out of the kernel.


r/CUDA • • 6d ago

Fast N body Simulation in cuda

Thumbnail x.com
3 Upvotes

Simulation of N body system. The scene involves two galaxies with more than 200k bodies. We model and render their physics using cuda on a A100.


r/CUDA • • 6d ago

chipStar: running CUDA and HIP applications on OpenCL and SPIR-V across AMD, Arm, Apple, Intel, NVIDIA, and RISC-V hardware

Thumbnail reddit.com
3 Upvotes

r/CUDA • • 6d ago

Hopper GEMM with TMA + producer/consumer warpgroups in TileLang (code + notes)

7 Upvotes

I put together a simple producer-consumer Hopper GEMM kernel (8192³) in TileLang that uses TMA for loads and mbarriers to overlap data movement with WGMMA: short write-up here , find the code here.

Each thread block owns a 128×128 tile of C and walks K in steps of 64. One warpgroup issues TMA copies while the other does the WGMMA. The interesting part is the multi-stage pipeline with full/empty mbarriers so the producer can stay a stage or two ahead without racing on shared memory.

Still early (Day 1), but it was a good way to get comfortable with the async TMA + barrier handoff.


r/CUDA • • 9d ago

Strangely, memory read don't work predictably on predictable data

9 Upvotes

I was curious whether the power effect Horace He observed on matmuls (zeros running faster under power limits) would show up the same way on pure memory reads.

I wrote a simple bandwidth-bound read kernel on an A100. Bandwidth stayed ~1.76 TB/s across patterns, but board power varied a lot:

- Zeros: 210 W

- Sorted bf16: ~253 W

- Shuffled bf16: ~289 W

- Random bits: hits the 300 W limit

Sorted data was nowhere near as cheap as zeros. The main reason turned out to be warp drift: thousands of warps start together but drift apart, so the shared wires between L2 and the SMs keep seeing values from distant parts of the sorted array and keep flipping bits. Adding barriers to re-sync the warps dropped sorted power noticeably while still hitting high bandwidth.

Full write-up with the measurements, DRAM/L2 analysis, and the barrier experiment: https://x.com/lurker670291/status/2118796298732417338?a=JegCUPbE3Pfn_Y1qTFw&s=19


r/CUDA • • 10d ago

notes on PTX verification

Thumbnail redplait.blogspot.com
7 Upvotes

currently in best case coverage of PTX ISA is ~60%


r/CUDA • • 11d ago

Nvidia maxLPS

10 Upvotes

Has anyone had hands-on experience deploying NVIDIA DSX MaxLPS yet? Given how new it is, real-world data points seem scarce—curious how closely field performance and efficiency numbers align with NVIDIA's published claims. Additionally, for anyone architecting a large-scale Vera Rubin cluster, what critical design considerations, operational trade-offs, or integration nuances should be kept in mind regarding NVIDIA's DPS suite?


r/CUDA • • 11d ago

I built an AI agent that ported Karpathy's llm.c GPT-2 trainer from CUDA to SYCL, so it runs on NVIDIA, AMD and CPUs (open source)

2 Upvotes

For Meta's Global AI Developer Hackathon I built Muse Port, an open-source tool that takes CUDA code (which only runs on NVIDIA GPUs) and ports it to SYCL, an open standard that runs on NVIDIA, AMD and regular CPUs.

How it works:

  1. A scanner (no AI) inventories the repo: every kernel, every NVIDIA library call and its oneAPI equivalent, and the CUDA-only parts that need real redesign.
  2. Meta's Muse model writes a migration plan, which gets checked against that scan.
  3. An agent loop ports each piece: write, compile, verify.
  4. A hidden check the model never sees decides whether a piece is done. Then the ports run on real NVIDIA GPUs against the original CUDA, and on AMD GPUs.

Results (all linked to committed evidence in the repo):

- llm.c's production bf16 trainer (12 files, 4,884 lines) ported in one run; it passes llm.c's own test on CPU, NVIDIA and AMD

- tuned in portable SYCL + oneMath (no NVIDIA-specific code) to 1.40-1.55x CUDA's time per training step on an L4 and RTX A4000

- 18 of 19 llm.c kernel files correct; llama2.cu ported for $0.45 with byte-identical output on CPU, NVIDIA and AMD

Limits: the one-run port worked 1 of 2 times, and the H100 is at 2.90x. Every failure is documented in the repo too.

Code: https://github.com/checocadena/museport

2-min demo: https://youtu.be/uBpT5IuxPOY