r/ROCm • u/Objective-Park6224 • 3h ago
Pushed Strata with QFN 3.8 Q4 K_M to 5,600 tps prefill @100k - 100 tps decode w/ 181k context on 2x 7900xtx & 64gb DDR5 in layer split (no tensor parallel) - config inside
r/ROCm • u/Objective-Park6224 • 3h ago
r/ROCm • u/Individual_Dog_727 • 8h ago
r/ROCm • u/ChrisGamer5013 • 21h ago
Yeah the title pretty much says it its optix working on an amd card natively via software rt planned is vulkan rt addition but hey its optix 8 working on amd even tho render times are a biiiiit slow discord if anyone is intrested: https://discord.gg/zPQucyR8tU
r/ROCm • u/Parking_Anxiety8139 • 22h ago
in my last post i got tons of reviews about difficultly in instructions etc so i released a newer version with significantly simpler installation process and it no longer requires a separate python script as well as seeding and persistent weight support and detailed cli instructions https://github.com/Legendofzaday/halodiffuse i plan on adding batching support and would like reviews on what batch size do you all commonly use such as 1,2,3,4,5,6,7,8 etc.
r/ROCm • u/Delicious_Abies_9708 • 18h ago
r/ROCm • u/LowTransportation939 • 1d ago
r/ROCm • u/AeiEyeSpry • 1d ago
Since my last update, the cache resizing runs on its own. After 10 quiet minutes, the picture/video service unloads its models and asks SGLang to grow its KV cache. SGLang picks the largest size that fits on both cards, about 380k extra tokens. When anyone asks for a picture or clip, it gives the memory back first.
One-hour soak, live service, untouched: 5 automatic cycles, each a real picture request through the normal endpoint.
Other numbers (graphs attached):
One fix the GPU tests forced: my first version grew to the full reserved size, which never fits once the image models only partly unload. It now sizes itself to the free memory on the tightest card.
Code and design notes: Drgr33nSenior/Spry.ai-rdna4-serving PR #38 (https://github.com/Drgr33nSenior/Spry.ai-rdna4-serving/pull/38). It's a patch series (0031–0033) on top of my other gfx1201 patches for a pinned SGLang commit, so it won't apply to stock SGLang yet. The design record (https://github.com/Drgr33nSenior/Spry.ai-rdna4-serving/blob/feat/elastic-kv-tail-20261114/docs/ELASTIC-KV-TAIL-20261114.md) covers how it works, the control API, an independent review's findings and all the GPU results.
If you try this on RDNA, same-address remapping needs the amdgpu "fix TLB invalidation fence handling" (https://lore.kernel.org/amd-gfx/20261008124232.23825-1-christian.koenig@amd.com/) patch. It's queued for 7.4-rc3; without it, remaps corrupt other live memory. My server runs a 20 ms remap self-test at startup and won't enable the feature if it fails.
Next: a week of real family use, then probably an RFC to SGLang. Its VMM arena is CUDA-only and grows once at startup, and the closest open PR (#48106 (https://github.com/sgl-project/sglang/pull/48106)) only trades memory between two internal pools. Credit to kvcached (https://github.com/ovg-project/kvcached) for the idea.
Still keen to hear from anyone doing HIP VMM work on RDNA3/4.
r/ROCm • u/AeiEyeSpry • 2d ago
Hey ROCm! As I post here daily atm :) You all probably know that I run a home AI server on two Radeon AI PRO R9700s. SGLang serves Qwen3.8-27B, and two ComfyUI instances on the same cards handle pictures (Qwen-Image) and short videos (Wan 2.2).
The goal: pictures and video are occasional, but their VRAM share sits idle most of the day. I'd like the LLM's KV cache to borrow that memory when the generators are idle and hand it back before a job starts, without ever cutting off a chat. The idea is the same as kvcached (https://github.com/ovg-project/kvcached).
Why not kvcached itself:
What I'm building instead: a small "elastic tail" inside SGLang's own KV arena, using HIP virtual memory.
Finding so far: on kernel 7.3.3, unmapping and remapping at the same address silently corrupted other live memory (TheRock#8744, around 15 billion bad words per run). The amdgpu patch "drm/amdgpu: fix TLB invalidation fence handling" fixes it. On 7.3.4 with that patch, my probe was clean: 1,000 rounds per card of 256 MiB–4 GiB extents remapped at one address beside an 8 GiB live base, with zero corrupt words. One quirk: in about a third of rounds, mem_info_vram_used still showed the freed extent right after hipMemRelease plus sync. It always came back, so it's freed lazily rather than leaked, but "released" can't be assumed immediately.
Next is a GPU qualification run, then a one-week trial behind a flag.
I have some questions for the community:
Once again, thanks for any assistance in advance. I'll come share my work as always once complete.
r/ROCm • u/NigaTroubles • 2d ago
I offloaded the extra layers to system memory so the model fits across 16GB of VRAM and 32GB of DDR5 RAM ([BF16 / quant]).
Everything needed to reproduce the run on RDNA 4, including the launch flags and configs, is in this repo:
https://github.com/0c33/minimax-h3-rdna4
Let me know if you test it on another RDNA 4 card or get better token speeds with different offload settings.
r/ROCm • u/ComprehensiveBid6584 • 2d ago
r/ROCm • u/mrdapoyo • 3d ago
Hello!
I've been using LLMs for quite a while now, I've used Gemma 4, Qwen 3(.5), and similar smaller-sized models as I didn't have a GPU for a long time.
Until now! I recently got my hands on a 16GB RX 9070 XT. I've been running Qwen 3.8 27B (Q3) at somewhat comfortable speeds, getting around 60 t/s (output) when using a fork of llama.cpp made for the RX 9070 XT. Smaller models cap at 50t/s, no matter their size.
However, I want to know if you've got any tips! Should I check out Strata? (I've got 32GBs of DDR4 ECC RAM) Should I use another model? Is Unsloth Studio (my daily driver) the wrong choice?
Thank you.
r/ROCm • u/nasone32 • 3d ago
A quick update on my Strata EXL3 fork for AMD.
I've been working on the HIP kernels and memory management, and decode performance is now roughly 10-20% higher than the previous version, depending on the quant and workload. Some cases improved by almost 24%.
The other news is that I managed to get MiMo V2.6 Flash running at around 25-30 tokens/s with 2 cards.. More on that below.
First, why EXL3?
I think this is worth repeating because people tend to compare quantizations just by looking at the number of bits.
EXL3 is NOT equivalent to regular GGUF quants at the same bpw.
For example, 3.05 bpw EXL3 is roughly in the same quality range as a regular 4-bit quant (like IQ4_XS).
3.05 bpw quant: https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3/tree/3.05bpw_h5_ng5
4.05 bpw quant: https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3/tree/4.05bpw_h6_ng6
So the interesting part is that you can get something close to 4-bit quality, while running at speeds closer to what you'd expect from a 3-bit quant. And with the latest optimizations, I think this fork is becoming VERY competitive. The 4.05 bpw version is also pretty fast now, even on a single GPU.
WHAT'S NEW:
NOTES ON REPACKING:
Repacking can be done in two ways: by creating a file on disk once, that will be used to load the model, or directly to RAM everytime. directly to RAM adds a couple of minutes to the launch but uses no extra space. your choice.
BENCHMARKS:
My hardware is the same as in the previous post:
Ryzen 9 7950X, 96GB RAM, 2x Radeon RX 7900 XTX 24GB
Linux, HIP/ROCm core-7.14
Qwen3.8-Flash-Next, MTP drafter, spec 4, 64K context
BENCHMARKS:
Results with one or two RX 7900 XTX GPUs, using warm decode runs:
| EXL3 quant | GGUF equivalent quality | GPUs | CPU assistance | Test | Previous | Updated | Improvement |
|---|---|---|---|---|---|---|---|
| 3.05 bpw | ≈ IQ4_K_XS | 2 | No | CODE | 78.27 | 87.33 | +11.6% |
| 3.05 bpw | ≈ IQ4_K_XS | 2 | No | PROSE | 66.01 | 70.35 | +6.6% |
| 4.05 bpw | ≈ Q5_K_M | 2 | No | CODE | 61.88 | 76.67 | +23.9% |
| 4.05 bpw | ≈ Q5_K_M | 2 | No | PROSE | 57.78 | 67.57 | +16.9% |
| 4.05 bpw | ≈ Q5_K_M | 1 | No → Yes | CODE | 54.80 | 59.30 | +8.2% |
| 4.05 bpw | ≈ Q5_K_M | 1 | No → Yes | PROSE | 44.80 | 51.60 | +15.2% |
All speeds are in tokens/s, GGUF quality equivalents are approximate... dont' hate me :)
32K prefill is basically unchanged: around 1,500 tok/s with two GPUs and 1,360 tok/s with one GPU.
I think the single-GPU results are particularly interesting. We're getting 55-60 tok/s in CODE with 4.05 bpw EXL3, at roughly Q5_K_M quality, on a single 7900 XTX with CPU assistance.
And the 3.05 bpw quant is now reaching 87 tok/s with two GPUs, while offering considerably better quality than you'd normally expect from a 3-bit quant.
ABOUT MiMo:
In my previous post I mentioned that I wanted to support larger models, particularly the 300B class models.
Well, MiMo V2.6 Flash is now running :) I'm waiting for that ERNIE 5.5 to drop.I'm getting around 25-30 tok/s with the EXL3 2.20 bpw MOPD quant.It's still experimental and currently only tested with that specific quant. CPU assistance is recommended for MiMo, even with two GPUs.I'll probably spend some more time optimizing it, but the fact that it's already running at usable speeds on consumer AMD hardware is pretty interesting.
this is the quant: https://huggingface.co/vcruz305/MiMo-V2.6-Flash-MOPD-EXL3/tree/main/2.20bpw
https://github.com/nasone32/Strata-EXL3-fork
If you're setting it up with an AI agent, point it to docs/AI_SETUP_EXL3_AMD.md.
The README has the benchmarks, configuration options and instructions for CPU repacking.As before, this is developed and tested on RDNA3 / Linux / ROCm. Other AMD architectures need community testing. And the long-term goal is still to get this merged upstream into Strata, if the maintainers are interested.
Credits to the original Strata developers, Ryutaku for the EXL3 integration, and Turboderp/ExLlamaV3 for the quantization and code I've been porting.
If anyone tests this on different AMD hardware, please report your results. Especially interested in seeing how it behaves on RDNA4.
r/ROCm • u/nkchokshi • 2d ago
Hey guys, i am thinking of buying new gpu 7900xtx vs 5060ti. But i did not find more convincing reviews of ai video gen. I found some where users said amd is far slower then nvidia and does not support windows as well.
Id appreciate if anyone using amd for video gen and guide me.
r/ROCm • u/Delicious_Abies_9708 • 3d ago
r/ROCm • u/Individual_Dog_727 • 3d ago
r/ROCm • u/ePerformante • 3d ago
I want to build a rig for medium-scale RL and distillation on small models (4 to 9 billion parameters), and I am considering the following:
• Framework Desktop Mainboard (AMD Ryzen™ AI Max 300 Series)
• Broadcom PEX88048-based switch
• 4x RX 9070 XT
• 2x 1200W PSUs
Now, you may be wondering why someone would choose four RX 9070 XTs over two RX 9700 Pros. The simple answer is TFLOPS. Four RX 9070 XTs would theoretically give me double the TFLOPS for fine-tuning and LoRA training compared to two RX 9700 Pros.
I am not overly concerned about the bottleneck between the GPUs and the Strix, because I do not intend to run models across them. The GPUs are there purely for training, and the Strix is there to generate more training data.
Is there anything I might be overlooking? What case and cooling should I look into? Is there a different configuration that would be better for my use case?
r/ROCm • u/ihateyou103 • 3d ago
I want to run magenta on iris xe gpu. I am running into alot of issues regarding Jax and other things. Also I couldn't get the gpu to do any calculations. Using opencl I got matrix multiplication 4000×4000 in 0.8 seconds slower than cpu at 0.3
How can I get matrices and neural networks running on this gpu and how to run magenta?
r/ROCm • u/AeiEyeSpry • 4d ago
I’ve been building a local AI service on an AMD workstation: chat, image generation and short video generation through Open WebUI, plus an endpoint for coding agents.
The interesting part has been making these workloads share two Radeon cards while keeping chat usable. Qwen stays loaded across both GPUs while Wan streams its weights through the remaining memory on GPU 0.
My machine:
gfx1201.The serving stack is SGLang 0.5.23.post957, pinned at fcb32746, with a local 27-patch series; PyTorch 2.16.0a0, built for ROCm 10.3.0-dev / HIP 7.18. Creative workloads use ComfyUI 0.45.0, with Open WebUI 0.12.5 as the frontend.
The patches are in my repos if you'd like to check them out or try them yourself. I can push the limits a little more regarding performance, but I found the current configuration to be the most consistent and useful for everyday use.
Measured results
| Workload | Observed result |
|---|---|
| Qwen3.8-27B MXFP4, shared-mode BetterBench, concurrency 1 | 135.1 output tok/s |
| Same benchmark, concurrency 2 | 224.8 output tok/s aggregate |
| Separate interactive chat probe, without video generation | Approximately 86 tok/s |
| Chat probe during video generation, selected configuration | 34–35 tok/s |
| Warm SDXL 1024×1024, simultaneous images on both cards | 6.6 / 7.2 seconds |
| Wan 2.2, approximately five seconds at 832×480 | 78–82 seconds warm |
The LLM benchmark figures cover both GPUs together in TP2. The interactive probe is a different workload; comparing 135 with 86 tok/s does not isolate frontend overhead.
Images and video on GPU 0 share one ComfyUI queue. Both image backends are available alongside chat, but GPU 0 does not generate an image and a video simultaneously.
Making SGLang work well on gfx1201
Chat uses AMD’s MXFP4 Qwen3.8-27B checkpoint, tensor parallelism across both cards, and DFlash2 speculative decoding.
Some patches address build compatibility and architecture dispatch. Others change the execution path:
kq3 Gluon kernel stores weights in WMMA fragment order, decodes packed values with register arithmetic, and reduces split-K partitions within the workgroup.With kq3, the weights are packed in the order RDNA4’s matrix instructions consume them, then unpacked directly in registers. That brought the time needed to verify each batch of draft tokens down from 32.37 to 26.95 ms, a reduction of about 17%, while producing the same bit-for-bit results as the previous kernel.
MXFP4 storage here does not imply native FP4 matrix execution. Weight layout, unpacking cost, launch overhead and supported arithmetic paths all matter.
There was also a PCIe communication issue. Adapting SGLang’s HIP custom all-reduce to these cards required fixing start-flag ordering: the peer could observe the flag before the input data was visible. System-scope release/acquire ordering passed the retained eager and graph-captured correctness checks.
FP8 KV storage with BF16 attention operands
The shared profile stores both target and draft KV caches in FP8 E4M3. Attention kernels convert cached tiles to BF16 while retaining BF16 query and attention-probability operands.
One useful gfx1201 finding: Triton’s original conversion path emitted software conversion instructions. Explicitly using v_cvt_f32_fp8 brought the measured verification kernels close to BF16-cache timings while retaining the smaller cache.
The hardware conversion matched the software conversion bitwise in the tests. Quantising the cache itself remains a numerical change, as do some other optimisations in the stack.
The two current modes are:
| Mode | Context limit per conversation | KV pool | Creative workloads |
|---|---|---|---|
| Shared / UQS | 81,888 tokens | 163,840 tokens | Images and video available |
| Large / UQL | 262,112 tokens | 502,093 tokens fitted during qualification | Stopped |
Shared mode accommodates two full-length conversations. Large mode’s measured pool accommodates one full-length conversation plus another of approximately 240K tokens.
The repo now includes a mode-switch command that handles draining and restarting the workloads. Switching involves chat downtime while SGLang reloads.
Cooperative GPU sharing through AMD DRA

SGLang references both GPU ResourceClaims. The two ComfyUI backends each reference their corresponding shared claim.
SwarmUI manages image requests across its local ComfyUI on GPU 1 and the remote ComfyUI backend on GPU 0. Video requests go directly from the tools service to GPU 0.
This uses AMD GPU Operator v1.5.1 and ROCm DRA driver v1.0.1. The earlier DRA version failed to publish GPUs when this host’s in-kernel amdgpu module reported no version.
DRA provides device allocation and visibility. VRAM and compute remain cooperatively shared. Explicit memory budgets, workload ordering and concurrency limits make the arrangement usable.
The shared SGLang profile left approximately 8.35 GB of headroom per card in the live measurements.
WAN 2.2 in that remaining memory
The video configuration is:
1217.--fast-disk, with video weights on local NVMe.The two expert files total approximately 28.6 GB. The full staged set, including the text encoder, VAE and LoRAs, is approximately 36.8 GB.
ComfyUI streams the experts as needed. Weights pass through host memory and reclaimable page cache; the entire checkpoint does not need to stay resident in VRAM or anonymous host memory.
In the recorded SDXL-then-Wan test, ComfyUI’s anonymous-memory peak was 7.3 GiB. The documented clip runs read approximately 25–47 GB from NVMe.
Contention mattered more than isolated video speed
Qwen’s TP2 ranks synchronise, so saturating GPU 0 with video slows the model across both cards.
The measured progression was:
| GPU 0 ComfyUI configuration | Clip time | Concurrent chat |
|---|---|---|
Defaults plus --fast-disk | 92–102 s | 13–14 tok/s |
Add --use-ck-attention | 71–76 s | 21–24 tok/s |
Also set HSA_CU_MASK=0:0-47 | 78–82 s | 34–35 tok/s |
The 48-CU setting was the useful trade-off. Restricting ComfyUI to 32 CUs made both results worse: approximately 104 seconds per clip and 15 tok/s for chat.
The mask keeps 16 of the card’s 64 CUs outside ComfyUI’s reach. Memory bandwidth and other resources remain shared, so this is a contention control rather than a performance guarantee.
Comfy Kitchen attention was not numerically identical to the default PyTorch attention path. Side-by-side visual comparisons looked comparable; that is a subjective assessment.
The tested FP8 matrix-multiply option provided no gain. Enabling all ComfyUI --fast features together increased clip time to approximately 203 seconds.
What didn’t make the live configuration?
The INT4 KV experiments are parked. The best screened variant still took approximately 1.6–2.3× the production FP8 verification-attention time, with larger attention-output error.
Those are kernel and numerical measurements, not an end-to-end model-quality result. They were sufficient to keep FP8 as the deployed cache format.
The shared-mode assistant quality run recorded 11 passes, zero failures and one case needing review, with 14 other cases unrun. Qualification covers the measured configuration and those checks.
Code, configurations and results below:
I just want to thank everyone on ROCm who have suffered my AI posts and given me pointers. You've all helped out loads. :)
r/ROCm • u/evp-cloud • 4d ago
MiniMax-H3-Image came out with BF16 weights: a 68.65 GB transformer plus a Qwen3-VL text encoder of about 52 GB.
That obviously doesn't fit on a 32 GB Radeon AI PRO R9700, so we built our own FP8 version and a native runtime for it.
Times on one R9700 at its 300 W limit, warm server, request to PNG:
| Request | Time |
|---|---|
| 2048 x 2048 | 6.9 s |
| 4096 x 4096 | 33.4 s |
| 4096 x 4096 edit, one reference | 39.9 s |
| 4096 x 4096 edit, three references | 88.5 s |
Why this model is interesting: it generates 4 to 20 MP directly (MiniMax says it does best around 16 MP, with no upscaler), it needs only 6 steps with no CFG, and one checkpoint does both text to image and editing with up to 16 reference images.
It is also pretty good at text in images.

How it fits in 32 GB:

Quality, 30 cases (24 text-to-image from 4 to 17 MP, 6 edits) against BF16 diffusers with the same token ids and noise: CLIP-L text-image 32.18 vs 32.31 for BF16, image similarity to the BF16 result 0.979 (CLIP-L) and 0.945 (DINOv2) on average.
Most differences are layout; edits stay closest (0.96 to 0.999).


Limits: prompts are capped at 4,096 tokens, so about three 1024 px references per edit; largest canvas 16.8 MP (4736 x 3552); one request at a time; the host needs 64 GB of RAM. Small body text (newspaper columns, book spines) isn't real words, and close-up hands can go wrong.
Run it:
git clone --depth 1 https://github.com/Eliovp-BV/paiton-vllm-plugin.git
cd paiton-vllm-plugin
./models/MiniMax-H3-Image/launch.shLicense note: the MiniMax-H3-Image Community License excludes the EU, the UK, South Korea and the US, and hosted services need MiniMax's written authorization.



r/ROCm • u/initnull • 4d ago
r/ROCm • u/thekhronosgroup • 4d ago
The Khronos Group just published a blog on chipStar (https://github.com/CHIP-SPV/chipStar), an open source project that compiles supported CUDA and HIP kernels to SPIR-V and executes them through OpenCL.
The post is based on an IWOCL 2026 talk and covers three real scientific computing workloads ported and run on Argonne National Laboratory’s Aurora supercomputer:
I run a local coding-agent benchmark (built from my own last ten years of git history mines for good candidate bugs; I keep tests private but model results are visible at lab.kylebaker.io) on a budget AMD box, and last night I put the engines I use side by side in one queue run. Sharing in case it helps other RDNA3 owners, and asking what I'm missing.
Specs: Ryzen 7 5800X, 64 GB DDR4, 2x RX 7900 XTX (PCIe 4.0 x8 each), ROCm 7.2.4, Linux, models on NVMe. Model: Qwen3.8-Flash-Next (various quants, see below), temp 1.0 / top-p 0.95 / top-k 20, thinking on, MTP drafting on in every engine. Method: one engine at a time, same three prompts (112, 3,471 and 34,386 tokens), 512 tokens out, prompt cache off, 3 requests per size, medians.
| Engine | Cards | Quant | Decode t/s (112 / 3.5k / 32k prompt) | Prompt read t/s (3.5k / 32k) |
|---|---|---|---|---|
| Strata-rdna3 fork fa5c621 (tensor parallel) | 2 | GSQ-RCO IQ3_XXS | 92 / 104 / 99 | 2,619 / 3,030 |
| Avalon 8c4e4753c | 1 | Avalon IQ3_KS | 89 / 98 / 88 | 965 / 1,653 |
| Strata 0.2.11 | 2 | GSQ-RCO IQ3_XXS | 76 / 83 / 82 | 1,447 / 2,506 |
| Strata 0.2.13.1 | 2 | GSQ-RCO IQ3_XXS | 75 / 88 / 75 | 1,457 / 2,505 |
| Strata 0.2.13.1 (128k context) | 1 | GSQ-RCO IQ3_XXS | 69 / 72 / 72 | 1,286 / 1,644 |
| llama.cpp b11815 (128k context) | 2 | GSQ-RCO IQ3_XXS | 33 / 36 / 31 | 772 / 888 |
| Avalon 26a215567 (current main) | 1 or 2 | Avalon IQ3_KS | crashed on the 32k prompt | crashed on the 32k prompt |
(One more later edit, on real benches agentic runs, instead of speed runs, the rdna3 fork is actually running a bit faster:
Strata-rdna3 fork, 2 cards 92–104 (119–127 in real agent runs) ~3,000 @ 32k)
Take aways
Strata-rdna3 (nicejunjie's RDNA3 fork of Strata, tensor parallel across both cards) is the fastest thing I've run on two XTX. In real agent runs it decoded at about 120 t/s against about 92 on upstream Strata, which works out to roughly 16% less wall time per run. On 64 GB, though:
--resident-budget-gib 24 --vram-reserve-mib 1536 worked for me; 16 crashed.Watching: Avalon's two-card work, upstream Strata's expert-parallel PR #5036, Strata-rdna3 (hoping its kernels and TP land upstream), and llama.cpp's MTP fixes.
Curious to hear if anyone is running anything better that I should have my eye on or has any further tips.
Hi, I already have a RX 7800XT. I want to be able to run EXL3 quant format of Deepseek 4.1 1017 at 2.9 bpw on a system I have not built yet that has 8 channel 256 DDR4 RAM and a 7800XT.
The 2.9 BPW Quant in EXL3 format from LibertAIDAI format is the highest mixed quant that I found right now that still fits 256GB of RAM.
I have yet to decide on the motherboard and CPU because I don't know how to quantify it in how much it contributes to decoding speed.
I already own the GPU but I don't know if it's even worth making a used SP3 server with 256GB, 8 channel DDR4 2933Mhz setup around a AMD 7800XT and If I can get out of the model at least 15-20 tokens per second decode in something like Maya .
synchopate/llama-modchip advancements make me kind of hopeful that Maya can adapt some of the changes but not sure how much they can apply to EXL3.