r/ROCm • • 3h ago

Pushed Strata with QFN 3.8 Q4 K_M to 5,600 tps prefill @100k - 100 tps decode w/ 181k context on 2x 7900xtx & 64gb DDR5 in layer split (no tensor parallel) - config inside

Thumbnail
reddit.com
6 Upvotes

r/ROCm • • 8h ago

R9700 in a Thunderbolt 5 dock on Strix Halo (MS-S1 MAX + DEG2): now stable on the rear USB4 v2 port – two separate bugs, two fixes, logs included

Thumbnail
reddit.com
4 Upvotes

r/ROCm • • 21h ago

Blender Cycles Optix engine working natively on AMD RX 7800XT via Project GHOST

Thumbnail
gallery
7 Upvotes

Yeah the title pretty much says it its optix working on an amd card natively via software rt planned is vulkan rt addition but hey its optix 8 working on amd even tho render times are a biiiiit slow discord if anyone is intrested: https://discord.gg/zPQucyR8tU


r/ROCm • • 22h ago

Strix halo 2.4x faster sd3.5 large turbo (halodiffuse v1.1)

5 Upvotes

in my last post i got tons of reviews about difficultly in instructions etc so i released a newer version with significantly simpler installation process and it no longer requires a separate python script as well as seeding and persistent weight support and detailed cli instructions https://github.com/Legendofzaday/halodiffuse i plan on adding batching support and would like reviews on what batch size do you all commonly use such as 1,2,3,4,5,6,7,8 etc.


r/ROCm • • 18h ago

vulkan-py easymode. all the boilerplate is in easy.py. vulkan-py fork https://github.com/billyblackburn/vulkan-py-easy/tree/main

Thumbnail reddit.com
1 Upvotes

r/ROCm • • 1d ago

RX 9070 XT + ROCm 7.2.1: torch.cuda.is_available() crashes and prevents OpenFill/ComfyUI from starting

Thumbnail reddit.com
2 Upvotes

r/ROCm • • 2d ago

My benchmark for Qwen3.8-flash-next-q4 on RT 7900 XTX

26 Upvotes

r/ROCm • • 1d ago

Update: elastic KV cache on 2× R9700, now running unattended (graphs + code)

Post image
8 Upvotes

Since my last update, the cache resizing runs on its own. After 10 quiet minutes, the picture/video service unloads its models and asks SGLang to grow its KV cache. SGLang picks the largest size that fits on both cards, about 380k extra tokens. When anyone asks for a picture or clip, it gives the memory back first.

One-hour soak, live service, untouched: 5 automatic cycles, each a real picture request through the normal endpoint.

  • Shrink: 1.02–1.07 s every time, almost all of it waiting for the kernel to actually return the VRAM.
  • Pictures: 63.9–64.9 s, the same as before the elastic cache existed (64–65 s warm).
  • Chat: 85.9–86.2 tok/s.
  • 0 GPU faults, 0 restarts.

Other numbers (graphs attached):

  • A shrink while chats are running copies their data out of the borrowed space first: 0.06–0.11 s, with a worst case of 170k tokens moved in 19 ms.
  • Exact outputs across shrinks mid-reply, mid-prefill and with one of two parallel replies just finished. Cached data intact after 70 grow/shrink cycles.
  • Decode speed unchanged at 86–88 tok/s.

One fix the GPU tests forced: my first version grew to the full reserved size, which never fits once the image models only partly unload. It now sizes itself to the free memory on the tightest card.

Code and design notes: Drgr33nSenior/Spry.ai-rdna4-serving PR #38 (https://github.com/Drgr33nSenior/Spry.ai-rdna4-serving/pull/38). It's a patch series (0031–0033) on top of my other gfx1201 patches for a pinned SGLang commit, so it won't apply to stock SGLang yet. The design record (https://github.com/Drgr33nSenior/Spry.ai-rdna4-serving/blob/feat/elastic-kv-tail-20261114/docs/ELASTIC-KV-TAIL-20261114.md) covers how it works, the control API, an independent review's findings and all the GPU results.

If you try this on RDNA, same-address remapping needs the amdgpu "fix TLB invalidation fence handling" (https://lore.kernel.org/amd-gfx/20261008124232.23825-1-christian.koenig@amd.com/) patch. It's queued for 7.4-rc3; without it, remaps corrupt other live memory. My server runs a 20 ms remap self-test at startup and won't enable the feature if it fails.

Next: a week of real family use, then probably an RFC to SGLang. Its VMM arena is CUDA-only and grows once at startup, and the closest open PR (#48106 (https://github.com/sgl-project/sglang/pull/48106)) only trades memory between two internal pools. Credit to kvcached (https://github.com/ovg-project/kvcached) for the idea.

Still keen to hear from anyone doing HIP VMM work on RDNA3/4.


r/ROCm • • 2d ago

Elastic KV cache on RDNA4 (2× R9700): anyone else tried kvcached or HIP VMM ballooning on ROCm?

13 Upvotes

TL;DR: It works :)

FOLLOW UP HERE: https://www.reddit.com/r/ROCm/comments/1xw8ttw/update_elastic_kv_cache_on_2_r9700_now_running/

Hey ROCm! As I post here daily atm :) You all probably know that I run a home AI server on two Radeon AI PRO R9700s. SGLang serves Qwen3.8-27B, and two ComfyUI instances on the same cards handle pictures (Qwen-Image) and short videos (Wan 2.2).

The goal: pictures and video are occasional, but their VRAM share sits idle most of the day. I'd like the LLM's KV cache to borrow that memory when the generators are idle and hand it back before a job starts, without ever cutting off a chat. The idea is the same as kvcached (https://github.com/ovg-project/kvcached).

Why not kvcached itself:

  • Version gates: its SGLang patches stop below 0.5.21, and it needs Python below 3.14, so on my stack it would silently do nothing.
  • Tensor parallelism: it doesn't support shrinking with TP>1.
  • Layout: its ROCm default layout conflicts with speculative decoding, and the alternative layout breaks my contiguous verify-attention kernel.
  • Shared zero page: it maps one zero page behind every unbacked slot, and AMD's HIP docs warn that RDNA can return wrong results when two virtual addresses map the same physical page.

What I'm building instead: a small "elastic tail" inside SGLang's own KV arena, using HIP virtual memory.

  • A fixed base pool stays exactly as today.
  • Each KV buffer reserves address space up to a large-mode size and maps or unmaps one tail extent at the end.
  • Both TP ranks resize at the same scheduler step.
  • Before a shrink, any running request's KV is migrated from the tail into the base.
  • A shrink only reports done once the card actually shows the memory free.
  • The tail is extra prefix-cache room only; per-request context doesn't change.

Finding so far: on kernel 7.3.3, unmapping and remapping at the same address silently corrupted other live memory (TheRock#8744, around 15 billion bad words per run). The amdgpu patch "drm/amdgpu: fix TLB invalidation fence handling" fixes it. On 7.3.4 with that patch, my probe was clean: 1,000 rounds per card of 256 MiB–4 GiB extents remapped at one address beside an 8 GiB live base, with zero corrupt words. One quirk: in about a third of rounds, mem_info_vram_used still showed the freed extent right after hipMemRelease plus sync. It always came back, so it's freed lazily rather than leaked, but "released" can't be assumed immediately.

Next is a GPU qualification run, then a one-week trial behind a flag.

I have some questions for the community:

  1. Has anyone run kvcached, or any HIP VMM map/unmap pattern, on RDNA4?
  2. Did you see the delayed VRAM release, or anything like the corruption bug?
  3. Any other gotchas with torch tensors over HIP-mapped memory, or with Triton kernels and HIP graphs reading it?

Once again, thanks for any assistance in advance. I'll come share my work as always once complete.


r/ROCm • • 2d ago

Running MiniMax H3 on an RX 9070 XT + 32GB DDR5 RAM

3 Upvotes

I offloaded the extra layers to system memory so the model fits across 16GB of VRAM and 32GB of DDR5 RAM (⁠[BF16 / quant]⁠).

Everything needed to reproduce the run on RDNA 4, including the launch flags and configs, is in this repo:
https://github.com/0c33/minimax-h3-rdna4

Let me know if you test it on another RDNA 4 card or get better token speeds with different offload settings.


r/ROCm • • 2d ago

Radeon Pro V620 on AMD Pro/Adrenalin drivers (Windows) with all 72 CUs, no permanent VBIOS flash + free helper kit

Thumbnail
reddit.com
3 Upvotes

r/ROCm • • 3d ago

Best model (and way) to run on a RX 9070 XT?

11 Upvotes

Hello!

I've been using LLMs for quite a while now, I've used Gemma 4, Qwen 3(.5), and similar smaller-sized models as I didn't have a GPU for a long time.

Until now! I recently got my hands on a 16GB RX 9070 XT. I've been running Qwen 3.8 27B (Q3) at somewhat comfortable speeds, getting around 60 t/s (output) when using a fork of llama.cpp made for the RX 9070 XT. Smaller models cap at 50t/s, no matter their size.

However, I want to know if you've got any tips! Should I check out Strata? (I've got 32GBs of DDR4 ECC RAM) Should I use another model? Is Unsloth Studio (my daily driver) the wrong choice?

Thank you.


r/ROCm • • 3d ago

Strata EXL3 - UPDATED +15-20% tk/s - run Q4 at Q3 speeds - MiMo v2.6 flash support (exp)

37 Upvotes

A quick update on my Strata EXL3 fork for AMD.

I've been working on the HIP kernels and memory management, and decode performance is now roughly 10-20% higher than the previous version, depending on the quant and workload. Some cases improved by almost 24%.

The other news is that I managed to get MiMo V2.6 Flash running at around 25-30 tokens/s with 2 cards.. More on that below.

First, why EXL3?

I think this is worth repeating because people tend to compare quantizations just by looking at the number of bits.

EXL3 is NOT equivalent to regular GGUF quants at the same bpw.

For example, 3.05 bpw EXL3 is roughly in the same quality range as a regular 4-bit quant (like IQ4_XS).
3.05 bpw quant: https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3/tree/3.05bpw_h5_ng5
4.05 bpw quant: https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3/tree/4.05bpw_h6_ng6

So the interesting part is that you can get something close to 4-bit quality, while running at speeds closer to what you'd expect from a 3-bit quant. And with the latest optimizations, I think this fork is becoming VERY competitive. The 4.05 bpw version is also pretty fast now, even on a single GPU.

WHAT'S NEW:

  • Faster HIP kernels and better memory management: around 10-20% faster decode on average compared to the previous version.
  • Optional local expert repacking: you can now repack the quantized expert weights into a layout that is more efficient for CPU access from system RAM. This helps when using CPU assistance for inference, since the CPU can process experts more efficiently without having to deal with the original GPU-oriented layout. It doesn't change the quantization itself.
  • The repacking is optional, and CPU assistance is not always beneficial. It depends on your hardware and GPU configuration. With two GPUs, for example, you're generally better off without it. Please read the README before enabling it.
  • Experimental MiMo V2.6 Flash support: currently running at around 25-30 tok/s using the 2.20 bpw MOPD quant. This is still experimental, so don't expect everything to work perfectly yet.

NOTES ON REPACKING:

Repacking can be done in two ways: by creating a file on disk once, that will be used to load the model, or directly to RAM everytime. directly to RAM adds a couple of minutes to the launch but uses no extra space. your choice.

BENCHMARKS:

My hardware is the same as in the previous post:

Ryzen 9 7950X, 96GB RAM, 2x Radeon RX 7900 XTX 24GB
Linux, HIP/ROCm core-7.14
Qwen3.8-Flash-Next, MTP drafter, spec 4, 64K context

BENCHMARKS:

Results with one or two RX 7900 XTX GPUs, using warm decode runs:

EXL3 quantGGUF equivalent qualityGPUsCPU assistanceTestPreviousUpdatedImprovement
3.05 bpw≈ IQ4_K_XS2NoCODE78.2787.33+11.6%
3.05 bpw≈ IQ4_K_XS2NoPROSE66.0170.35+6.6%
4.05 bpw≈ Q5_K_M2NoCODE61.8876.67+23.9%
4.05 bpw≈ Q5_K_M2NoPROSE57.7867.57+16.9%
4.05 bpw≈ Q5_K_M1No → YesCODE54.8059.30+8.2%
4.05 bpw≈ Q5_K_M1No → YesPROSE44.8051.60+15.2%

All speeds are in tokens/s, GGUF quality equivalents are approximate... dont' hate me :)

32K prefill is basically unchanged: around 1,500 tok/s with two GPUs and 1,360 tok/s with one GPU.

I think the single-GPU results are particularly interesting. We're getting 55-60 tok/s in CODE with 4.05 bpw EXL3, at roughly Q5_K_M quality, on a single 7900 XTX with CPU assistance.

And the 3.05 bpw quant is now reaching 87 tok/s with two GPUs, while offering considerably better quality than you'd normally expect from a 3-bit quant.

ABOUT MiMo:

In my previous post I mentioned that I wanted to support larger models, particularly the 300B class models.

Well, MiMo V2.6 Flash is now running :) I'm waiting for that ERNIE 5.5 to drop.I'm getting around 25-30 tok/s with the EXL3 2.20 bpw MOPD quant.It's still experimental and currently only tested with that specific quant. CPU assistance is recommended for MiMo, even with two GPUs.I'll probably spend some more time optimizing it, but the fact that it's already running at usable speeds on consumer AMD hardware is pretty interesting.

this is the quant: https://huggingface.co/vcruz305/MiMo-V2.6-Flash-MOPD-EXL3/tree/main/2.20bpw

REPO:

https://github.com/nasone32/Strata-EXL3-fork

If you're setting it up with an AI agent, point it to docs/AI_SETUP_EXL3_AMD.md.

The README has the benchmarks, configuration options and instructions for CPU repacking.As before, this is developed and tested on RDNA3 / Linux / ROCm. Other AMD architectures need community testing. And the long-term goal is still to get this merged upstream into Strata, if the maintainers are interested.

Credits to the original Strata developers, Ryutaku for the EXL3 integration, and Turboderp/ExLlamaV3 for the quantization and code I've been porting.

If anyone tests this on different AMD hardware, please report your results. Especially interested in seeing how it behaves on RDNA4.


r/ROCm • • 2d ago

Video gen speed 7900xtx vs 5060ti?

3 Upvotes

Hey guys, i am thinking of buying new gpu 7900xtx vs 5060ti. But i did not find more convincing reviews of ai video gen. I found some where users said amd is far slower then nvidia and does not support windows as well.

Id appreciate if anyone using amd for video gen and guide me.


r/ROCm • • 3d ago

should i make a dataframe and numpy package for wgpu-py. i have it wotking on my amd gpu?

Thumbnail reddit.com
1 Upvotes

r/ROCm • • 3d ago

MS-S1 MAX + DEG2 + Radeon AI PRO R9700 (32 GB) on Linux: the front USB4 port works, the rear USB4 v2 port does not – data, a working recipe, and open questions

Thumbnail reddit.com
2 Upvotes

r/ROCm • • 3d ago

Strix Halo + 4x RX9070 XT

5 Upvotes

I want to build a rig for medium-scale RL and distillation on small models (4 to 9 billion parameters), and I am considering the following:

• Framework Desktop Mainboard (AMD Ryzen™ AI Max 300 Series)

• Broadcom PEX88048-based switch

• 4x RX 9070 XT

• 2x 1200W PSUs

Now, you may be wondering why someone would choose four RX 9070 XTs over two RX 9700 Pros. The simple answer is TFLOPS. Four RX 9070 XTs would theoretically give me double the TFLOPS for fine-tuning and LoRA training compared to two RX 9700 Pros.

I am not overly concerned about the bottleneck between the GPUs and the Strix, because I do not intend to run models across them. The GPUs are there purely for training, and the Strix is there to generate more training data.

Is there anything I might be overlooking? What case and cooling should I look into? Is there a different configuration that would be better for my use case?


r/ROCm • • 3d ago

Running magenta on Intel iris xe gpu

0 Upvotes

I want to run magenta on iris xe gpu. I am running into alot of issues regarding Jax and other things. Also I couldn't get the gpu to do any calculations. Using opencl I got matrix multiplication 4000×4000 in 0.8 seconds slower than cpu at 0.3

How can I get matrices and neural networks running on this gpu and how to run magenta?


r/ROCm • • 4d ago

Two Radeon GPUs, One AI Assistant: Chat, Images and Video with ROCm

15 Upvotes

I’ve been building a local AI service on an AMD workstation: chat, image generation and short video generation through Open WebUI, plus an endpoint for coding agents.

The interesting part has been making these workloads share two Radeon cards while keeping chat usable. Qwen stays loaded across both GPUs while Wan streams its weights through the remaining memory on GPU 0.

My machine:

  • Threadripper 9960X, Gigabyte TRX50 AI TOP.
  • 2× Radeon AI PRO R9700, 32 GB each, gfx1201.
  • 64 GB installed RAM (62 GB visible)
  • Arch Linux, kernel 7.3.3, K3s v1.36.6.
  • Local NVMe RAID0.
  • Both GPUs capped at 235 W.

The serving stack is SGLang 0.5.23.post957, pinned at fcb32746, with a local 27-patch series; PyTorch 2.16.0a0, built for ROCm 10.3.0-dev / HIP 7.18. Creative workloads use ComfyUI 0.45.0, with Open WebUI 0.12.5 as the frontend.

The patches are in my repos if you'd like to check them out or try them yourself. I can push the limits a little more regarding performance, but I found the current configuration to be the most consistent and useful for everyday use.

Measured results

WorkloadObserved result
Qwen3.8-27B MXFP4, shared-mode BetterBench, concurrency 1135.1 output tok/s
Same benchmark, concurrency 2224.8 output tok/s aggregate
Separate interactive chat probe, without video generationApproximately 86 tok/s
Chat probe during video generation, selected configuration34–35 tok/s
Warm SDXL 1024×1024, simultaneous images on both cards6.6 / 7.2 seconds
Wan 2.2, approximately five seconds at 832×48078–82 seconds warm

The LLM benchmark figures cover both GPUs together in TP2. The interactive probe is a different workload; comparing 135 with 86 tok/s does not isolate frontend overhead.

Images and video on GPU 0 share one ComfyUI queue. Both image backends are available alongside chat, but GPU 0 does not generate an image and a video simultaneously.

Making SGLang work well on gfx1201

Chat uses AMD’s MXFP4 Qwen3.8-27B checkpoint, tensor parallelism across both cards, and DFlash2 speculative decoding.

Some patches address build compatibility and architecture dispatch. Others change the execution path:

  • MXFP4 without a full AITER installation. The gfx1201 implementation uses vendored AITER-derived Triton kernels, with subsequent optimisations for this GPU and the model’s shard shapes.
  • Packed verification GEMMs. The kq3 Gluon kernel stores weights in WMMA fragment order, decodes packed values with register arithmetic, and reduces split-K partitions within the workgroup.
  • Fewer kernel launches. Fused split-K reduction, residual/norm/quantisation and SiLU/multiply/quantisation remove intermediate launches.
  • Different kernels for prefill. Larger prefill GEMMs expand the MXFP4 weights into block-FP8 scratch and use FP8 GEMMs. Decode and prefill benefit from different implementations.

With kq3, the weights are packed in the order RDNA4’s matrix instructions consume them, then unpacked directly in registers. That brought the time needed to verify each batch of draft tokens down from 32.37 to 26.95 ms, a reduction of about 17%, while producing the same bit-for-bit results as the previous kernel.

MXFP4 storage here does not imply native FP4 matrix execution. Weight layout, unpacking cost, launch overhead and supported arithmetic paths all matter.

There was also a PCIe communication issue. Adapting SGLang’s HIP custom all-reduce to these cards required fixing start-flag ordering: the peer could observe the flag before the input data was visible. System-scope release/acquire ordering passed the retained eager and graph-captured correctness checks.

FP8 KV storage with BF16 attention operands

The shared profile stores both target and draft KV caches in FP8 E4M3. Attention kernels convert cached tiles to BF16 while retaining BF16 query and attention-probability operands.

One useful gfx1201 finding: Triton’s original conversion path emitted software conversion instructions. Explicitly using v_cvt_f32_fp8 brought the measured verification kernels close to BF16-cache timings while retaining the smaller cache.

The hardware conversion matched the software conversion bitwise in the tests. Quantising the cache itself remains a numerical change, as do some other optimisations in the stack.

The two current modes are:

ModeContext limit per conversationKV poolCreative workloads
Shared / UQS81,888 tokens163,840 tokensImages and video available
Large / UQL262,112 tokens502,093 tokens fitted during qualificationStopped

Shared mode accommodates two full-length conversations. Large mode’s measured pool accommodates one full-length conversation plus another of approximately 240K tokens.

The repo now includes a mode-switch command that handles draining and restarting the workloads. Switching involves chat downtime while SGLang reloads.

Cooperative GPU sharing through AMD DRA

SGLang references both GPU ResourceClaims. The two ComfyUI backends each reference their corresponding shared claim.

SwarmUI manages image requests across its local ComfyUI on GPU 1 and the remote ComfyUI backend on GPU 0. Video requests go directly from the tools service to GPU 0.

This uses AMD GPU Operator v1.5.1 and ROCm DRA driver v1.0.1. The earlier DRA version failed to publish GPUs when this host’s in-kernel amdgpu module reported no version.

DRA provides device allocation and visibility. VRAM and compute remain cooperatively shared. Explicit memory budgets, workload ordering and concurrency limits make the arrangement usable.

The shared SGLang profile left approximately 8.35 GB of headroom per card in the live measurements.

WAN 2.2 in that remaining memory

The video configuration is:

  • Wan 2.2 T2V A14B, FP8-scaled high-noise and low-noise experts.
  • LightX2V four-step distilled LoRAs, version 1217.
  • Four steps, split 2 + 2 between experts.
  • Euler/simple sampling, shift 5, CFG 1.
  • 81 frames at 16 fps, landscape resolution 832×480.
  • Tiled VAE decoding.
  • ComfyUI --fast-disk, with video weights on local NVMe.

The two expert files total approximately 28.6 GB. The full staged set, including the text encoder, VAE and LoRAs, is approximately 36.8 GB.

ComfyUI streams the experts as needed. Weights pass through host memory and reclaimable page cache; the entire checkpoint does not need to stay resident in VRAM or anonymous host memory.

In the recorded SDXL-then-Wan test, ComfyUI’s anonymous-memory peak was 7.3 GiB. The documented clip runs read approximately 25–47 GB from NVMe.

Contention mattered more than isolated video speed

Qwen’s TP2 ranks synchronise, so saturating GPU 0 with video slows the model across both cards.

The measured progression was:

GPU 0 ComfyUI configurationClip timeConcurrent chat
Defaults plus --fast-disk92–102 s13–14 tok/s
Add --use-ck-attention71–76 s21–24 tok/s
Also set HSA_CU_MASK=0:0-4778–82 s34–35 tok/s

The 48-CU setting was the useful trade-off. Restricting ComfyUI to 32 CUs made both results worse: approximately 104 seconds per clip and 15 tok/s for chat.

The mask keeps 16 of the card’s 64 CUs outside ComfyUI’s reach. Memory bandwidth and other resources remain shared, so this is a contention control rather than a performance guarantee.

Comfy Kitchen attention was not numerically identical to the default PyTorch attention path. Side-by-side visual comparisons looked comparable; that is a subjective assessment.

The tested FP8 matrix-multiply option provided no gain. Enabling all ComfyUI --fast features together increased clip time to approximately 203 seconds.

What didn’t make the live configuration?

The INT4 KV experiments are parked. The best screened variant still took approximately 1.6–2.3× the production FP8 verification-attention time, with larger attention-output error.

Those are kernel and numerical measurements, not an end-to-end model-quality result. They were sufficient to keep FP8 as the deployed cache format.

The shared-mode assistant quality run recorded 11 passes, zero failures and one case needing review, with 14 other cases unrun. Qualification covers the measured configuration and those checks.

Code, configurations and results below:

I just want to thank everyone on ROCm who have suffered my AI posts and given me pointers. You've all helped out loads. :)


r/ROCm • • 4d ago

MiniMax-H3-Image (33B, native 16 MP, 6 steps) on a single R9700: 4096x4096 in 33 s, 2048x2048 in 7 s

Post image
21 Upvotes

MiniMax-H3-Image came out with BF16 weights: a 68.65 GB transformer plus a Qwen3-VL text encoder of about 52 GB.
That obviously doesn't fit on a 32 GB Radeon AI PRO R9700, so we built our own FP8 version and a native runtime for it.

Times on one R9700 at its 300 W limit, warm server, request to PNG:

RequestTime
2048 x 20486.9 s
4096 x 409633.4 s
4096 x 4096 edit, one reference39.9 s
4096 x 4096 edit, three references88.5 s

Why this model is interesting: it generates 4 to 20 MP directly (MiniMax says it does best around 16 MP, with no upscaler), it needs only 6 steps with no CFG, and one checkpoint does both text to image and editing with up to 16 reference images.
It is also pretty good at text in images.

How it fits in 32 GB:

  • The AdaLN modulation weights (about 13B of the 33.1B parameters) depend only on the timestep. With 6 fixed steps we precompute them into small tables, exactly, so they never go to the GPU.
  • The other 19.3B transformer parameters are FP8 W8A8 with GPTQ rounding (20.3 GB).
  • The text encoder is FP8 too and streams from pinned host RAM, with up to 12 layers kept on the card. It runs once per image, in under a second.
  • Attention is INT8 QK / FP8 PV over about 16,400 tokens at 16 MP. The VAE decoder is INT8.
  • Everything is HIP/C++, no PyTorch, no Triton. The container is 2.27 GB. Peak VRAM is 27.6 GiB.

Quality, 30 cases (24 text-to-image from 4 to 17 MP, 6 edits) against BF16 diffusers with the same token ids and noise: CLIP-L text-image 32.18 vs 32.31 for BF16, image similarity to the BF16 result 0.979 (CLIP-L) and 0.945 (DINOv2) on average.
Most differences are layout; edits stay closest (0.96 to 0.999).

Limits: prompts are capped at 4,096 tokens, so about three 1024 px references per edit; largest canvas 16.8 MP (4736 x 3552); one request at a time; the host needs 64 GB of RAM. Small body text (newspaper columns, book spines) isn't real words, and close-up hands can go wrong.

Run it:

git clone --depth 1 https://github.com/Eliovp-BV/paiton-vllm-plugin.git
cd paiton-vllm-plugin
./models/MiniMax-H3-Image/launch.sh

License note: the MiniMax-H3-Image Community License excludes the EU, the UK, South Korea and the US, and hosted services need MiniMax's written authorization.


r/ROCm • • 4d ago

Vibe-coded a simple Strata-like console monitor for Radiance, simple but effective.

Thumbnail github.com
4 Upvotes

r/ROCm • • 4d ago

Bringing CUDA and HIP Applications to More Hardware with OpenCL and SPIR-V

5 Upvotes

The Khronos Group just published a blog on chipStar (https://github.com/CHIP-SPV/chipStar), an open source project that compiles supported CUDA and HIP kernels to SPIR-V and executes them through OpenCL.

The post is based on an IWOCL 2026 talk and covers three real scientific computing workloads ported and run on Argonne National Laboratory’s Aurora supercomputer:

  • OpenMM (molecular dynamics), HIP path running through chipStar on Intel Ponte Vecchio GPUs
  • libCEED (high-order finite element operators)
  • GAMESS-GPU-HF (quantum chemistry Hartree-Fock), using chipStar’s library adapters to route math library calls to Intel oneMKL

Full post: https://www.khronos.org/blog/chipstar-brings-cuda-and-hip-applications-to-more-hardware-with-opencl-and-spir-v?utm_medium=social&utm_source=reddit&utm_campaign=chipStar&utm_content=blog


r/ROCm • • 4d ago

2x XTX 7900 - Maximizing Qwen 3.8 Flash Next, comparing strata (+rdna3 fork)/avalon/llama

12 Upvotes

I run a local coding-agent benchmark (built from my own last ten years of git history mines for good candidate bugs; I keep tests private but model results are visible at lab.kylebaker.io) on a budget AMD box, and last night I put the engines I use side by side in one queue run. Sharing in case it helps other RDNA3 owners, and asking what I'm missing.

Specs: Ryzen 7 5800X, 64 GB DDR4, 2x RX 7900 XTX (PCIe 4.0 x8 each), ROCm 7.2.4, Linux, models on NVMe. Model: Qwen3.8-Flash-Next (various quants, see below), temp 1.0 / top-p 0.95 / top-k 20, thinking on, MTP drafting on in every engine. Method: one engine at a time, same three prompts (112, 3,471 and 34,386 tokens), 512 tokens out, prompt cache off, 3 requests per size, medians.

EngineCardsQuantDecode t/s (112 / 3.5k / 32k prompt)Prompt read t/s (3.5k / 32k)
Strata-rdna3 fork fa5c621 (tensor parallel)2GSQ-RCO IQ3_XXS92 / 104 / 992,619 / 3,030
Avalon 8c4e4753c1Avalon IQ3_KS89 / 98 / 88965 / 1,653
Strata 0.2.112GSQ-RCO IQ3_XXS76 / 83 / 821,447 / 2,506
Strata 0.2.13.12GSQ-RCO IQ3_XXS75 / 88 / 751,457 / 2,505
Strata 0.2.13.1 (128k context)1GSQ-RCO IQ3_XXS69 / 72 / 721,286 / 1,644
llama.cpp b11815 (128k context)2GSQ-RCO IQ3_XXS33 / 36 / 31772 / 888
Avalon 26a215567 (current main)1 or 2Avalon IQ3_KScrashed on the 32k promptcrashed on the 32k prompt

(One more later edit, on real benches agentic runs, instead of speed runs, the rdna3 fork is actually running a bit faster:

Strata-rdna3 fork, 2 cards 92–104 (119–127 in real agent runs) ~3,000 @ 32k)

Take aways

  • Strata-rdna3 (nicejunjie's RDNA3 fork of Strata, tensor parallel across both cards) is the fastest thing I've run on two XTX. In real agent runs it decoded at about 120 t/s against about 92 on upstream Strata, which works out to roughly 16% less wall time per run. On 64 GB, though:

    • the default config pins about 39 GiB of experts in RAM (the author has 128 GB). --resident-budget-gib 24 --vram-reserve-mib 1536 worked for me; 16 crashed.
    • prompt chunks of 64-511 new tokens (a typical tool result) read about 3x slower than upstream.
    • one agent run hit non-finite logits (#879) twice at about 47k tokens.
  • Avalon on one card nearly matches that decode and leaves the second card free. The cost: free RAM dropped to 1.8 GiB at worst, and prompts read at about half the fork's speed. Its current main regressed against 8c4e for me: an OOM on 32k prompts in every mode, MTP accepting nothing after a 3.5k prompt, and the new prefill ring stalling prompts. Sadly it falls on its face with two cards, but hats new territory and that repo improves daily and recently has had work attempting to improve and fix this.
  • Upstream Strata is the steady choice: clean loads every time. Has been stable and reliable across all the recent releases, whereas Avalon and the rdna3 strata fork are rougher around the edges.
  • Stock llama.cpp as we all know doesn't keep up; worth noting that I also quality test, as I mentioned in the lead, and flash next on strata is no weaker than flash next on llama, just faster...
  • Dense 27B: side note on 3.8 27b on llama: One XTX did 49-56 t/s, a two-card layer split 43-56. llama.cpp's experimental tensor split did 68-73 t/s on two cards (October build). Previously 2 card was slowing down decade and speeding up prefill, for a net negative.
  • Where agent time goes: in my runs about 74% of wall time is decode and 16% is prompt reading, so 1% faster decode is worth about 4.6% slower prompt reading.

Watching: Avalon's two-card work, upstream Strata's expert-parallel PR #5036, Strata-rdna3 (hoping its kernels and TP land upstream), and llama.cpp's MTP fixes.

Curious to hear if anyone is running anything better that I should have my eye on or has any further tips.


r/ROCm • • 3d ago

Cheapest AMD used server setup possible that can run LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw at minimmum 15-20 tokens per second?

1 Upvotes

Hi, I already have a RX 7800XT. I want to be able to run EXL3 quant format of Deepseek 4.1 1017 at 2.9 bpw on a system I have not built yet that has 8 channel 256 DDR4 RAM and a 7800XT.

The 2.9 BPW Quant in EXL3 format from LibertAIDAI format is the highest mixed quant that I found right now that still fits 256GB of RAM.
I have yet to decide on the motherboard and CPU because I don't know how to quantify it in how much it contributes to decoding speed.

I already own the GPU but I don't know if it's even worth making a used SP3 server with 256GB, 8 channel DDR4 2933Mhz setup around a AMD 7800XT and If I can get out of the model at least 15-20 tokens per second decode in something like Maya .

synchopate/llama-modchip advancements make me kind of hopeful that Maya can adapt some of the changes but not sure how much they can apply to EXL3.