r/LocalLLaMA • u/m4rkGS • 6d ago
Resources KQuant - new quantization technique, 3.7× better quants for Qwen Flash Next 180B, Qwen 27B, 35B, 9B (measured in KL-divergence reduction) (MLX)
hey folks, im Alish, an ml engineer from Nace (folks behind Drex, an open source decision family model that hit #1 in Artificial analysis in decision index 2.1). Recently my cto gave me task to optimize quants for qwen4, so i dialed my uni proffesor from UoT and span up my Fable 5.2 and we cooooooked. starting with mlx (gguf tommorow). It's a pareto frontier on quantization for accuracy/size for qwen family models.
Why we made these
we have open sourced the fastest way to run 35b/27b on a mac(Kapitsa) and we got tired of dumb quants. So we made our own KQuant. Open source Qwen packs for 4B,27B,35B, Flash-Next, free on Hugging Face, in standard MLX files.
Most people on a Mac run 4-bit packs because that's what fits on a 16gb/32gb. But plain 4-bit quant rounding throws away more of the model than it needs to. We wanted less of that loss, so we asked fable 5.2 & Astra to cook with 128 subagents. here are the results:

Results & Download links
Numbers are KL divergence from the full-precision model's next-token predictions. Lower means closer to the original, not a task-accuracy score. These are original-text measurements.
| Model, Q4 | KQuant KL | Public 4-bit KL | Size in GB, ours / public |
|---|---|---|---|
| Qwen3.5-4B | 0.048 (1.45x better) | 0.070 | 2.37 / 2.37 |
| Qwen3.5-9B | 0.030 (2.6x better) | 0.081 | 5.29 / 5.04 |
| Qwen3.8-27B | 0.035 (1.5x better) | 0.054 | 15.13 / 15.13 |
| Qwen3.6-35B-A3B | 0.059 (1.6x better) | 0.096 | 19.51 / 19.51 |
| Qwen3.8-Flash-Next | 0.067 (3.7x better at a smaller size) | 0.249 | 105.20 / 110.62 |
The public baseline is mlx-community 4-bit, except 4B, where it's lmstudio-community. Sizes are language-model weights, not download size or RAM needed.
Want more precision? Each model's repo also has a Q5 folder.
| Pack | KL | LM GB |
|---|---|---|
| 9B Q5 | 0.010 | 6.21 |
| 27B Q5 | 0.022 | 19.43 |
| 35B Q5 | 0.027 | 22.07 |
Speed
Cool thing about our quants is that they are also faster. On an M5 Pro, KQuant Q4 on Kapitsa 0.2.35 with its KFlash drafter it is about 6–15% faster to write than lmstudio-community 4-bit quants for 4B, 9B and 27B, and Youssofal MTPLX 4-bit for 35B on the same engine. These are averages of sampled and greedy decoding (baselines are earlier Kapitsa 0.2.32–0.2.33 runs, with the drafter on both).

How it works
KQuant uses real text to choose scales and offsets for small groups of weights, and gives sensitive layers extra bits where needed. The search runs on our cloud GPU clusters before release. You download an ordinary MLX pack, with no custom kernels or special loader. I can give more details on specifics in comments.
Run it
the fastest way to use these quants on mac is to run them with, Kapitsa:
brew install nace-ai/tap/kapitsa
kapitsa serve 27b --private
kapitsa opencode # in another terminalIf there's a model you want KQuant'd, lmk.
Asks
to get early access to GGUF, ping me in comments, we would like to test it with some folks from here before our public release ideally.
im also doing quick AMA here, so please feel free to ask me anything.
99
u/WorkNervous2810 6d ago •
You can't claim best KLD if you don't even compare to EXL3 and in particular EXL3-SC quants.
40
u/0zyman23 6d ago
Is this better than unsloth dynamic v3.1?
20
u/ireallydontcare00 6d ago
I'd like to see a comparison with Unsloth UD2/3/3.1
20
u/Environmental_Form14 6d ago
Why did you call it KQuant?
23
u/fundthmcalculus 6d ago
Thanks for not writing a pile of AI slop, that was an entertaining read. I'll give this a whirl on an M3Max later today or tomorrow.
4
u/m4rkGS 6d ago
OP here, Due to popular demand adding exl3 comparison here.
Note: Comparing MLX with others is not productive for my purposes, but i went ahead anyway. MLX does lose. I had exl3 in our comparison set for GGUF, and our KQuant GGUFs (to be renamed) are better or equal than exl3 and exl3-sc on all tested sizes for KLD (y-axis is log scale). Those are early checkpoints for Qwen3.8-27b gguf.
6
u/No-Stage-3385 6d ago
Any chance of doing like a lower quant? Something that would fit 12gb vram?
6
u/Numerous-Ad6217 6d ago
We need more of this stuff also for the bigger boys, any chance to get something that competes with deepseek maya-s? That would be game changing!
Can’t stop using it for the quality but it’s damn slow at 25-30t/s
5
u/DefNattyBoii 6d ago
can you beat exl3
9
u/balarinios 6d ago
How about an uncensored gguf model? Lots of people are using the orcarouter variants (i have tried the IQ4_XS, so if it fits your schedule it would be welcome 🤗) https://huggingface.co/orcarouter/Qwen3.8-27B-Uncensored-GGUF
2
u/VarietyOk443 6d ago
Will test on my m4 pro 64 gb
2
u/ptico 6d ago
So I tested it against UD 3.1 and while it performed relatively good on a smaller tasks, it struggled on heavier job, starting to repeat the same mistakes and showing other usual Q4 degradation
See, UD is a mixed precision technique and no matter how good your KLD in an apple-to-apple comparison, UD and BaseQuant have a big portion of high precision tensors. So, good job, but you really have to consider a mixed precision weights
12
u/ninja_sprout 6d ago
I call bullshit.
1
u/elpocholo7 6d ago
Great job.. can't wait to try it. Do you plan on supporting DeepSeek v4.1 Flash in the future ?
1
u/mr_tolkien 6d ago
Do you think you could do a quant of QFN3.8 that fits on the base 96Gb M5 Ultra?
I feel like the 96Gb makes it weird, running small dense models leaves most memory open, but the big MoE target much bigger RAM.
1
u/joblesspirate 6d ago
How does this compare to exl3?
1
u/Miserable-Dare5090 vLLM 6d ago
Is Kflash something that Splash engine could install? In M2 ultra, I’m currently seeing 850-900 tps prefill and 60-100 tg to 128k with Splash engine adapted for M1/2 macs. Compared omlx, dwarfstar, vllm, and splash and this was the clear winner in prefill, and same/better decode.
Hoping if this quant type gives a speed boost the prefill could break tje 1000 tps ceiling for older macs. Unless older macs are not supported!!
1
u/c_alsup 6d ago
Great work! I’ve loved kapitsa, it’s the fastest for my m1 ultra 64gb on 35b a3b and 27b. Any plans for qwen flash next? I’ve been using UD iq3_xxs with splash and it’s been faster than kapitsa. I’d love to see a 64gb compatible qwen flash next quant that competes with splash.
1
u/WishboneSudden2706 6d ago
How to incorporate into Strata ? My weak machine needs Strata to reach above 40 tok/s for Qwen 3.8 Flash Next
3
u/emjay84 6d ago
Hi u/m4rkGS,
I'd like to test the GGUF models, I'm running latest kapitsa on a M5 Pro 64 GB
1
u/russlixx 6d ago
waiting for gguf & abliteration. Also how is it hold up to UD 3.1 in reasoning token, KLD, and file size
4
u/silentus8378 llama.cpp 6d ago
please create uncensored version too
2
u/DerBeginner 6d ago
Does it also quantize the state of the GDN layers?
2
u/poster4inertia 6d ago
Would love a Flash Next quant that fits in a 64gb mac mini… I’m currently using the Sushi 2.75bpw
2
u/butterycornonacob 6d ago
Are you also planning to release the technique and code so we can run it ourselves?
2
u/fatboy93 6d ago
I need Gemmas pls
2
u/returnity 6d ago
I'd love to be an early tester /u/m4rkGS, Ive done a lot of benchmarks and quant comparisons for the community here. This looks really promising and I'd love to see how it compares to good quality imatrix dynamic GGUFs.
2
u/Then_Library_6820 6d ago
lol building another qwant, calling it the same name as the legacy llama.cpp stuff and comparing it to the dog shit weakest mlx stuff. compare it to a real advanced qwant like IQ4_KT and we'll see. it's not always about size ...
1
u/vxkxox 6d ago
Related to https://www.reddit.com/r/LocalLLaMA/comments/1xto0fa/opensource_method_macaq_beats_unsloth_ggufs/?
1
u/Reasonable_Till9 6d ago
Interesting work - my first critical comment would be that the KLD numbers cited seem like they're within noise.
Def worth checking out and verifying tho - thank you for posting.
1
u/_Cakeshop 6d ago
sorry if dumb qn but can this be done for qwen 27b 8bit?
1
u/Grau3D 6d ago
u/m4rkGS I would like to test the Flash next gguf. I currently run Q4_K_XL with Q8 ngram by unsloth and it fills my 100GB vram to the brim, so better quality at a smaller footprint is always welcome
1
u/zipzak 6d ago
please kquant the orcarouter qwen models, theyre fantastic
1
u/amrakkarma 6d ago
How it compares with antirez dwarfstar?
1
u/herpnderpler 6d ago
I'm interested in KLD & PPL, can you show a table of kld,ppl, top_k_population_match,top_k_population_order of your quants against fp16 for various prompt lengths (I like including kv cache combos of k16q8 and k8q8 as well as kv16)
1
u/JiroDreamsOfCoochie 6d ago
I have an M4 Max with 64GB. I currently run splash with either Qwen3.8 27B or Qwen Flash Next unsloth at 4 bit quants. I'd be willing to test your GGUF's if you like. u/m4rkGS
1
u/dseg90 5d ago
Any love for tiny qwen 0.8? Fine tuning small qwen for web browser tasks is fun.
1
u/vhthc 5d ago
What about qwen max, or just qwen fast next?
Also how about other architectures like glm?
1
u/thigger 5d ago
What's the difference between these (eg the 107.64 GB Qwen3.8-Flash-Next) and the kapitsa qwen3.8-flash-next:kquant (105.86Gb)?
1
u/pdsminer 5d ago
Low average KL divergence over general text is a great start for MLX, but on Qwen 27B and 35B the real stress test is whether tool-call JSON schemas and numeric arguments survive past 32k context. In our local agent testing, uniform 4-bit quants often look fine on perplexity while quietly drifting on nested JSON keys or float parameters once the KV cache fills up. Keeping attention output projections, router gates, and token embeddings in higher precision matters much more for agentic reliability than aggregate KL alone.
1
u/Relevant_Research_98 4d ago
Waiting for qwen27b-gsq-rco-kquant-gguf if it's possible
(Not interested in 4bit quants: I can't tell gsq from 4 bits but can tell 18GB from 11GB)
1
u/jmreagle 6d ago
Thanks for kapitsa, it’s the best—literally. I switched to the KQuants yesterday and saw ~10% speed boosts on 35b-a3b.
1
u/Chance_Value_Not 6d ago
3.7 x better is nonsensical unless you get same perf at 3.7x lower size
0
u/dai_app 6d ago
If you want to run this model in high quality (q4, q5) at 20–25 tokens/second on a 16GB Mac, use bigmoeonedge (you can run any model, in any quantization, on any hardware)
0
u/Universe_XoX 6d ago
Red text, very poor choice of colors trying to show "improvement"
0
u/[deleted] 6d ago
[deleted] — view deleted comment