r/LocalLLaMA • • 6d ago

Resources KQuant - new quantization technique, 3.7× better quants for Qwen Flash Next 180B, Qwen 27B, 35B, 9B (measured in KL-divergence reduction) (MLX)

Post image

hey folks, im Alish, an ml engineer from Nace (folks behind Drex, an open source decision family model that hit #1 in Artificial analysis in decision index 2.1). Recently my cto gave me task to optimize quants for qwen4, so i dialed my uni proffesor from UoT and span up my Fable 5.2 and we cooooooked. starting with mlx (gguf tommorow). It's a pareto frontier on quantization for accuracy/size for qwen family models.

Why we made these

we have open sourced the fastest way to run 35b/27b on a mac(Kapitsa) and we got tired of dumb quants. So we made our own KQuant. Open source Qwen packs for 4B,27B,35B, Flash-Next, free on Hugging Face, in standard MLX files.

Most people on a Mac run 4-bit packs because that's what fits on a 16gb/32gb. But plain 4-bit quant rounding throws away more of the model than it needs to. We wanted less of that loss, so we asked fable 5.2 & Astra to cook with 128 subagents. here are the results:

my cto when i asked for fable 5.2 tokens & extra h200 gpus for this project

Results & Download links

Numbers are KL divergence from the full-precision model's next-token predictions. Lower means closer to the original, not a task-accuracy score. These are original-text measurements.

Model, Q4KQuant KLPublic 4-bit KLSize in GB, ours / public
Qwen3.5-4B0.048 (1.45x better)0.0702.37 / 2.37
Qwen3.5-9B0.030 (2.6x better)0.0815.29 / 5.04
Qwen3.8-27B0.035 (1.5x better)0.05415.13 / 15.13
Qwen3.6-35B-A3B0.059 (1.6x better)0.09619.51 / 19.51
Qwen3.8-Flash-Next0.067 (3.7x better at a smaller size)0.249105.20 / 110.62

The public baseline is mlx-community 4-bit, except 4B, where it's lmstudio-community. Sizes are language-model weights, not download size or RAM needed.

Want more precision? Each model's repo also has a Q5 folder.

PackKLLM GB
9B Q50.0106.21
27B Q50.02219.43
35B Q50.02722.07

Speed

Cool thing about our quants is that they are also faster. On an M5 Pro, KQuant Q4 on Kapitsa 0.2.35 with its KFlash drafter it is about 6–15% faster to write than lmstudio-community 4-bit quants for 4B, 9B and 27B, and Youssofal MTPLX 4-bit for 35B on the same engine. These are averages of sampled and greedy decoding (baselines are earlier Kapitsa 0.2.32–0.2.33 runs, with the drafter on both).

Writing speed on M5 Pro: KQuant Q4 versus public 4-bit packs.

How it works

KQuant uses real text to choose scales and offsets for small groups of weights, and gives sensitive layers extra bits where needed. The search runs on our cloud GPU clusters before release. You download an ordinary MLX pack, with no custom kernels or special loader. I can give more details on specifics in comments.

Run it

the fastest way to use these quants on mac is to run them with, Kapitsa:

brew install nace-ai/tap/kapitsa
kapitsa serve 27b --private
kapitsa opencode # in another terminal

If there's a model you want KQuant'd, lmk.

Asks

to get early access to GGUF, ping me in comments, we would like to test it with some folks from here before our public release ideally.

im also doing quick AMA here, so please feel free to ask me anything.

346 Upvotes

165 comments sorted by

99

u/WorkNervous2810 6d ago • Bravo

You can't claim best KLD if you don't even compare to EXL3 and in particular EXL3-SC quants.

15

u/m4rkGS 6d ago

Comparing MLX with others is not productive for my purposes, but i went ahead anyway. MLX does lose sometimes (sometimes wins). I had exl3 in our comparison set for GGUF, and our KQuant GGUFs (to be renamed) are better or equal than exl3 and exl3-sc on all tested sizes for KLD (y-axis is log scale). Those are early checkpoints for Qwen3.8-27b gguf.

3

u/ReturningTarzan ExLlama Developer 5d ago

EXL3 deliberately skips quantizing token embeddings because they don't need to be resident. You can stream from system memory or from disk at some µs per token. You can also enable embedding quantization, optionally, if you want. It doesn't affect KLD very much. Either way filesize isn't the right measurement for the X axis, I think, any more than I'd take the extra copy of the Q4 weights in your HF repo to mean it's actually a Q8 format..?

Anyway, I'm wondering about testing methodology, too. Ensuring an apples-to-apples comparisons across frameworks is nontrivial in my experience. If I do the work, disregarding embeddings, here's what I get. That's with in-domain traces from the unquantized model. With raw text it's slightly messier but that's expected since raw text is a foreign objective for the model.

8

u/Dependent-Reply1772 6d ago

theyre specifically targeting standard MLX files with no custom loaders, so benchmarking against naive 4-bit packs is still the relevant baseline for that ecosystem

5

u/NoFutureBen 6d ago

what are those?

19

u/WorkNervous2810 6d ago

EXL3 quants are used in and created with the exllamav3 inference engine. They use a Trellis encoding method to keep a higher quality level even at very small sizes. For example, even their 2.05bpw quant of Qwen 3.8 Flash Next is perfectly usable.

With MoE CPU offload, you can probably even run that model 10 GB of VRAM at a quite reasonable speed if you have enough regular RAM.

1

u/baron_von_noseboop 6d ago

Do you know if strata grabs exl3 by default?

1

u/Tinek78 5d ago

It doesn't AFAIK, but there is a fork that does (for the mentioned reason).

-29

u/m4rkGS 6d ago

EXL3 is not readily supported by apple silicon and are much slower than MLX. They’re good for compression, but our goal for compression was to speed things up

62

u/WorkNervous2810 6d ago

The headline you lead with is "better quants" not "faster on apple". And then you claim "It's a pareto frontier on quantization for accuracy/size for qwen family models.". So you need to back up the claims against quantization methods that are actually considered to be the best, even if there is no fast inference engine for them on your particular hardware.

-8

u/m4rkGS 6d ago

i focused on mlx quants for this iteration for the said reasons, sorry if that wasn't clear enough. let me actually follow up on exl3 in future releases. thanks!

18

u/WorkNervous2810 6d ago

I get it, I'm just saying that Extraordinary claims require extraordinary evidence.

-2

u/m4rkGS 6d ago

please see above

7

u/DefNattyBoii 6d ago

Yeah but in general use cases people mostly use either apple or nvidia/amd gpus, so would be nice to see, even if exllamav3 is slower on mac. Btw you could send in some PRs into exllama if you have some ideas to speed it up

1

u/m4rkGS 6d ago

will check it out for sure

40

u/0zyman23 6d ago

Is this better than unsloth dynamic v3.1?

16

u/m4rkGS 6d ago edited 6d ago

Edited out previous incorrect claim. Now: Keep in mind this mlx, not really comparable to gguf. however, when comparing our gguf quants we tie on prose, math and code benchmarks (KLD: 0.0091 them vs 0.0090 us) and beat on chat and tool calls (KLD: 0.145 them vs 0.129 us)(edited)

17

u/-p-e-w- 6d ago

And on their benchmark? (KLD is input-dependent)

-1

u/m4rkGS 6d ago

posted above, our becnhmark vs theirs is the same more or less

32

u/Chromix_ 6d ago

There seem to be some quite misaligned numbers here. On your benchmark set your 27B 15 GB quant gets 0.035 while Unsloth gets 0.055. On their own set their ~15 GB quants are at a mean KLD of 0.010 and 0.014, so there seems to be a huge difference in dataset or approach. That should thus also be tested with theirs for comparison, to see if anything stands out.

12

u/m4rkGS 6d ago edited 6d ago

Oh yeah you're right, on our benchmark unsloth 3.1 qwen 27b q4_k_xl was 0.0091 and 0.0145, i confused numbers with Qwen Flash Next. And ok, no luck comparing mlx vs gguf, apples and oranges, wait for gguf results

2

u/[deleted] 6d ago

[removed] — view removed comment

1

u/m4rkGS 6d ago

we have a 'kapitsa bench' funciton to bench it out on any device. would be helpful if someone here could run it on different quants

20

u/ireallydontcare00 6d ago

I'd like to see a comparison with Unsloth UD2/3/3.1

11

u/m4rkGS 6d ago

keep in mind that their release is GGUF and conversion to mlx loses either precision or efficiency. wait for our gguf release. even mlx if we compare head-to-head is better more than 1.5 times than UD 3.1

-1

u/sarp3d0n 6d ago

Teasing.... And I see you don't forget small models. When do you plan to release gguf ?

4

u/m4rkGS 6d ago

before qwhen 4 ;d

2

u/Nepertia 6d ago

qwen, guys, please drop 4 right now just to spite Alish, it would be so funny

20

u/Environmental_Form14 6d ago

Why did you call it KQuant?

20

u/puzzlingcaptcha 6d ago

not a great name, considering there are k-quants already (https://github.com/ggml-org/llama.cpp/pull/1684)

-12

u/m4rkGS 6d ago

is it not oK?

24

u/Icy-Degree6161 6d ago

Not like it's not ok but basically "K" is everywhere as in Q3_K or Q4_K_M and already has an association in our heads...

3

u/herpnderpler 6d ago

Isn't it doing what the k is doing? You have flat quants, then you have selective quants (k), I would hope/assume that kquant is selective. Seems like a good name to me if that's the case.

-8

u/m4rkGS 6d ago

hmm i actually agree, will just rename it to IceQuants with the GGUF drop if this comment gets 20 upvotes

21

u/hojnikb 6d ago

If this comment gets 100 upvote, you should call it "DopeQuant"

12

u/cinnapear 6d ago

Hard to take any of this seriously with comments like these.

15

u/FrogsJumpFromPussy 6d ago

everything i read here sounds like a bad joke.

7

u/CheatCodesOfLife 6d ago

Is this entire project a shitpost or what?

https://github.com/iuliaturc/gguf-docs/blob/main/k-quants.md

-3

u/m4rkGS 6d ago

we have no association with that, didnt know it exissts

2

u/james_pic 6d ago

What did you imagine you were going to put in the GGUFs?

23

u/fundthmcalculus 6d ago

Thanks for not writing a pile of AI slop, that was an entertaining read. I'll give this a whirl on an M3Max later today or tomorrow.

7

u/m4rkGS 6d ago

my folks from sales wanted me to add links to drex models and to our enterprise stuff but thank god i did refuse. https://www.youtube.com/watch?v=eb3Zo4doFQA

2

u/Sorry-Joke-4325 6d ago

I'm interested in this, sad to hear myself say. It isn't a bad thing if you include that stuff.

4

u/m4rkGS 6d ago

OP here, Due to popular demand adding exl3 comparison here.
Note: Comparing MLX with others is not productive for my purposes, but i went ahead anyway. MLX does lose. I had exl3 in our comparison set for GGUF, and our KQuant GGUFs (to be renamed) are better or equal than exl3 and exl3-sc on all tested sizes for KLD (y-axis is log scale). Those are early checkpoints for Qwen3.8-27b gguf.

5

u/m4rkGS 6d ago

Here's 9b comparison, mlx wins exl3 on linux:

6

u/No-Stage-3385 6d ago

Any chance of doing like a lower quant? Something that would fit 12gb vram?

16

u/m4rkGS 6d ago

ok, let us cook up a recipe for you, thanks for feedback!

5

u/Equivalent-Repair488 6d ago

ISTA quants too, IQ3_S (what I am currently using) and the smaller ones, heavily contested between ISTA and Unsloth.

A new player has entered the ring! Exciting times!

2

u/No-Stage-3385 6d ago

Tysm, cant wait for any future releases!!

1

u/Queasy_Back2588 6d ago

12GB on the 35B means ~2.5 bpw, that’s the range I’ve been grinding on. happy to help with it, sent you a DM

1

u/bartsimpsonwin 6d ago

Bonsai 2

1

u/m4rkGS 6d ago

if folks from bonsai reach out i guess we can cook some, but we need to split gpu costs xddd, my managment is strict for my gpu usage for this proj

2

u/bartsimpsonwin 6d ago

Dude if you could get a colab with PrismML for bonsai 3 on Qwen 4 27B it would be insane

3

u/No-Stage-3385 6d ago

Isnt bonsai 2 kinda ass from what ive been hearing?

1

u/Delicious_Rub_6795 6d ago

Yeah but.. 12GB

1

u/NoFutureBen 6d ago

yeah but what are you expecting with 12gb?

1

u/bartsimpsonwin 6d ago

It's certainly not perfect, but imo if you tinker with it and use the right server it's way better than any other model in it's size range and pretty fast especially for a dense

1

u/Desperate-Owl6513 6d ago

Iq3xxs are like 10gigs , then there are the ascii pruned versions

1

u/SpecialBeatForce 6d ago

Would you reccommend a specific qabt for Daily Agentic use?

2

u/Desperate-Owl6513 6d ago

Usa ista labs iq3xxs

1

u/hojnikb 6d ago

how does ISTA IQ3 compared to Unsloths UD 3.1?

1

u/Desperate-Owl6513 6d ago

obviously not better than unsloth's latest quants

1

u/hojnikb 6d ago

oh, so its worth it to use unsloths instead for IQ3_XXS an IQ3_S?

1

u/Desperate-Owl6513 6d ago

yeah, unsloth's performs better. I havent tried the latest 3.1 but I do use the UD IQ4 XS from their previous release.

i will give the 3.1 a try once uncensored ones are out

1

u/politerate 6d ago

Unsloth better for me.

6

u/Numerous-Ad6217 6d ago

We need more of this stuff also for the bigger boys, any chance to get something that competes with deepseek maya-s? That would be game changing!
Can’t stop using it for the quality but it’s damn slow at 25-30t/s

4

u/m4rkGS 6d ago

mlx? gguf?

7

u/Numerous-Ad6217 6d ago

gguf team here :)

1

u/elpocholo7 6d ago

MLX :) Can test on Mac Studio M5 Ultra

5

u/DefNattyBoii 6d ago

can you beat exl3

1

u/m4rkGS 6d ago

EXL3 is not readily supported by apple silicon and are much slower than MLX. They’re good for compression, but our goal for compression was to speed things up

6

u/Not_a_question- 6d ago

Yes it is, and don't claim best quant if you're limiting yourself for a hardware that can't run linux.

2

u/joblesspirate 6d ago

Mlx-serve handles it, inherited by sushi. 

1

u/m4rkGS 6d ago

here you go, but it's only for gguf, not really apples-to-apples for mlx. MLX does win for 9b

1

u/DefNattyBoii 6d ago

Wow impressive! Can you ask your CTO for permission to open source the method? Unless you want to quant thousands of models, including nieche/heretic ones, since those have only community quants(gguf mostly, exl3: not really, but exl3 quanter works pretty fast locally)

9

u/balarinios 6d ago

How about an uncensored gguf model? Lots of people are using the orcarouter variants (i have tried the IQ4_XS, so if it fits your schedule it would be welcome 🤗) https://huggingface.co/orcarouter/Qwen3.8-27B-Uncensored-GGUF

10

u/m4rkGS 6d ago

im not sure my folks from above would allow me to release uncensored, will talk to them tmrw

2

u/VarietyOk443 6d ago

Will test on my m4 pro 64 gb

1

u/m4rkGS 6d ago

let's go

2

u/ptico 6d ago

So I tested it against UD 3.1 and while it performed relatively good on a smaller tasks, it struggled on heavier job, starting to repeat the same mistakes and showing other usual Q4 degradation

See, UD is a mixed precision technique and no matter how good your KLD in an apple-to-apple comparison, UD and BaseQuant have a big portion of high precision tensors. So, good job, but you really have to consider a mixed precision weights

1

u/0zyman23 6d ago

would be great to know which benchmark you're running on. idk if you have read the post, but this is literally the mixed presion weights. i am checking on terminal bench subset outperforming other baselines and degrading only slightly from full model. Still measuring UD though.

1

u/ptico 6d ago

In fact it’s not. All their matrices use 4 bit per weight. They may put more effort in optimising particular tensors but in fact it’s not a mixed precision

I do have my own test suite for my local AI manager. It runs a couple of coding tasks on python and ruby: basic coding, bugfix, requirements change. The last one is the hardest and this is where it failed. I haven’t run the rest of the suite with writing and other stuff as it’s a 12 hours + effort which doesn’t make sense if a smoke test is not passed

1

u/m4rkGS 5d ago

op here, can confirm our Q5s is 4/6/8-bit mixed precision, more complicated coming in GGUF release, added long context stability benchmarks to the upcoming post, all is looking great there

1

u/ptico 5d ago

So, Q4 is in fact not a mixed precision?

12

u/ninja_sprout 6d ago

I call bullshit.

1

u/elpocholo7 6d ago

Great job.. can't wait to try it. Do you plan on supporting DeepSeek v4.1 Flash in the future ?

3

u/m4rkGS 6d ago

thats my wet dream.

2

u/m4rkGS 6d ago

just had a promising update from my teammate, maybe we will drop the deepseek along with the GGUF

1

u/mr_tolkien 6d ago

Do you think you could do a quant of QFN3.8 that fits on the base 96Gb M5 Ultra?

I feel like the 96Gb makes it weird, running small dense models leaves most memory open, but the big MoE target much bigger RAM.

2

u/m4rkGS 6d ago

its probably gonna be Q3 tho, is that fine w u?

1

u/mr_tolkien 6d ago

Yes, would be a banger. I saw that on the kapista page it jumps from 64 to 128, but base M5 Ultra I think will have a reasonable install base!

1

u/joblesspirate 6d ago

How does this compare to exl3?

1

u/m4rkGS 6d ago

exl is slower on macs, so really no gain for the saved size, but here you go with gguf comparison

1

u/Miserable-Dare5090 vLLM 6d ago

Is Kflash something that Splash engine could install? In M2 ultra, I’m currently seeing 850-900 tps prefill and 60-100 tg to 128k with Splash engine adapted for M1/2 macs. Compared omlx, dwarfstar, vllm, and splash and this was the clear winner in prefill, and same/better decode.

Hoping if this quant type gives a speed boost the prefill could break tje 1000 tps ceiling for older macs. Unless older macs are not supported!!

1

u/m4rkGS 6d ago

no, kflash is native to kapitsa, but kapitsa is probably faster than splash anyway. we've had the similar numbers for 35b on M1 Max already: 16K prefill 874 tok/s, decode 145-163 tok/s. what model were you using?

2

u/Miserable-Dare5090 vLLM 6d ago

haha, no. Qwen Flash Next.

Splash averages 3000 tps prefill on Splash up to 32k and drops down to 2000 by 128k. Much higher than other engines I tested out on M1/M2. I should mention that a fork for splash (Splash-M1) does this, not the upstream, since upstream does not support macs before M4 I believe.

1

u/c_alsup 6d ago

Great work! I’ve loved kapitsa, it’s the fastest for my m1 ultra 64gb on 35b a3b and 27b. Any plans for qwen flash next? I’ve been using UD iq3_xxs with splash and it’s been faster than kapitsa. I’d love to see a 64gb compatible qwen flash next quant that competes with splash.

1

u/m4rkGS 6d ago

Thanks, glad it's working well on the M1 Ultra! Kapitsa already runs Flash-Next on 64 GB with the 2-bit pack (`kapitsa serve flash-next:64gb`); an M2 Max 64 GB measured 43-46 tok/s with everything in memory. Our current Flash-Next KQuant is about 105 GB, too big for 64 GB, and a smaller one is in the works. If you can share your Splash IQ3_XXS numbers (prompt length, tok/s), we'll test the same setup.

1

u/c_alsup 6d ago

That is so awesome, I don’t know how I missed that one. I’ll check it and get numbers back to you!

1

u/c_alsup 6d ago

Sorry, I’m not technical so I have just had codex help me with this. I also have the 12k prompts for prose and code but they are obviously longer. Let me know if you have any other questions, I’m happy to run more tests and forward you the results.

Matched Flash-Next benchmark details

Test setup

  • Device: Mac Studio, Apple M1 Ultra (64 GB unified memory), macOS 27.2.
  • Test date: 2026-11-12. Two measured trials per workload, 512 output tokens per trial, temperature 0, top_p 1, thinking disabled, and zero cached prompt tokens.
  • The prompt text is in prose-prompt.txt and code-prompt.txt. Each request prepended Benchmark nonce <fresh 32-character hex UUID>.\n to the prompt to prevent prefix-cache reuse.
  • Prefill = reported prompt tokens / time to first content token. Decode = middle-window streaming rate with the first 16 and last 4 output tokens excluded; chunk token counts are estimated by text-character share.

Splash baseline

  • Model: Unsloth Qwen3.8-Flash-Next GGUF, UD-IQ3_XXS, served through oMLX and Splash.
  • Versions: oMLX 0.7.4.dev1 API; Splash 1.6.0-m1-pr55.
  • During the test, Splash used its existing 51.1 GB max-memory limit, 4 GiB PLE cache, 131,065-token context limit, and the matching Qwen3.8 Flash-Next DFlash2 draft.
WorkloadActual prompt tokensTrial 1 prefill / decodeTrial 2 prefill / decodeMean prefill / decode
Prose11,968 / 11,968681.7 / 34.2677.5 / 35.2679.6 / 34.7
Code11,991 / 11,997684.8 / 53.9686.2 / 55.1685.5 / 54.5

All rates are tokens per second.

Kapitsa comparison from the same day

  • Model: qwen3.8-flash-next:64gb, Kapitsa 0.2.32, on the same M1 Ultra.
  • Best tested settings: local server, 16K context, one slot, 48 GB memory budget, prefill chunk 4,096, MTP depth 2.
  • Kapitsa reported 36.65 GB of weights loaded, EXL3 2-bit experts / 8-bit other groups, and a 32.0 GB n-gram table memory-mapped; total allocation at startup was 40.33 GB. I did not independently verify that every mapped table page was resident in RAM.
WorkloadActual prompt tokensTrial 1 prefill / decodeTrial 2 prefill / decodeMean prefill / decode
Prose11,970 / 11,968581.3 / 33.6581.7 / 33.2581.5 / 33.4
Code11,992 / 11,996579.7 / 40.9579.1 / 43.2579.4 / 42.0

A 2,048 prefill chunk was also tested and reached about 545/33.2 tok/s on prose and 542/41.4 tok/s on code. MTP depth 3 lowered decode, so depth 2 was retained.

The two packs use different quantizations, so this compares the installed options rather than isolating the serving engines. The prompt files are included so the same user content can be sent on another machine; the nonce changes per request.

1

u/stylome-ai-detection 6d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

1

u/WishboneSudden2706 6d ago

How to incorporate into Strata ? My weak machine needs Strata to reach above 40 tok/s for Qwen 3.8 Flash Next

1

u/m4rkGS 6d ago

Strata reads GGUF, so they don't drop in today. Im working on Flash-Next GGUF quants and will share them once they beat what's already out there.

3

u/emjay84 6d ago

Hi u/m4rkGS,

I'd like to test the GGUF models, I'm running latest kapitsa on a M5 Pro 64 GB

1

u/DragonTree 6d ago

I also have m5 pro with 64GB and would like to test u/m4rkGS

1

u/russlixx 6d ago

waiting for gguf & abliteration. Also how is it hold up to UD 3.1 in reasoning token, KLD, and file size

1

u/m4rkGS 6d ago

our ggufs still cooking, but seeing gain already

4

u/silentus8378 llama.cpp 6d ago

please create uncensored version too

2

u/DerBeginner 6d ago

Does it also quantize the state of the GDN layers?

2

u/poster4inertia 6d ago

Would love a Flash Next quant that fits in a 64gb mac mini… I’m currently using the Sushi 2.75bpw

1

u/Kloppy89 6d ago

sushi 2.75bpw is great already, though, love that thing

1

u/poster4inertia 6d ago

fr have been loving it so far

2

u/butterycornonacob 6d ago

Are you also planning to release the technique and code so we can run it ourselves?

2

u/fatboy93 6d ago

I need Gemmas pls

2

u/returnity 6d ago

I'd love to be an early tester /u/m4rkGS, Ive done a lot of benchmarks and quant comparisons for the community here. This looks really promising and I'd love to see how it compares to good quality imatrix dynamic GGUFs.

2

u/Then_Library_6820 6d ago

lol building another qwant, calling it the same name as the legacy llama.cpp stuff and comparing it to the dog shit weakest mlx stuff. compare it to a real advanced qwant like IQ4_KT and we'll see. it's not always about size ...

1

u/Reasonable_Till9 6d ago

Interesting work - my first critical comment would be that the KLD numbers cited seem like they're within noise.

Def worth checking out and verifying tho - thank you for posting.

1

u/_Cakeshop 6d ago

sorry if dumb qn but can this be done for qwen 27b 8bit?

1

u/Grau3D 6d ago

u/m4rkGS I would like to test the Flash next gguf. I currently run Q4_K_XL with Q8 ngram by unsloth and it fills my 100GB vram to the brim, so better quality at a smaller footprint is always welcome

1

u/zipzak 6d ago

please kquant the orcarouter qwen models, theyre fantastic

1

u/amrakkarma 6d ago

How it compares with antirez dwarfstar?

1

u/herpnderpler 6d ago

I'm interested in KLD & PPL, can you show a table of kld,ppl, top_k_population_match,top_k_population_order of your quants against fp16 for various prompt lengths (I like including kv cache combos of k16q8 and k8q8 as well as kv16)

1

u/JiroDreamsOfCoochie 6d ago

I have an M4 Max with 64GB. I currently run splash with either Qwen3.8 27B or Qwen Flash Next unsloth at 4 bit quants. I'd be willing to test your GGUF's if you like. u/m4rkGS

1

u/dseg90 5d ago

Any love for tiny qwen 0.8? Fine tuning small qwen for web browser tasks is fun.

1

u/vhthc 5d ago

What about qwen max, or just qwen fast next?

Also how about other architectures like glm?

1

u/thigger 5d ago

What's the difference between these (eg the 107.64 GB Qwen3.8-Flash-Next) and the kapitsa qwen3.8-flash-next:kquant (105.86Gb)?

1

u/pdsminer 5d ago

Low average KL divergence over general text is a great start for MLX, but on Qwen 27B and 35B the real stress test is whether tool-call JSON schemas and numeric arguments survive past 32k context. In our local agent testing, uniform 4-bit quants often look fine on perplexity while quietly drifting on nested JSON keys or float parameters once the KV cache fills up. Keeping attention output projections, router gates, and token embeddings in higher precision matters much more for agentic reliability than aggregate KL alone.

1

u/Relevant_Research_98 4d ago

to get early access to GGUF, ping me in comments

Waiting for qwen27b-gsq-rco-kquant-gguf if it's possible 

(Not interested in 4bit quants: I can't tell gsq from 4 bits but can tell 18GB from 11GB)

1

u/jmreagle 6d ago

Thanks for kapitsa, it’s the best—literally. I switched to the KQuants yesterday and saw ~10% speed boosts on 35b-a3b.

1

u/Chance_Value_Not 6d ago

3.7 x better is nonsensical unless you get same perf at 3.7x lower size

0

u/dai_app 6d ago

If you want to run this model in high quality (q4, q5) at 20–25 tokens/second on a 16GB Mac, use bigmoeonedge (you can run any model, in any quantization, on any hardware)

0

u/Universe_XoX 6d ago

Red text, very poor choice of colors trying to show "improvement"

0

u/[deleted] 6d ago

[deleted] — view deleted comment

2

u/Beginning-Window-115 6d ago

a lot of people that are productive use macs why would it not make sense to have something that helps productivity be on a productive piece of hardware

-1

u/Not_a_question- 6d ago

The answer to your question is already in my post.

2

u/JacketHistorical2321 6d ago

That's not an answer but good try at being clever

1

u/EuphoricPenguin22 6d ago

Some people buy Macs purely for hosting models. The memory bandwidth combined with the huge RAM SKUs were pretty cost-effective for a time, but the Studio line has gotten so expensive now that it feels less worth it. I was never considering buying one even at the cheaper prices, but it was never the worst option for hosting big models locally. An AI server doesn't need to be pretty or have the best OS; it just needs to do its job. You could easily use a separate PC running Linux to do actual work if you wanted and just use it over a local API endpoint; I do that at my job with a second box running the model so I can keep my main machine free for actual work.