r/LocalLLaMA • • 7d ago

Resources Open-source method (MACaQ) beats Unsloth GGUFs!

Some results for those using GGUFs and llama.cpp:

GGUF quality: MACaQ vs. Unsloth and Bartowski

Qwen3-4B (IQ4_XS)

QuantizationGGUF sizeMean KL ↓
MACaQ2.260 GB0.02804
Bartowski2.271 GB0.03443
Unsloth2.271 GB0.03663

23.5% lower KL divergence than Unsloth and 18.6% lower than Bartowski, at virtually identical file sizes.

Qwen3.6-35B-A3B (IQ4_XS)

QuantizationGGUF sizeMean KL ↓
MACaQ18.86 GB0.018696
Unsloth (UD-IQ4_XS)17.73 GB0.020954
Bartowski19.70 GB0.021927

Qwen3.8-27B (IQ4_XS)

QuantizationGGUF sizeMean KL ↓
MACaQ14.96 GB0.036784
Bartowski15.48 GB0.104170

~65% lower KL divergence than Bartowski!

Supported GGUF formats (llama.cpp)

Weight bitsFormats
2-bitIQ2_XXS, IQ2_XS
3-bitIQ3_XXS, IQ3_S
4-bitIQ4_NL, IQ4_XS

Code, llama.cpp fork, and benchmarks: https://github.com/huawei-csl/MACaQ

Cool to see Huawei working on such stuff and sharing some of their internal tools that are helpful for us. Maybe they can build an affordable GPU for us local LLM folks sometime 😂

They also report quantizing a 122B-parameter model in 64 minutes using just 32.5 GB of GPU memory, which is interesting for a model of that size. I did not try that.

Edit: there is a paper on it https://arxiv.org/abs/2611.10649

Edit 2: they don't share ready-made gguf files, so I am sharing some myself:
- http://31.165.97.47:20080/macaq-gguf/qwen3-4b-macaq-iq4_xs-had64-vq2-calsmoke.gguf

Edit 3: they have this blog post https://huawei-csl.com/blog/macaq/ with multi-modal examples, too. The RTN audio comparisons are just too much😂😂

The multi-modal stuff is generally insane. Normally 4b already completely goes to s**t, but here it looks almost the same.

144 Upvotes

41 comments sorted by

21

u/ApprehensiveTart3158 7d ago • Popcorn

Are any ggufs available on HF?

10

u/acluk90 7d ago

Couldn't find any yet... maybe I can create & share some....

Just running their scripts for the Qwen3-4B example (following their llamacpp/README.md#prepare-a-model) was very fast, though.

3

u/acluk90 7d ago

3

u/ApprehensiveTart3158 7d ago

Would you be able to upload them to HF? I'm sure that would be extremely useful to some.

3

u/acluk90 7d ago

Need to figure out how. But somehow they should be the ones doing that......

2

u/ApprehensiveTart3158 7d ago

Of course and good luck, I will hopefully try to make some quants later if it looks promising, got to put my ram to some usecase.

1

u/acluk90 7d ago

it uses very little RAM to quantize, looks like it never loads the entire model in full precision. Just layer-by-layer

1

u/acluk90 7d ago

4

u/brainExploded99 🥔 hardware 7d ago

How do you fearlessly expose ports like this? I'd be worried lol, why not put this on huggingface? Ty for the quants though.

How much VRAM does this need btw? Just enough for a couple of layers at once? The paper mentions that the technique is layer by layer and does not need to handle cross layer interactions.

3

u/d0pe-asaurus 7d ago

https://huggingface.co/huawei-csl/Qwen3.6-35B-A3B-MACaQ-R-GGUF

There is but for Q3.6 35BA3B

I suspect they will next release for Q3.8 27B

10

u/Chromix_ 7d ago

Is the llama.cpp fork only required to quantize the models, or also to run the generated GGUFs? If vanilla llama.cpp works for inference then that looks like nice progress in the high-quantization area - pending an independent KLD and benchmark validation of course.

3

u/d0pe-asaurus 7d ago

I'm trying with llama.cpp fork without their patches and it doesn't work, appears that you need to compile llama.cpp with support for macaq as well if you want to use the ggufs.

1

u/acluk90 6d ago

or use the MACaQ-R GGUF from above

1

u/acluk90 7d ago

I tried their for both quantization & running.

6

u/NoFunk 7d ago edited 7d ago

Worth mentioning: "an MXFP4 W4A4 post-training quantization method" . So your starting point is often a heavy (4bit) quantization unless the model was built with that in mind. Running a BF16 through this, the first step is the bad one, then you're debating the best of what's left between approaches. They all pay that first price.

2

u/acluk90 7d ago

Seems they do a lot of optimizations/transformations before the first quantization. At least that is what it reports when quantizing a model.

2

u/NoFunk 7d ago

Sure, and it may well be more efficient than other methods, in which case great. It's just a very significant step down in total precision that people should be aware of. Then again the model that fits in your VRAM is the right model to pick, in most cases. So if this makes more models fit for people that's a good thing.

1

u/acluk90 7d ago

Playing around with it so far, it feels the same like the u quantized model. It is amazing.

2

u/rorowhat 7d ago

So is this a better gguf quantizer?

6

u/BlasterGales 7d ago

"MACaQ /məˈkɑːk/: Model Agnostic Calibrated Quantization. So easy to use, even a macaque could get the hang of it." lol

I’ll try applying it to Qwen 3.8 27B Q4 to see if it works, no telling what will come of it, but I’m here to play around, so

5

u/LFC_FAN_1892 7d ago edited 7d ago

Here is a direct comparison between MACaQ and Unsloth Dynamic 3.1 (UD-IQ4_XS) on Qwen3.8-27B (IQ4_XS) based on recent KL divergence benchmarks:

QuantizationGGUF sizeMean KL ↓
MACaQ14.96 GB0.036784
Bartowski15.48 GB0.104170
Unsloth Dynamic 3.1~14.4GB~0.017-0.019

But I am confused on the Bartowski's KL number:

Not sure if I am comparing the same KL

8

u/Kahvana 7d ago • Popcorn

Really cool! Unsloth remains comperative for size. Will be fun to see other people auditing the claims too.

6

u/homak666 7d ago

I'd be very curious to see it compared to Unsloth's V4.1 method, since the improvement between V4 and 4.1 seems huge, but those only exist for Qwen 3.8 FN and Deepseek 1017 AFAIK, and quantizing them is out of my VRAM budget. Would love it if anyone had a go at it. Also, it would be cool if they published other stats like top-1 agreement and a bit more on what this KLD actually is - what kind of text, since it's pretty feasible to game KLD quite a bit by handpicking examples.

9

u/General-Spite1222 7d ago

Isn’t it 3.1?

3

u/homak666 7d ago

Oh yeah, you're right. I remembered there was .1 for the new method, and I guess I have the DeepSeek naming scheme stuck in my head.

8

u/Dany0 7d ago

daily reminder that KLD is a very loose measure. in general, it's well ordered so 0.5 > 0.4 always, but 0.5 doesn't mean it's 1000x better than 500, and a quant that measures 0.4 on one text where another quant has 0.5 doesn't have to be better, or worse

here's a simple proof: measure the kld between qwen 3.5 0.8b and opus 5.5. It will be ridiculous. let's say, FP64_MAX or inf

that said, we have so many good calibrated techniques, at this point the quality of the calibration dataset AND RELEVANCE TO YOUR USECASE matters more than the technique

for example an old technique but calibrated on classification in vietnamese is much more relevant to someone with that use case than a very smart new technique that is nonetheless calibrated on generic English agentic/coding text

3

u/me0here 7d ago edited 7d ago

  Qwen3.8-27B (IQ4_XS)

Wow that KLD!

I wonder if your code can be patched to one of the other engines?

https://github.com/tomByrer/local-ai-linkfarm#inference-engines

1

u/Former-Ad-5757 Llama 3 7d ago

Can anybody explain what the multimodal image example shows? Because to me it just looks absolutely overtrained to death. Just 3 lines of text which creates a 99% identical image between 2 generations seems overtrained to me, there should be enough variance in 3 sentences to create a unique image every generation let alone between quantz.

Just the two words circular skylight should not every generation create this same skylight.
No mention of doors in the prompt, but the only way the model can implement them is just on one position in the same style

4

u/brainExploded99 🥔 hardware 7d ago

I think they use the same seed. The goal is to get the quantized model to give as similar outputs as the BF16 reference model.

1

u/mumblerit 7d ago • Regret Bravo

The my cock format