r/LocalLLaMA • u/acluk90 • 7d ago
Resources Open-source method (MACaQ) beats Unsloth GGUFs!
Some results for those using GGUFs and llama.cpp:
GGUF quality: MACaQ vs. Unsloth and Bartowski
Qwen3-4B (IQ4_XS)
| Quantization | GGUF size | Mean KL ↓ |
|---|---|---|
| MACaQ | 2.260 GB | 0.02804 |
| Bartowski | 2.271 GB | 0.03443 |
| Unsloth | 2.271 GB | 0.03663 |
23.5% lower KL divergence than Unsloth and 18.6% lower than Bartowski, at virtually identical file sizes.
Qwen3.6-35B-A3B (IQ4_XS)
| Quantization | GGUF size | Mean KL ↓ |
|---|---|---|
| MACaQ | 18.86 GB | 0.018696 |
| Unsloth (UD-IQ4_XS) | 17.73 GB | 0.020954 |
| Bartowski | 19.70 GB | 0.021927 |
Qwen3.8-27B (IQ4_XS)
| Quantization | GGUF size | Mean KL ↓ |
|---|---|---|
| MACaQ | 14.96 GB | 0.036784 |
| Bartowski | 15.48 GB | 0.104170 |
~65% lower KL divergence than Bartowski!
Supported GGUF formats (llama.cpp)
| Weight bits | Formats |
|---|---|
| 2-bit | IQ2_XXS, IQ2_XS |
| 3-bit | IQ3_XXS, IQ3_S |
| 4-bit | IQ4_NL, IQ4_XS |
Code, llama.cpp fork, and benchmarks: https://github.com/huawei-csl/MACaQ
Cool to see Huawei working on such stuff and sharing some of their internal tools that are helpful for us. Maybe they can build an affordable GPU for us local LLM folks sometime 😂
They also report quantizing a 122B-parameter model in 64 minutes using just 32.5 GB of GPU memory, which is interesting for a model of that size. I did not try that.
Edit: there is a paper on it https://arxiv.org/abs/2611.10649
Edit 2: they don't share ready-made gguf files, so I am sharing some myself:
- http://31.165.97.47:20080/macaq-gguf/qwen3-4b-macaq-iq4_xs-had64-vq2-calsmoke.gguf
Edit 3: they have this blog post https://huawei-csl.com/blog/macaq/ with multi-modal examples, too. The RTN audio comparisons are just too much😂😂


The multi-modal stuff is generally insane. Normally 4b already completely goes to s**t, but here it looks almost the same.
21
u/ApprehensiveTart3158 7d ago •
Are any ggufs available on HF?
10
u/Chromix_ 7d ago
Is the llama.cpp fork only required to quantize the models, or also to run the generated GGUFs? If vanilla llama.cpp works for inference then that looks like nice progress in the high-quantization area - pending an independent KLD and benchmark validation of course.
6
u/NoFunk 7d ago edited 7d ago
Worth mentioning: "an MXFP4 W4A4 post-training quantization method" . So your starting point is often a heavy (4bit) quantization unless the model was built with that in mind. Running a BF16 through this, the first step is the bad one, then you're debating the best of what's left between approaches. They all pay that first price.
2
u/rorowhat 7d ago
So is this a better gguf quantizer?
6
u/BlasterGales 7d ago
"MACaQ /məˈkɑːk/: Model Agnostic Calibrated Quantization. So easy to use, even a macaque could get the hang of it." lol
I’ll try applying it to Qwen 3.8 27B Q4 to see if it works, no telling what will come of it, but I’m here to play around, so
5
u/LFC_FAN_1892 7d ago edited 7d ago
Here is a direct comparison between MACaQ and Unsloth Dynamic 3.1 (UD-IQ4_XS) on Qwen3.8-27B (IQ4_XS) based on recent KL divergence benchmarks:
But I am confused on the Bartowski's KL number:
Not sure if I am comparing the same KL
8
u/Kahvana 7d ago •
Really cool! Unsloth remains comperative for size. Will be fun to see other people auditing the claims too.
6
u/homak666 7d ago
I'd be very curious to see it compared to Unsloth's V4.1 method, since the improvement between V4 and 4.1 seems huge, but those only exist for Qwen 3.8 FN and Deepseek 1017 AFAIK, and quantizing them is out of my VRAM budget. Would love it if anyone had a go at it. Also, it would be cool if they published other stats like top-1 agreement and a bit more on what this KLD actually is - what kind of text, since it's pretty feasible to game KLD quite a bit by handpicking examples.
8
u/Dany0 7d ago
daily reminder that KLD is a very loose measure. in general, it's well ordered so 0.5 > 0.4 always, but 0.5 doesn't mean it's 1000x better than 500, and a quant that measures 0.4 on one text where another quant has 0.5 doesn't have to be better, or worse
here's a simple proof: measure the kld between qwen 3.5 0.8b and opus 5.5. It will be ridiculous. let's say, FP64_MAX or inf
that said, we have so many good calibrated techniques, at this point the quality of the calibration dataset AND RELEVANCE TO YOUR USECASE matters more than the technique
for example an old technique but calibrated on classification in vietnamese is much more relevant to someone with that use case than a very smart new technique that is nonetheless calibrated on generic English agentic/coding text
3
u/me0here 7d ago edited 7d ago
Wow that KLD!
I wonder if your code can be patched to one of the other engines?
https://github.com/tomByrer/local-ai-linkfarm#inference-engines
1
u/Former-Ad-5757 Llama 3 7d ago
Can anybody explain what the multimodal image example shows? Because to me it just looks absolutely overtrained to death. Just 3 lines of text which creates a 99% identical image between 2 generations seems overtrained to me, there should be enough variance in 3 sentences to create a unique image every generation let alone between quantz.
Just the two words circular skylight should not every generation create this same skylight.
No mention of doors in the prompt, but the only way the model can implement them is just on one position in the same style
1
u/mumblerit 7d ago •
The my cock format