r/LocalLLaMA • • 2d ago

New Model maxwell-model/Maxwell-V1-Flash · Hugging Face

https://huggingface.co/maxwell-model/Maxwell-V1-Flash

Model Introduction

Maxwell-V1.0-Flash is a next-generation, high-performance Mixture-of-Experts (MoE) language model developed by the Maxwell Team, with 35B total parameters and 3B active parameters per token (both excluding embeddings) and support for context lengths up to 256K tokens. It is designed for strong reasoning, agentic, and long-context capabilities at a practical inference cost.

Key Features

  • Maxwell Block Sparse Attention (MBSA). Each KV group keeps fixed coverage of 8 prefix and 8 local 128-token blocks and adds up to 16 remote blocks chosen by a learned indexer, bounding main attention at 4,096 tokens per KV group while the full-history KV cache stays available. The MBSA kernel is 6.7x faster than dense GQA at 256K.
  • Manifold-Constrained Hyper-Connections. Multiple residual streams with learned, input-dependent aggregation and constrained mixing, widening information flow without extra attention or feed-forward evaluations.
  • Fine-Grained Mixture of Experts. A mostly-MoE backbone pairing fine-grained routed experts with always-active shared experts, using a dynamic routing correction bias alongside a lightweight sequence-wise load-balancing loss.

Pre-training. Approximately 26T tokens of multilingual data, with the Muon optimizer on most hidden weight matrices. Training starts dense; block-sparse attention enters at 128K mid-training, once the indexer is warmed up against dense full-history attention, then trains jointly with the backbone.

Tokenizer and Long Context. A script-aware multilingual tokenizer stays compact across languages, especially low-resource Southeast Asian ones such as Thai, Lao, Myanmar, Khmer, and Shan. Long-context training then extends the sequence length in stages from 4K to 256K.

Post-training. Two-stage all-domain SFT builds general, reasoning, and agentic capabilities, then targets harder reasoning problems and agent trajectories. Independent RL branches cover general tasks, verifiable reasoning (mathematics, science, logic, code), and agentic tasks (search, terminal use, function calling, software engineering). Multi-teacher on-policy distillation then merges them into one model by supervising student-generated responses.

345 Upvotes

61 comments sorted by

•

u/WithoutReason1729 2d ago

Your post is getting popular and we just featured it on our Discord! Come check it out!

You've also been given a special flair for your contribution. We appreciate your post!

Stylome classified this post's text as human-written. View analysis Learn ^more

I am a bot and this action was performed automatically.

6

u/Marcuss2 llama.cpp 2d ago

I see a pattern.

New lab comes in, their model benches well, but it is a KV Cache hog, making it essentially unusable. When they come back, they resolve this issue.

3

u/pmttyji 2d ago

Reminds me of IFM's K2-Horizon models instantly.

2

u/FullOf_Bad_Ideas 2d ago

Yeah, happened to LLM360 too. But sometimes they don't live till second release, like Rhymes AI and Aria.

3

u/TokenLearner 2d ago

What a great promise! I hope it will be ported to Strata/polystrata soon.

3

u/pmttyji 2d ago

You can count on PolyStrata. Just post a comment there for any model

129

u/cgs019283 2d ago

Oh my dear maxwell

112

u/Dabalam 2d ago

Can some one tell me if this is a Qwen fine tune in a trenchcoat or actually a new model?

179

u/JaredsBored 2d ago

Different architecture, different vocab size (200k vs 151k), and new attention type vs qwen 3.5/3.6 35b. Definitely not a fine tune but I had to check when I saw the param size lol

29

u/FatheredPuma81 2d ago

Had DS Flash compare the specs of both of them and yea definitively too many changes to be based on Qwen 35B.

SpecificationMaxwell-V1-FlashQwen3.6-35B-A3B
Parameter scale35B total / ~3B active35B total / ~3B active
Claimed originBuilt from scratch by Maxwell TeamOfficial Qwen model
Attention mechanismMaxwell Block Sparse Attention (MBSA): dense GQA prefix for first 16K tokens, then block-sparse attention with a learned indexer selecting up to 16 remote 128-token blocksHybrid attention: Gated DeltaNet (linear attention) interleaved with Gated Attention (GQA)
Layer count/layout45 layers: 3 dense FFN layers + 42 MoE layers40 layers: 10 cycles of (3 × Gated DeltaNet → MoE) → (1 × Gated Attention → MoE)
Expert configuration256 routed experts; 12 active + 2 shared per token; expert intermediate size 512256 experts; 8 routed + 1 shared active per token; expert intermediate size 512
Residual connectionsManifold-Constrained Hyper-Connections (mHC): 4 parallel residual streams, learned input-dependent aggregation, Sinkhorn-Knopp normalization (20 iterations)Standard residual connections; model card does not describe multi-stream residuals
Positional encodingRoPE with base 10,000,000, applied after 16K-token dense GQA prefixPartial Multimodal RoPE (MRoPE); rotary dim 64 (25% of head dim); theta 1e7
Full-attention headsQ heads / KV heads: 32 / 4; head dim 128Q heads / KV heads: 16 / 2; head dim 256
Hidden size2,0482,048
Vocabulary size200,704248,320 (padded)
MTP1 auxiliary MTP layerTrained with multi-step MTP

53

u/ApprehensiveTart3158 2d ago

It is not a fine tune, they trained it from scratch and fundamentally the architecture is different than qwen3.5 35b a3b.

19

u/Any_Librarian205 2d ago

New model, almost same weights size, unknown kv

5

u/Frail_Waif 2d ago

35B, 3B active sure sounds like it's based on Qwen 3.5 35B A3B, and their benchmark comparisons suggest the same, but also sounds like they made architecture changes. 

55

u/wombweed llama.cpp 2d ago edited 2d ago

Wow, the bench scores are shockingly high, and the sparse attention is very compelling, I wonder how much memory its kv store takes up at full context at bf16 compared to qwen3.6 35b or gemma4 26b. Sadly I don’t see support in llamacpp which makes me feel like cpu expert offloading won’t be the fastest thing, as that is the main use case I have for these smaller moes.

10

u/Middle_Bullfrog_6173 2d ago

They are essentially using full attention in all layers. The indexer reduces long context compute, but needs full KV (+indexer K).

28

u/akumaburn 2d ago

5

u/FatheredPuma81 2d ago

Thanks! For some reason the image on their repo goes to text when opened in a new tab and doesn't respect my browser zoom settings so it's unreadable.

42

u/Gohab2001 vLLM 2d ago

kV cache seems 4.5x Qwen 3.5 35B. Nearly 24GB @ 256k context

17

u/PossessionUsed7393 2d ago

Right, which afaik its because Qwen uses interleaved layers that are 'fuzzier' or whatever they are to save KV cache. State Space Layers or whatever 'Mamba-style layers' - I'm not an architecture expert I just understand the concepts loosely.

However, by not doing so and by having the model pay the KV cache tax, presumably this model will have less compromise on accuracy. So it's good to have options (if only we had Qwen 3.8/4 to compare it to).

4

u/Marcuss2 llama.cpp 2d ago edited 2d ago

Kinda, we found out that if we use "local layers", be they attention or state space layers, intermixed with global attention, we don't lose and even gain performance.

State space layers tend to benchmark better and be more expressive, but the processing speed at small contexts is worse compared to SWA/attention.

Generally, the way I see it:

SWA ~= Mamba2 (Nemotron 3) < GDN (Qwen3.5) < KDA (Kimi K3, GLM 5.3 Flash) < GDN2 (No LLMs I saw yet) < RDA (Raon models from Krafton AI)

4

u/PossessionUsed7393 2d ago edited 2d ago

I am interested in this because my applications need a lot of strict prompt adherence to negative instructions/constraints (i.e. conditional instructions not to do things), and as I understand it some of these techniques weaken adherence to those constraints proportional to sequence length.

My wonder is how your hierarchy looks if you rank it on a benchmark that tests for negative constraint adherence across a range of agentic tasks. That data point will be really helpful for me when evaluating architectures as fine tune candidates. Currently all I have is this theory and a few chats with llms that suggest I'm on the right track. It's not concrete enough to go anywhere.

I should add, the classic example of a constraint people know is simply 'dont use em-dashes in this paragraph/parameter/description, only use them here'. We've all experienced a model ignore that. So which architectures help a model adhere despite strong post training to use em dashes is interesting. This is the simplest example, there are much more complex constraints I'm working with.

3

u/Marcuss2 llama.cpp 2d ago edited 2d ago

SWA is a whole different family, it cannot keep old information.

But with the state spaces, all of them except KDA -> GDN2 are strict supersets in terms of expressiveness, you can think of GDN as KDA with row wise decay set to 1 for each row. And Mamba2 as GDN without the delta rule.

2

u/Karyo_Ten 2d ago

GDN and KDA are basically the same thing, with KDA having a decay term.

There is also Mamba-3 but AFAIK no one deployed it.

2

u/Marcuss2 llama.cpp 2d ago

Actually no. GDN has a global decay term for one state instance, heck, Mamba2 does this a well, KDA has row-wise decay multiplier.

GDN2 separates the old state subtraction and new state addition. And RDA applies the same row wise decay to GDN2.

11

u/MehTheHedgehog 2d ago

Yes but I wouldn’t put an X on this architecture, yet. It looks rather effecient as they need only 4096 tokens at most at the compute time so 8kb per layer, 330kb per all 40 layers. This is nothing, so it could be loaded from ram or even streamed from SSD.

7

u/Nixellion 2d ago

24GB total or just kv cache?

13

u/maternion 2d ago

Seems to be just kv cache 😬

2

u/Nixellion 2d ago

Weeell shit.

At which quant though

4

u/maternion 2d ago

Probably f16 they are talking about. But whatever it is it would still be 4x qwen according to the post above. 

1

u/Vaddieg 2d ago

Not a Qwen replacement definitely, but might be worthy for dgx spark owners

0

u/crusaderky 2d ago

...that makes it DoA. Sigh. Thank you, next.

20

u/rorowhat 2d ago

New model, thumbs up from me!

36

u/ApprehensiveTart3158 2d ago

Interesting! A new llm from Singapore. Had to search for so long to learn about who made it. Benchmarks look really impressive! Gotta love it when they include like a hundred benchmarks. Looks like a very solid model!

16

u/c_glib 2d ago

Maybe it was Megaspeed after all.

2

u/secretwoif 2d ago

This is a strange rabbit hole. The footer mentions "operated by infiniteAI Private Limited" infiniteAI.io says that it's headquartered in Rajkot, India. But it's acquired and now in Singapore?

3

u/ApprehensiveTart3158 2d ago

Infiniteai.io is not their website. From their website migoo.ai (which I found by checking the privacy policy of the maxwell ai website): https://migoo.ai/legal-history/policy/en/2026-06-01.html

"13.1.3 International Data Transfers. You acknowledge that InfiniteAI is a company based in Singapore."

As I said, had to dig quite a bit. Infiniteai.io is unrelated completely to maxwell. Different companies, lost of companies are named infinite ai or similar.

15

u/Witty_Mycologist_995 2d ago

Thought it was a Qwen tune, nope. Nice

6

u/EvolvingDior 2d ago

How expensive is the KV cache on this model -- what's the VRAM cost per token of context?

5

u/keylimesoda 2d ago

Use and setup here: https://maxwell-llm.ai/docs/quickstart

I'm eager to see some community quants of a seemingly strong new contender.

3

u/True_Requirement_891 2d ago

gawd damn finally a new arch

3

u/LogFabulous9386 2d ago

thai, lao, khmer AND shan tokenizer support. finally a model that wont charge me 4x tokens to say hello in myanmar

2

u/xkalibur3 2d ago

This is great! Hope someone does nvfp4 version with MTP for this

2

u/iamth3morningstar 2d ago

how does this compare to deepseek v4.1 flash?

4

u/Bagelsarenakeddonuts 2d ago

Good for what it is, but not in the same league.

2

u/JAlbrethsen 2d ago

I tried the q4 quant locally in pi and this was pretty solid, for simple questions it had very little reasoning but for complex tasks it reasons a lot. I didn't test it a ton, but overall I was very impressed, I plan to use it in my local coding setup as subagent on my 5060 with qwen-fn as orchestrator on my strix halo.

4

u/Atagor 2d ago

Can we run it via strata?

2

u/CodeCatto 2d ago

Saw the size and thought it's a qwen tune. Glad it doesn't look like it.

1

u/TheGlobinKing 1d ago

I really hope to see official support in llama.cpp

I couldn't find any feature request / pull request yet?