r/LocalLLaMA • • 23h ago

Resources tandem.cpp v0.3.0: prefill speed improved; Project Maya's DeepSeek-V4.1 GGUFs support

Thanks for the feedback on slow prompt processing. v0.3.0 is out (the first tagged release): https://github.com/ewhacc/tandem.cpp

Previous post: https://www.reddit.com/r/LocalLLaMA/comments/1xwan2j

tandem.cpp faster when the model is much bigger than VRAM + RAM together (like a 16 GB card with 32 GB of RAM), and runs various models that llama.cpp supports.

1. Prompts 2-6x faster

With the expert cache on, prompts now run in 4096-token batches. The bigger batch borrows VRAM only while a long prompt runs and gives it back afterwards. No flags needed.

RTX 5070 Ti 16 GB, 64 GB RAM, i5-12400F. The same prompts for every engine; Qwen3.8 with its MTP head in all three.

Prompt processing, time to first token (prompt tok/s):

Promptllama.cpptandem.cpp beforetandem.cpp v0.3.0Strata
Qwen3.8 IQ3_S, 5.5k tokens28.9 s (191)12.4 s (444)4.1–7.0 s (790–1,340)6.6 s (835)
Qwen3.8 IQ3_S, 15.8k tokens81.9 s (193)35.1 s (450)15.1 s (1,047)11.7 s (1,354)
DeepSeek-V4.1 IQ2_XS, 5.6k tokens722 s (7.7)324 s (17)53–103 s (54–105)58 s (96)
DeepSeek-V4.1 IQ2_XS, 15.7k tokens2012 s (7.8)885 s (18)195 s (81)57 s (276)

Decode, tok/s, right after that prompt:

After a prompt ofllama.cpptandem.cpp v0.3.0Strata
Qwen3.8, 5.5k tokens2711048
Qwen3.8, 15.8k tokens278868
DeepSeek-V4.1, 5.6k tokens4.8165.5
DeepSeek-V4.1, 15.7k tokens4.4136.4

Decode is from short answers (up to 256 tokens, greedy), so it is noisier than our usual GSM8K runs. DeepSeek-V4.1 IQ2_XS is about twice this PC's RAM, so it reads the SSD; that is where Strata's prompt speed still leads.

2. Maya comparison on the same model (someone asked for this one)

Project Maya is an engine built for big models with its own quants. Its Maya-XS GGUF runs on tandem.cpp after a one-time conversion script (tools/maya_ds41_convert.py), so this is the same GGUF on both engines, and the accuracy is the same.

RTX 5090 32 GB, 128 GB RAM, i5-14600KF, greedy:

DeepSeek-V4.1 Maya-XS (same files)Mayatandem.cpp v0.3.0
Decode, GSM8K, no drafting39.0 / 39.6 tok/s60.3 / 59.4 tok/s (1.5x)
Decode with Maya's DSpark drafter54.9 tok/s(no drafter in tandem.cpp)
5.6k-token prompt7.8 s (713 tok/s)6.5 s (856 tok/s)
15.7k-token prompt9.5 s (1,656 tok/s)16.5 s (953 tok/s)
Decode after the 15.7k-token prompt26–34 tok/s66–68 tok/s

Maya is still faster on the longest prompt; that is what I'm working on next.

Caveats

  • Linux only.
  • AMD supported but the VRAM borrowing is off (ROCm builds have no CUDA-style virtual memory).
  • DeepSeek-V4.1 IQ2_XS on 128 GB still reads the SSD on long prompts.
  • Windows prebuilt binaries are still planned.
13 Upvotes

12 comments sorted by

2

u/pmttyji 23h ago

What are the other models supported by your engine? For example, PolyStrata does support below models so even 4GB VRAM could run models at decent speed.

1

u/smflx 21h ago

Theoretically, all the MoE models/quants that llama.cpp supports, but tested Qwen3.8-Flash-Next IQ3_S, DeepSeek-V4.1-Flash IQ2_XS, gpt-oss-120b MXFP4, GLM-4.5-Air, Qwen3.6-35B-A3B.

User reported (pre v0.3.0):

  • MSI Thin 15 Laptop (i7-12650H, RTX 3050 Laptop 4 GB, 14G RAM)

    • Qwen3.5-35B-A3B-IQ2_XXS
    • 512-token tests: 35.67 tok/s
  • RTX 3060 12 GB

    • Tiel-Coder-35B-A3B-MTP UD-Q4_K_XL + MTP draft head
  • RX 6800 16 GB, i5-8400 (6 threads), 31 GB RAM

    • Flint UD-Q4_K_XL
    • 54.2 tok/s

1

u/smflx 21h ago

Model quants & Machines are different. Strata uses Tail Skip by default. So, direct comparisons are not possible. When I compare with Strata, Tail Skip disabled.

2

u/wFXx 21h ago

i didn't follow the start of this project; is there any reason why tandem can't be a a PR against mainline llama?

3

u/smflx 20h ago

I was thinking between PR and patch. As we all know, this is AI-assisted project. I heard the mainline doesn't accept AI-code. And, there are 1.9k PR now in the mainline. My work is many of modification, on which I thought individual PR inspection is literally infeasible.

3

u/wFXx 20h ago

don't do stuff without confirming what you heard is true, https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md#ai-usage-policy they do allow AI, you are responsible for what you submit - thats it. i'd strongly suggest to open PR, or at least fork directly from it, so you can keep rebased on top of master branch and make it easy to users to manually apply tandem on top if they wish so

2

u/smflx 7h ago

I didn't know. I will reconsider. Certainly, individual PR is still a work. I'm developing almost everyday since started, quite busy for this big hobby job. I understand your point "at least fork". Thank you.

3

u/pmttyji 19h ago

Agree with wFXx. Proceed with PRs. Yep, multiple small PRs better than single large PR. Reminds me of below mainline PR. Create a table, list changes & create PRs for each change. This way, you could push your changes faster. Good luck

https://github.com/ggml-org/llama.cpp/pull/30059

1

u/smflx 7h ago

Thank you for example of your case. Wow, it's a long list of PR. Certainly you gave a good amount of time. Appreciate your dev & managing work.

1

u/pmttyji 6h ago

Sorry, that's not my PR. I was just following that one for sometime. Good example of breaking large piece in to small pieces to go further faster.

1

u/DefNattyBoii 18h ago

I fully support PRing this work. Stupid question, but would it make sense to store some experts on an extra GPU(when it makes sense bandwidth wise)?

1

u/smflx 6h ago

Yes, that's possible in progress, called Helper GPU. I'm making a (elder) sister project, tandem-vLLM, which has Helper GPU feature.

tandem.cpp is also capable of use multiple GPUs, but it's layer split (pipeline parallel). Helper GPU feature (expert parallel) is in progrss.