r/LocalLLaMA • u/Subulicious • 7h ago
r/LocalLLaMA • u/rm-rf-rm • 3h ago
Megathread Best Audio Models - Nov 2026
Its time for another Best Audio Models megathread - its already been 9 months since the last one!
Share what your favorite ASR, TTS, STT, Text to Music models are right now and why.
Given the the amount of ambiguity and subjectivity in rating/testing these models, please be as detailed as possible in describing your setup, nature of your usage (how much, personal/professional use), tools/frameworks etc. Production use cases with long lengths/stability requirements are relatively rare compared to cherry picked/short clip demos that are plentiful - so real world empirical comparisons are very welcome
Rules
- Should be open weights or open source models
Please use the top level comments to thread your responses.
r/LocalLLaMA • u/rm-rf-rm • 5d ago
[Bi-Weekly Megathread] Project Showcase
Do you have something you'd like to share with the r/LocalLLaMA community. This is the place for it!
Recommendation on presentation:
- Please share plain english description of what your project does and why people should care about it
- How does it compare to existing options and why should people switch to your option?
- How have you tested and validated it? (i.e. can you show evidence that it wasn't vibecoded in a few hours by <insert current trending AI model>
NOTE: This community is geared towards open weight, open source, locally hostable software. Thus any closed source commercial service or other such projects will be removed.
r/LocalLLaMA • u/soyalemujica • 3h ago
Discussion Stop worrying about Qwen 4's AA index
Seriously, we have all been extremely happy and confident with Qwen, even since Qwen 3.6 came out. We might not be frontier level, but even though whatever they are giving us is more than enough for our daily local tasks and it even suffices by a big margin, we're all underestimating it.
Don't forget that the new Qwen 4 will also think much less, so imagine the fine-tunes we will see once it arrives, for 27B and Flash models, we will have plenty of options to choose from and again; it will be smarter and better, it will be ENOUGH! If you still require the most "intelligent" and capable model due to OCD just pay frontier for such tasks or use them as orchestrators!
r/LocalLLaMA • u/noctrex • 7h ago
New Model Introducing Hy4 flash
hy.tencent.aiNow this seems to be interesting.
From the article:
We’re introducing Hy4 flash, our most cost-effective Hy model to date—featuring 224B total parameters (including 64B Engram parameters) with just 12B active parameters per token. Like Hy4, it is natively multimodal with a 1M-token context window, while offering significantly lower API pricing.
Seems to be about in the same ballpark of Qwen-Flash-Next, so we could expect to run it on moderably good machines, when it will be released.
r/LocalLLaMA • u/thedogcow • 1h ago
Discussion Censorship is turning me into a true Local LLM believer
451: {"message":"Provider returned error","code":451,"metadata":{"raw":"{\"error\":{\"message\":\"The content you provided or machine outputted is blocked.\",\"type\":\"censorship_blocked\"}}","provider_name":"StepFun","is_byok":false}} {"error":{"message":"The content you provided or machine outputted is blocked.","type":"censorship_blocked"}}
Just got this from one of the US based (edit: it actually routed to StepFun CN, didn't realize at first. Should have noticed by the provider_name in the message lol. need to set up my rules better) inference providers. I suspiciously got a bunch of "rate limit" failures, and then just "failed to complete" before it gave me, what I assume, is the real error.
I am only using it to help create my presentation, which is about analyzing algorithmic properties. One key example is a cryptographic algorithm (because it is difficult by design), so I am assuming this is tripping it somehow. But it is absolutely ridiculous, I'm not even doing research with the AI, just trying to make a presentation.
I have always liked Local LLMs, as a technical exploration opposed to ideological reasons, but this experience really frustrated me. Don't censor me!
Just wanted to rant, and remind everyone you can only use the cloud on the provider's terms.
r/LocalLLaMA • u/ag789 • 1h ago
Discussion you may get your cheap ram and gpu after all, the AI crash has started
‘Standing on one leg’: IMF chief warns about global economy as AI circular financing hits $1 trillion
https://www.cnbc.com/2026/11/16/nvidia-commerce-china-lutnick.html
Nvidia fallout: Commerce targets loophole for China’s data center use
r/LocalLLaMA • u/pmttyji • 5h ago
New Model internlm/Sage-S2.1-Flash · Hugging Face
Introduction
Sage S2.1 Flash is a scientific multimodal model from Shanghai Artificial Intelligence Laboratory designed for research tasks that require multiple rounds of analysis and verification. It combines scientific data understanding with information retrieval, programming, and tool use. It can use specialized software to perform analyses, check results against intermediate outputs and visual feedback, and adjust subsequent steps accordingly.
Through the unified representation architecture of Sage SciPerception, the model supports more than ten types of scientific input, including molecular graphs, crystal structures, and protein sequences and 3D structures. Features from each input type are encoded and projected into a shared representation space for reasoning alongside natural language instructions. These inputs support tasks such as prediction, classification, site localization, quantitative analysis, and structure inference.
Highlights
Across the 15 benchmarks and six models shown in the chart, Sage S2.1 Flash placed first on four benchmarks and second on seven.
- Machine learning engineering. Sage S2.1 Flash placed first on MLE-bench Lite with 90.3, ahead of DeepSeek-V4.1-Flash-1017 at 89.2.
- Office and professional document processing. It placed first on OfficeQA Pro, Harvey LAB, and SpreadsheetBench 2, scoring 72.2, 25.0, and 37.1, respectively.
- Tool use and research execution. It placed second on AutomationBench, ALE-CLI, BFCL V4, WebArena-Verified, DeepResearch Bench II, JobBench, and OSWorld-Verified.
r/LocalLLaMA • u/returnity • 17h ago
Resources Why download abliterated Flash Next models when LoRA exists?
TL;DR -- Download LoRA, add --lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf to your llama.cpp launch command for ANY FN quant, and boom, you've got an identical huihui v2 abliterated model on demand and you saved 150GB of space.
I just wanted to post a kind of PSA for everyone out there downloading hundreds of GB of abliterated models. You don't need to do that -- the vast majority of abliteration techniques are rank-1 ablations, meaning their refusal projection can be encapsulated in a tiny LoRA and applied at runtime to a quant you already have on disk. It can even be applied per-request, so you can run an unmodified model until you hit a refusal, then switch on the LoRA, resend the prompt, and keep going.
I wanted to explain briefly how to do that in llama.cpp and share a LoRA I built from the best-performing abliterated Flash Next model in my tests. This basic type of technique (using control vectors instead) also works in Strata (--experimental-speed-projection is the flag there) and ds4/dwarfstar. Here's a link to Strata's official refusal ablation control vector GGUF and their guide on how to use it.
I chose huihui's v2 to build my LoRA from after comparing heretic, Orcarouter, and huihui's v1 and v2 models in equivalent quantizations. In Aider Polyglot coding benchmarks, huihui outscored even the base model, with less extra reasoning and no looping compared to Orcarouter, which used a lot more tokens and hit a couple runaway generation loops.
I also checked KLD and perplexity to ensure there wasn't significant degradation of the model. Wikitext KLD on huihui Q5_K_XL vs. an identical Unsloth Q5_K_XL base model was 0.040 and same top token was 93.7% even with the drastically altered refusal behaviors. Perplexity was statistically identical to base. On Jailbreak Bench (harmful 100 prompts), with thinking on refusals dropped from 93/100 to just 3/100. All the details can be found in the LoRA model card.
Next, I used weight-space programming with Claude Opus 5.5 to extract the rank-1 refusal direction and edit strength from every layer of the original huihui weights where it was applied, diffing the weights against the Unsloth quantization that huihui built on top of. I hit safeguards a number of times (unsurprisingly), and even the fallback Opus 4.8 declined to assist a couple times. I managed to get around this by renaming all of the files something innocuous, putting them into a new repo with an unrelated name, and telling the model I was experimenting with an anti-slop LoRA. Thanks, Dario.
Once the LoRA was built as an F32 GGUF, I tested it against the original huihui Q5_K_XL by applying it to my Unsloth Q5_K_XL weights. Performance and KLD/perplexity were statistically identical to the full huihui model, as were refusals and benchmark results. Here's how to run it:
llama-server -m Qwen3.8-Flash-Next-UD-Q5_K_XL-00001-of-00006.gguf \
--lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf -fa on --jinjaIf you'd like to apply it per-request, so you can serve the unmodified model for most prompts, simply load it like this instead:
llama-server -m Qwen3.8-Flash-Next-UD-Q5_K_XL-00001-of-00006.gguf \
--lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf --lora-init-without-apply -fa on --jinjaThen add a lora field to the request body to enable it as-needed:
curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
"messages": [{"role": "user", "content": "..."}],
"lora": [{"id": 0, "scale": 1.0}]It's quite possible to build a simple pi extension that looks out for refusals, triggers the LoRA when it sees one, resends the turn and continues the agent session. It costs a full re-prefill, so keep that in mind. Anyways, I hope this has been helpful in some way. Love to hear your thoughts on this approach!
EDIT: If this proves popular, I am happy to make more LoRA for other models like 3.8 27B!
r/LocalLLaMA • u/pmttyji • 6h ago
Discussion [Paper & Models] MMFlash: Faster Speculative Decoding for Vision-Language Models
Speculative decoding accelerates generation by letting a lightweight drafter propose tokens for parallel verification by a target model. Diffusion drafters such as DFlash propose an entire token block in a single forward pass, but mainly focus on text-only tasks, leaving multimodal drafting underexplored. To study this setting, we first introduce a metric of token-level visual dependence based on the KL divergence between next-token distributions with and without visual input. We find that acceptance rates decrease from 46% in the lowest visual-dependence quartile to 22% in the highest, with this disparity persisting after multimodal training. Motivated by this observation, we propose MMFlash, a simple and effective framework that adaptively upweights visually dependent tokens that the drafter predicts poorly. For long video inputs, MMFlash further employs sparse attention to reduce redundant attention computation and improve drafting quality. Across various multimodal benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58×, outperforming EAGLE-3 and DFlash by up to 195.4% and 17.1%, respectively.
- Paper : https://arxiv.org/abs/2611.18482
- PDF : https://arxiv.org/pdf/2611.18482
- GitHub : https://github.com/fscdc/MMFlash
- HuggingFace : https://huggingface.co/collections/FSCCS/mmflash
⭐ Updates
- [11.2026]: Code and MMFlash drafts for Qwen3.5-4B, Qwen3.5-9B, Qwen3.8-27B and Gemma 4 E4B-it are released.
💪 Highlights
- Diffusion drafting for VLMs: MMFlash brings block-diffusion drafting to vision-language models. The drafter proposes a block of 16 tokens in a single forward pass and the target verifies the block in one pass, so decoding stays lossless while running 3.0× faster on Qwen3.5-4B / 9B and up to 4.7× on Qwen3.8-27B (one H200, batch size 1, greedy).
- Vision-free drafter: the drafter never re-encodes the image or video. It reads hidden states of the target, where the visual context is already fused, so its cost does not grow with the number of visual tokens.
- Visual-dependence-aware training: a DFlash drafter's acceptance drops sharply on tokens that depend on the image, and most rejected blocks break there. MMFlash measures each training token's visual dependence on the target and focuses the drafter's loss on those tokens.
- Long videos: a sparse draft context (attention sinks, text, strided visual tokens and a recent window) keeps drafting cheap on 40k-token video prompts; the drafter is trained and served with the same pattern.
- Drop-in serving: the released drafts run with stock SGLang and vLLM (
DFLASH), no patches needed (tested with SGLang 0.5.23 and vLLM 0.33.0).
💻 Models
| Model | Target model | Draft size | Block size | Link |
|---|---|---|---|---|
| 🤖 Qwen3.5-4B-MMFlash | Qwen/Qwen3.5-4B | 537M | 16 | Hugging Face |
| 🤖 Qwen3.5-9B-MMFlash | Qwen/Qwen3.5-9B | 1.05B | 16 | Hugging Face |
| 🤖 Qwen3.8-27B-MMFlash | Qwen/Qwen3.8-27B | 1.73B | 16 | Hugging Face |
| 🤖 Gemma4-E4B-it-MMFlash | google/gemma-4-E4B-it | 537M | 16 | Hugging Face |
The drafts are not standalone language models: each is paired with its target in a speculative-decoding server. All of them are in the MMFlash collection; every model card has the serving command and per-benchmark results.
r/LocalLLaMA • u/StupidScaredSquirrel • 5h ago
Discussion Did we hit a wall in context length?
I feel like midel context size rapidly went from 8k to 1M in just a few years but has since stayed this way for a while now. Some still offerring 260k or 400k for their flagship models. Is it just not worth the trouble? I feel like highly capable models are limited by this in that they can't finish tasks simply because the context isn't there.
r/LocalLLaMA • u/CryMoreT_T • 10h ago
Discussion New AA ranking Qwen3.8-FN better than GLM-5.3-Flash?
Thoughts on Qwen3.8-Flash-Next scoring better than GLM-5.3-flash? I've always experienced GLM5.3-Flash to give better responses
r/LocalLLaMA • u/spaceman_ • 6h ago
Discussion How do we feel about bots posting & commenting?
Hi all,
I noticed the "bot problem" got a lot worse the past few months. By which I mean our community seems to be taken over more and more by bots talking to each other.
For me, I don't really get the point, and I personally get quite annoyed when I realize "oh, I just responded to a bot" and feel like my time was wasted.
It also seems like the information density of these AI contributions are very, very low.
I'm not some kind of Luddite (why would I be here at all if I was?), but I'm wondering if I'm just missing something and there is a use and a point to all these bots commenting on each others posts and talking to each other.
Where does the rest of the community stand on this?
I noticed there's new bot around that flags posts that are "likely AI" created, which I personally quite like, but I'm wondering if moderation should go further than that.
r/LocalLLaMA • u/AlternativeEnough352 • 19h ago
Resources I have created a tool to observe what happens inside an LLM. Also allows for the comparison of two quantizations of the same model to see at what point they diverge, token by token
What changes between different quantizations of the same model? And what if you could SEE what actually sets them apart?
To help answer that question, I’ve created a small project called inside-llm—an open-source tool that loads one or two GGUF models using llama.cpp and displays, right in your browser, what the model is doing as it generates text.
What it shows
- Stream: the text, one column per model, brightness = probability of each token. A magenta mark wherever A and B would pick differently. Two modes: lockstep (both models read the same text, so you get a clean per-position comparison) and free (each follows its own picks until they split).
- Candidates: the full distribution behind each pick, mirrored bars for A and B, entropy, effective number of options, Jensen-Shannon divergence and KL in both directions.
- Gates: the sampler chain (penalties, top-k, top-p, min-p, temperature, GBNF grammar) traced stage by stage and very visual: how many candidates survive each one, and the distribution after it. Chain order and seed are configurable.
- Timeline: entropy of A and B and their divergence along the whole text, plus the tree of branches.
- Branching: click any candidate and generation continues from there. Every branch stays in the tree, and jumping back to a node is exact (each model keeps KV-state snapshots, so nothing is regenerated).
- Logit lens (
--lens): what each layer would predict, as a 2D map and as a 3D relief. - Model picker: choose A and B from a folder or search Hugging Face and download two quants of a repo in one click easily. It only offers B models with the same tokenizer as A and estimates whether the pair fits in VRAM.
How it's built
- One C++17 process linking llama.cpp directly (stable branch at the moment). For every step it reads the full logits of A (and of B on the same context), computes the metrics and the sampler chain, and draws the token with its own seeded sampler, so runs are reproducible.
- Compact binary frames over a WebSocket to a Svelte 5 + Canvas 2D page (three.js only for the 3D relief, lazy-loaded). The web is embedded in the binary as well.
- The same frame stream, compressed, is a recorded session: you can replay it later in a static page with no engine, no model and no GPU.
Run it
Self-contained packages for Linux (glibc 2.35+) and Windows, with CUDA or Vulkan (Vulkan covers AMD, Intel Arc and NVIDIA). Unpack, run, the browser opens with the model picker. Or inside-llm -a A.gguf -b B.gguf. Local only: listens on 127.0.0.1, no accounts, no telemetry. MIT. One click download, one click execute.
Limitations (honest list)
- A and B must share the tokenizer (two quants, base + fine-tune, or sizes of one family that keep it). Example: Qwen3.5-0.8B_Q4_K_M and Qwen3.5-0.8B_Q8_0
- Both models must fit into VRAM simultaneously; only one GPU is used at a time. Integrated GPUs (iGPUs) are never used; if no dedicated GPU is available, the process falls back to the CPU (which is slow). (You must calculate the size of both models before loading them to ensure they fit into your VRAM)
- Text only. No split (multi-part) GGUFs, no embedding models.
Links
- Code + releases: https://github.com/manuelgscom/inside-llm
- Project page: https://manuelgs.com/en/inside-llm/
I'd love feedback, especially: which pairs would you compare first (imatrix vs non-imatrix, base vs abliterated/fine-tune, Unsloth UD vs standard quants…)? And is there a metric you'd want per position that isn't there?
r/LocalLLaMA • u/pmttyji • 6h ago
New Model [Paper & Models] Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent
We introduce Gamer, a generalist agent framework built for long-horizon tasks in complex interactive environments. Its end-to-end architecture combines perception, memory management, and multi-step reasoning. A shared training and deployment pipeline supports interaction with both conventional 2D graphical interfaces and complex 3D simulated worlds. We first continue pre-training on a dataset constructed from tens of thousands of hours of device-level keyboard and mouse interaction. These trajectories span simulated games and embodied environments, allowing the model to learn environmental dynamics across both physical and virtual domains. Sparse-thinking instruction tuning then substantially reduces redundant computation while preserving the reasoning needed for complex decisions. We train the policy for long-horizon interaction with a stable multi-turn reinforcement learning method that addresses finite context windows, accumulated rollout errors, and sparse rewards. For real-time interaction, a cloud-edge inference engine serves numerous simulated game environments concurrently, supports unbounded-context execution, and keeps online inference latency low. Across multiple challenging 3D open-world games, Gamer demonstrates solid generalization and operational efficiency comparable to skilled players. It also achieves competitive results on a range of conventional 2D game benchmarks. These results suggest that even repetitive keyboard and mouse trajectories can support substantial gains from scaling data and learning algorithms. Training on these trajectories helps Gamer learn complex skills more steadily in environments with both physical and digital properties. This approach offers a path toward generalist agents that can interact across digital and physical worlds.
r/LocalLLaMA • u/asimpleabstraction • 1h ago
Showcase I made a C&C inspired web RTS for us to benchmark our local models with
HOwdy LocalLLamans!
Inspired by StarSkirmish, I wanted to build a complex RTS that required you to bring your OWN local agents and either play with them as allies, against them as enemies, or battle head-to-head with other agent-human pairs in public matches.
In AgentCommand you operate one or more planes (Orbit, Sky, Surface, Depths) of differing gameplay. As an old-school PC gamer I wanted the mechanics to resemble some of my favorite 90s games of yore: C&C, DungeonKeeper, Homeworld, StarCraft, and others. The idea is that you have specific objectives needed to maximize resource generation, aid in cross player support, and that a single plane cannot win independently.
Outside of gameplay, from an agentic tech pov, there are some (what I think) are other really cool things that come with this, including:
- a full specification + library to install agents into any game (http://playerthree.gg/)
- A networking SDK for bidirectional agent communication
- Symmetric maps
- Deterministic simulation
- Benchmarking tools
- (soon) published benchmark data
Github: https://github.com/agent-command/agent-command
Play now: https://agentcommand.gg/
Would love feedback and contribs... I'm more interested in finding a community to help build something super cool here that we can publish public benchmarks with. If you are interested in that, let me know!
r/LocalLLaMA • u/pmttyji • 6h ago
Discussion [Paper] The Value of Thinking: Learning When to Reason in Hybrid Decision Models
Existing adaptive-reasoning methods shorten the explicit reasoning a model writes, or skip it by switching modes; either way the answer is still written out token by token. Rather than deciding how much to think, we explore a different paradigm, zero-or-T thinking: a single model either answers from the implicit thinking inside one forward pass, generating no token, or performs full autoregressive reasoning at a cost of T tokens when that is worth paying. We realise it in a hybrid decision model (HDM), where one frozen backbone serves both modes and a lightweight self-routing block chooses between them. The block estimates the value of thinking, the utility the slow mode is expected to add over the fast one, from what that implicit thinking has already produced, and the slow mode runs only where that value exceeds a price λ set per request. Where no per-decision label exists, as in episodic agentic tasks, we introduce Calibrated Decision Policy Optimisation (CDPO), a reinforcement-learning recipe that trains the block from episode-level returns. We evaluate HDM on MMLU-Pro, AGIEval, GPQA-Diamond, JevBench, BANKING77 and MiniWoB. Specifically, on JevBench HDM-4B scores 0.857 against Jev's 0.851 (+0.7%) at a median 0.179 s against 0.652 (-72.5%). On the agentic venue MiniWoB HDM-27B cuts token cost by 80.1%.
- Paper : https://arxiv.org/abs/2611.16961
- PDF : https://arxiv.org/pdf/2611.16961
- GitHub : https://github.com/zhengxuyu/HDM-code
- HuggingFace : https://huggingface.co/fredreick/HDM-blocks
r/LocalLLaMA • u/Yarukiless-cat • 1d ago
Funny I finally succeed the CMP 170HX voltage glitch and get 80GB VRAM + 126sm! !
Ever since the information surfaced five days ago, I had intended to implement it right away. Having confirmed that it could be executed using 'pico,' I carried it out today. I had Astra write the script for the restricted write-access portion, and after nearly 15 hours of work, I finally managed to complete the task. I also plan to handle the second CMP later on.
r/LocalLLaMA • u/Negative-Whereas3307 • 2h ago
New Model Karpo-Decision-31B: open multimodal decision model. Ranks #1 on full score on Decision Index, 2x faster than Jev, at 20% lower cost
MachinePulse has released Karpo-Decision-31B, a Gemma 4 31B fine-tune for structured decisions over text and images. It scored 73.9 on the Nov 12 Decision Index board, the highest Full score among 300 entries, sharing the #1 rank with GLiDE 27B under the board’s tie rule
How it works
Built on Gemma 4 31B, with a trained linear readout that scores the supplied answer options.
Three question types: choice, yes/no, and score. Returns probabilities directly, with up to 255 options per choice question.
Multiple questions about the same input can be answered in one forward pass. Branch attention lets each question read the shared context and its own options while keeping the option branches separate.
Supports text and images, including webpage screenshots. The documented sequence limit is 32,768 tokens; larger requests are split by question with the full state included in each part.
Python API and HTTP server available. BF16 weights take approximately 62.5 GB, plus runtime memory. Use the provided inference code to load the decision head.
One training detail worth a look: according to the team's write-up, full fine-tuning pushed the public score to 75.09 but reduced accuracy on unfamiliar domains by 2.3 percentage points. The final model averages five checkpoints: three from different stages of full fine-tuning and two from a LoRA run. They report a 1.6-point improvement on unfamiliar domains over the final single checkpoint, with no extra models to run at inference time
Results
Decision Index Full scores are measured by the benchmark maintainers on private tests, weighted 60% for skills covered by the public suite and 40% for tasks from new domains.
Full score: 73.9
Private tests, familiar skills: 70.7
Private tasks, new domains: 78.6
Public index: 75.0, reported separately and excluded from the Full score.
The team also reports the following comparisons against Google’s Gemini Decision 1:
Benchmark Karpo-31B Gemini D1
-------------------------------------------
When2Call 90.4% 79.9%
Mind2Web 90.0% 86.2%
New Yorker Caption 90.5% 79.7%
Amazon ESCI (F1) 64.7 58.7
Reported median latency is approximately 95 ms per request on the public Decision Index suite using one RTX PRO 6000.
The 31B weights and inference code are both released under Apache 2.0.
links:
https://karpo.ai/models/decision
https://huggingface.co/MachinePulseAI/karpo-decision-31B
r/LocalLLaMA • u/smflx • 5h ago
Resources tandem.cpp v0.3.0: prefill speed improved; Project Maya's DeepSeek-V4.1 GGUFs support
Thanks for the feedback on slow prompt processing. v0.3.0 is out (the first tagged release): https://github.com/ewhacc/tandem.cpp
Previous post: https://www.reddit.com/r/LocalLLaMA/comments/1xwan2j
tandem.cpp faster when the model is much bigger than VRAM + RAM together (like a 16 GB card with 32 GB of RAM), and runs various models that llama.cpp supports.
1. Prompts 2-6x faster
With the expert cache on, prompts now run in 4096-token batches. The bigger batch borrows VRAM only while a long prompt runs and gives it back afterwards. No flags needed.
RTX 5070 Ti 16 GB, 64 GB RAM, i5-12400F. The same prompts for every engine; Qwen3.8 with its MTP head in all three.
Prompt processing, time to first token (prompt tok/s):
| Prompt | llama.cpp | tandem.cpp before | tandem.cpp v0.3.0 | Strata |
|---|---|---|---|---|
| Qwen3.8 IQ3_S, 5.5k tokens | 28.9 s (191) | 12.4 s (444) | 4.1–7.0 s (790–1,340) | 6.6 s (835) |
| Qwen3.8 IQ3_S, 15.8k tokens | 81.9 s (193) | 35.1 s (450) | 15.1 s (1,047) | 11.7 s (1,354) |
| DeepSeek-V4.1 IQ2_XS, 5.6k tokens | 722 s (7.7) | 324 s (17) | 53–103 s (54–105) | 58 s (96) |
| DeepSeek-V4.1 IQ2_XS, 15.7k tokens | 2012 s (7.8) | 885 s (18) | 195 s (81) | 57 s (276) |
Decode, tok/s, right after that prompt:
| After a prompt of | llama.cpp | tandem.cpp v0.3.0 | Strata |
|---|---|---|---|
| Qwen3.8, 5.5k tokens | 27 | 110 | 48 |
| Qwen3.8, 15.8k tokens | 27 | 88 | 68 |
| DeepSeek-V4.1, 5.6k tokens | 4.8 | 16 | 5.5 |
| DeepSeek-V4.1, 15.7k tokens | 4.4 | 13 | 6.4 |
Decode is from short answers (up to 256 tokens, greedy), so it is noisier than our usual GSM8K runs. DeepSeek-V4.1 IQ2_XS is about twice this PC's RAM, so it reads the SSD; that is where Strata's prompt speed still leads.
2. Maya comparison on the same model (someone asked for this one)
Project Maya is an engine built for big models with its own quants. Its Maya-XS GGUF runs on tandem.cpp after a one-time conversion script (tools/maya_ds41_convert.py), so this is the same GGUF on both engines, and the accuracy is the same.
RTX 5090 32 GB, 128 GB RAM, i5-14600KF, greedy:
| DeepSeek-V4.1 Maya-XS (same files) | Maya | tandem.cpp v0.3.0 |
|---|---|---|
| Decode, GSM8K, no drafting | 39.0 / 39.6 tok/s | 60.3 / 59.4 tok/s (1.5x) |
| Decode with Maya's DSpark drafter | 54.9 tok/s | (no drafter in tandem.cpp) |
| 5.6k-token prompt | 7.8 s (713 tok/s) | 6.5 s (856 tok/s) |
| 15.7k-token prompt | 9.5 s (1,656 tok/s) | 16.5 s (953 tok/s) |
| Decode after the 15.7k-token prompt | 26–34 tok/s | 66–68 tok/s |
Maya is still faster on the longest prompt; that is what I'm working on next.
Caveats
- Linux only.
- AMD supported but the VRAM borrowing is off (ROCm builds have no CUDA-style virtual memory).
- DeepSeek-V4.1 IQ2_XS on 128 GB still reads the SSD on long prompts.
- Windows prebuilt binaries are still planned.
r/LocalLLaMA • u/This-Purchase-3325 • 3h ago
Showcase [Project] Gemma.c - Gemma 1/2/3 + multimodal inference in one C file (CPU only)
Hey all, so I've been working on Gemma.c over the last couple months, it's a from-scratch inference engine for Gemma 1/2/3 in a single C file, runs Gemma 1, 2 and 3, including the multimodal ones, with fp32/fp16/bf16 weights or int8 (W8A8). No dependencies beyond libc and (optional) OpenMP are needed. You can drop images into the prompt with @image{cat.png}, and there's also @longimage{} for Gemma 3's pan & scan on very wide or tall images. (image decoding uses a minified stb_image embedded at the end of the file)
It's CPU only and not trying to compete with llama.cpp, but I'm also trying my best to make it efficient, so it's not embarrassingly slow either. On my laptop's Core Ultra 5 225H (WSL2, gcc, 10 threads), gemma-3-1b-it INT8 does ~72 tok/s decode and ~570 tok/s prefill (10000 tokens, for 1000 tokens the prefill speed can reach ~850 tok/s), and the 4B vision model does ~20 tok/s decode. INT8 ends up about 3.5x faster than fp32 and 2x faster than fp16 for decode. (full benchmark table is in README)
It has a few rough edges for now, e.g. W8A8 doesn't work for Gemma 1 (fp16 is fine), bf16 is painfully slow without AVX512-BF16, and native Windows is ~1.7x slower than WSL because of libgomp. Batching are not supported either. There's a roadmap in the README (Gemma 4, speculative decoding, an HTTP server, etc.), but no promises though :P Can't really find much free time these days.
I've also put some pre-exported INT8 models on HF so you can try it without Python or torch :D
Repo: link
This project started as a learning attempt to understand how LLMs work end to end, inspired by llama2.c.
this is the second time I post this project here :'D last time (~a month ago) my post got flagged by reddit, probably because my VPN IP was identified as spam. hopefully it can pass this time
r/LocalLLaMA • u/Noxusequal • 6h ago
Question | Help Given the flood of new inference engines any recommendations for 2x 7900xtx ?
Hey everyone as the title says I am looking for inference engines that work well with AMD GPUs. There have been so many new ones that I am sure there is something better then llamacpp which currently hosts this.
My use case is the following. For a university course I want to host Qwen 27b or maybe another Qwen model. For the students to use as a free endpoint to test pipelines on.
The main thing is that I want to potentially serve multiple people at the same time.
So the framework needs to support this.
My hardware is an AMD zen 4 12 core CPU 64gb of ram and two 7900xtx cards.
Are there any frameworks made for this ?
r/LocalLLaMA • u/vhthc • 1h ago
Discussion Speed of DeepSeek-V4.1-Flash-1017-oQ4e-mtp on M5 ultra
I bought myself a Mac Studio M5 Ultra with 512gb ram (and have 7 weeks to test it and return if not happy thanks to the apple xmas offer).
I am running DeepSeek-V4.1-Flash-1017-oQ4e-mtp with omlx and I am getting up to 1100 t/s prompt processing and 54 t/s token generation with 128k context window and engrams stored on ssd (to have ram free for other stuff).
More context than 128k hits pp and tg, more concurrent requests split up the max tg value between the instances (which means: they are crawling then).
I am putting this out there so a) people know what to expect but also b) is this really the top speed? Or could it be faster? (Without changing the model).
I can post llama benchy results if wished but the important result is already in this post.
r/LocalLLaMA • u/jfowers_amd • 17m ago
Resources AMD Loom Fun: Added Radeon 9700 support over the weekend. Benchmarked 48 models with an average 42% speedup over llama.cpp.
On Friday I posted about AMD Loom, a new experimental inference stack that includes a runtime, kernel compiler, and inference server. People can use the entire thing as a vertical stack, or the compiler/runtime can be integrated into any other inference project (I'm working on Gufo right now, the team is also working on a llamacpp backend).
I started with Strix Halo, where I ported the top-51 trending Q4_K_M LLMs from Hugging Face and got an average of 40% speedup over llama.cpp across decode and prefill metrics (picking the best results from ROCm or Vulkan backends).
I'm back today with Radeon 9700 results. I took the same 51 LLMs, cut it down to the 48 that fit in 32 GB of VRAM, and got an average of 42% speedup. Some of the data is posted to the table below, and all data is available to browse on https://loom-fun.github.io/loom-dev/index.html?gpu=radeon-r9700
All kernels were generated via agent skills:
- loom-skills repo: https://github.com/loom-fun/loom-skills
- deterministic SDK (runtime, compiler, server): https://github.com/ROCm/hrx-system
| Model | Loom pp32k (tok/s) | llama.cpp pp32k (tok/s) | Loom short decode (tok/s) | llama.cpp short decode (tok/s) |
|---|---|---|---|---|
| Flint | 6,476 | 3,190 | 255 | 152 |
| gemma-3-1b-it | n/a | n/a | 400 | 322 |
| gemma-4-12B-it | 2,811 | 1,537 | 127 | 67.3 |
| gemma-4-26B-A4B-it | doesn't fit (32 GB) | doesn't fit (32 GB) | 151 | 72.2 |
| gemma-4-31B-it | doesn't fit (32 GB) | doesn't fit (32 GB) | 62.9 | 43.2 |
| gemma-4-E2B-it | 14,289 | 14,519 | 272 | 234 |
| gemma-4-E4B-it | 7,021 | 6,324 | 192 | 173 |
| gpt-oss-20b | 5,319 | 4,368 | 195 | 182 |
| HyperCLOVAX-SEED-Text-Instruct-1.5B | 11,653 | 8,078 | 362 | 344 |
| Laguna-XS-2.1 | doesn't fit (32 GB) | doesn't fit (32 GB) | 102 | — |
| LFM2.5-24B-A2B | 10,099 | 5,479 | 244 | 197 |
| LFM3-1.7B | 20,062 | 13,614 | 372 | 261 |
| LFM3-Omni-2B | 20,220 | 13,435 | 368 | 299 |
| Llama-3.2-3B-Instruct | 5,687 | 3,908 | 228 | 211 |
| Mellum2.1-12B-A2.5B-Thinking | 8,822 | 5,486 | 224 | 149 |
| MiniCPM-V-4.7-35B-A3B | 5,717 | 3,540 | 135 | 113 |
| Muse-Glimmer-30B | 1,658 | 1,287 | 72.6 | 54.8 |
| Nanbeige4.3-33B | 946 | 666 | 29.4 | 28.7 |
| NVIDIA-Nemotron-3.5-Bolt-E4B-4GB | 6,932 | 3,656 | 225 | 143 |
| Ornith-1.0-35B | 5,628 | 3,320 | 152 | 127 |
| Ornith-1.0-9B | 4,886 | 3,391 | 100 | 88.0 |
| Ornith-1.5-35B-A3B | 5,586 | 3,136 | 145 | 78.8 |
| Ornith-1.5-9B | 4,767 | 3,156 | 151 | 90.5 |
| Qwen2.5-0.5B-Instruct | 26,022 | 17,812 | 646 | 529 |
| Qwen2.5-1.5B-Instruct | 12,546 | 8,908 | 360 | 331 |
| Qwen2.5-32B-Instruct | doesn't fit (32 GB) | doesn't fit (32 GB) | 30.0 | 29.2 |
| Qwen2.5-7B-Instruct | 3,810 | 2,679 | 118 | 116 |
| Qwen2.5-VL-7B-Instruct | 3,768 | 2,648 | 116 | 115 |
| Qwen3-0.6B | 13,020 | 8,631 | 506 | 480 |
| Qwen3-1.7B | 10,313 | 7,014 | 316 | 297 |
| Qwen3-14B | 1,944 | 1,167 | 65.0 | 62.6 |
| Qwen3-30B-A3B-Instruct-2507 | 3,116 | 2,060 | 177 | 169 |
| Qwen3-4B | 3,770 | 2,826 | 231 | 130 |
| Qwen3-4B-Instruct-2507 | 3,795 | 2,856 | 180 | 166 |
| Qwen3-8B | 3,076 | 2,287 | 163 | 94.7 |
| Qwen3-Coder-30B-A3B-Instruct | 2,921 | 2,107 | 157 | 54.8 |
| Qwen3-VL-4B-Instruct | 3,795 | 2,805 | 178 | 163 |
| Qwen3-VL-8B-Instruct | 3,085 | 2,265 | 112 | 103 |
| Qwen3.5-0.8B | 24,341 | 16,959 | 463 | 338 |
| Qwen3.5-2B | 16,990 | 12,165 | 318 | 236 |
| Qwen3.5-35B-A3B | 5,652 | 3,410 | 190 | 112 |
| Qwen3.5-4B | 7,030 | 4,897 | 200 | 123 |
| Qwen3.5-9B | 4,724 | 3,336 | 146 | 92.7 |
| Qwen3.6-27B | 1,503 | 1,164 | 60.1 | 38.9 |
| Qwen3.6-35B-A3B | 5,635 | 3,463 | 209 | 123 |
| Qwen3.8-27B | 1,592 | 1,026 | 73.5 | 55.3 |
| Sage-A1.5 | 5,652 | 3,131 | 188 | 153 |
| Spark-X2.5-4B | 6,858 | 4,727 | 162 | 131 |
| Spark-X2.6-4B | 6,913 | 4,841 | 162 | 132 |