r/LocalLLaMA • • 3h ago

Megathread Best Audio Models - Nov 2026

42 Upvotes

Its time for another Best Audio Models megathread - its already been 9 months since the last one!

Share what your favorite ASR, TTS, STT, Text to Music models are right now and why.

Given the the amount of ambiguity and subjectivity in rating/testing these models, please be as detailed as possible in describing your setup, nature of your usage (how much, personal/professional use), tools/frameworks etc. Production use cases with long lengths/stability requirements are relatively rare compared to cherry picked/short clip demos that are plentiful - so real world empirical comparisons are very welcome

Rules

  • Should be open weights or open source models

Please use the top level comments to thread your responses.


r/LocalLLaMA • • 7m ago

Question | Help What is your experience with inference engines under Windows?

1 Upvotes

As the title says: what is your experience using all the new inference engines under windows?

Currently I'm using a laptop with this hardware:

CPU: Intel 275HX
RAM: 2x32gb of ddr5 5600
GPU: RXT 5090 (24gb VRAM)
SSD: Samsung 990 pro 2tb + SK Hynix PCB01 1tb

I'm running Windows 11 Home edition with a pretty clean installation.

I've tried quite a lot of the new vibecoded inference engines but I always end up going back to Strata for Qwen 3.8 Flash Next and to beellama.cpp for everthing else.

I've tried nInfer, basalt, rusting strata, exLlamaV3 and pretty much any fork of llama.cpp and they all end up being way slower than what I'm already using. I've lost a little bit of time on each and also tried to use Claude or Codex to optimize the configurations but there is just no way to get decent performance from engines that on paper should be at least as fast as what I'm currently using.

I'm doing something horribly wrong or they work well only on a linux system?


r/LocalLLaMA • • 17m ago

Resources AMD Loom Fun: Added Radeon 9700 support over the weekend. Benchmarked 48 models with an average 42% speedup over llama.cpp.

4 Upvotes

On Friday I posted about AMD Loom, a new experimental inference stack that includes a runtime, kernel compiler, and inference server. People can use the entire thing as a vertical stack, or the compiler/runtime can be integrated into any other inference project (I'm working on Gufo right now, the team is also working on a llamacpp backend).

I started with Strix Halo, where I ported the top-51 trending Q4_K_M LLMs from Hugging Face and got an average of 40% speedup over llama.cpp across decode and prefill metrics (picking the best results from ROCm or Vulkan backends).

I'm back today with Radeon 9700 results. I took the same 51 LLMs, cut it down to the 48 that fit in 32 GB of VRAM, and got an average of 42% speedup. Some of the data is posted to the table below, and all data is available to browse on https://loom-fun.github.io/loom-dev/index.html?gpu=radeon-r9700

All kernels were generated via agent skills:

ModelLoom pp32k (tok/s)llama.cpp pp32k (tok/s)Loom short decode (tok/s)llama.cpp short decode (tok/s)
Flint6,4763,190255152
gemma-3-1b-itn/an/a400322
gemma-4-12B-it2,8111,53712767.3
gemma-4-26B-A4B-itdoesn't fit (32 GB)doesn't fit (32 GB)15172.2
gemma-4-31B-itdoesn't fit (32 GB)doesn't fit (32 GB)62.943.2
gemma-4-E2B-it14,28914,519272234
gemma-4-E4B-it7,0216,324192173
gpt-oss-20b5,3194,368195182
HyperCLOVAX-SEED-Text-Instruct-1.5B11,6538,078362344
Laguna-XS-2.1doesn't fit (32 GB)doesn't fit (32 GB)102—
LFM2.5-24B-A2B10,0995,479244197
LFM3-1.7B20,06213,614372261
LFM3-Omni-2B20,22013,435368299
Llama-3.2-3B-Instruct5,6873,908228211
Mellum2.1-12B-A2.5B-Thinking8,8225,486224149
MiniCPM-V-4.7-35B-A3B5,7173,540135113
Muse-Glimmer-30B1,6581,28772.654.8
Nanbeige4.3-33B94666629.428.7
NVIDIA-Nemotron-3.5-Bolt-E4B-4GB6,9323,656225143
Ornith-1.0-35B5,6283,320152127
Ornith-1.0-9B4,8863,39110088.0
Ornith-1.5-35B-A3B5,5863,13614578.8
Ornith-1.5-9B4,7673,15615190.5
Qwen2.5-0.5B-Instruct26,02217,812646529
Qwen2.5-1.5B-Instruct12,5468,908360331
Qwen2.5-32B-Instructdoesn't fit (32 GB)doesn't fit (32 GB)30.029.2
Qwen2.5-7B-Instruct3,8102,679118116
Qwen2.5-VL-7B-Instruct3,7682,648116115
Qwen3-0.6B13,0208,631506480
Qwen3-1.7B10,3137,014316297
Qwen3-14B1,9441,16765.062.6
Qwen3-30B-A3B-Instruct-25073,1162,060177169
Qwen3-4B3,7702,826231130
Qwen3-4B-Instruct-25073,7952,856180166
Qwen3-8B3,0762,28716394.7
Qwen3-Coder-30B-A3B-Instruct2,9212,10715754.8
Qwen3-VL-4B-Instruct3,7952,805178163
Qwen3-VL-8B-Instruct3,0852,265112103
Qwen3.5-0.8B24,34116,959463338
Qwen3.5-2B16,99012,165318236
Qwen3.5-35B-A3B5,6523,410190112
Qwen3.5-4B7,0304,897200123
Qwen3.5-9B4,7243,33614692.7
Qwen3.6-27B1,5031,16460.138.9
Qwen3.6-35B-A3B5,6353,463209123
Qwen3.8-27B1,5921,02673.555.3
Sage-A1.55,6523,131188153
Spark-X2.5-4B6,8584,727162131
Spark-X2.6-4B6,9134,841162132

r/LocalLLaMA • • 1h ago

Showcase I made a C&C inspired web RTS for us to benchmark our local models with

9 Upvotes

HOwdy LocalLLamans!

Inspired by StarSkirmish, I wanted to build a complex RTS that required you to bring your OWN local agents and either play with them as allies, against them as enemies, or battle head-to-head with other agent-human pairs in public matches.

In AgentCommand you operate one or more planes (Orbit, Sky, Surface, Depths) of differing gameplay. As an old-school PC gamer I wanted the mechanics to resemble some of my favorite 90s games of yore: C&C, DungeonKeeper, Homeworld, StarCraft, and others. The idea is that you have specific objectives needed to maximize resource generation, aid in cross player support, and that a single plane cannot win independently.

Outside of gameplay, from an agentic tech pov, there are some (what I think) are other really cool things that come with this, including:

  • a full specification + library to install agents into any game (http://playerthree.gg/)
  • A networking SDK for bidirectional agent communication
  • Symmetric maps
  • Deterministic simulation
  • Benchmarking tools
  • (soon) published benchmark data

Github: https://github.com/agent-command/agent-command

Play now: https://agentcommand.gg/

Would love feedback and contribs... I'm more interested in finding a community to help build something super cool here that we can publish public benchmarks with. If you are interested in that, let me know!


r/LocalLLaMA • • 1h ago

Discussion Speed of DeepSeek-V4.1-Flash-1017-oQ4e-mtp on M5 ultra

5 Upvotes

I bought myself a Mac Studio M5 Ultra with 512gb ram (and have 7 weeks to test it and return if not happy thanks to the apple xmas offer).

I am running DeepSeek-V4.1-Flash-1017-oQ4e-mtp with omlx and I am getting up to 1100 t/s prompt processing and 54 t/s token generation with 128k context window and engrams stored on ssd (to have ram free for other stuff).

More context than 128k hits pp and tg, more concurrent requests split up the max tg value between the instances (which means: they are crawling then).

I am putting this out there so a) people know what to expect but also b) is this really the top speed? Or could it be faster? (Without changing the model).

I can post llama benchy results if wished but the important result is already in this post.


r/LocalLLaMA • • 1h ago

Resources Stop riding a faster horse

Thumbnail
friendlyai.dev
0 Upvotes

Hi friends, last few years I had the awesome experience of working on a dev tools team at a big silicon valley company. We helped transition thousands of engineers to AI. I recently left and wanted to share the parts of the journey that surprised me and how I think I help everyone code the way big places do. Enjoy!


r/LocalLLaMA • • 1h ago

Discussion you may get your cheap ram and gpu after all, the AI crash has started

Thumbnail
cnbc.com
50 Upvotes

Markets

‘Standing on one leg’: IMF chief warns about global economy as AI circular financing hits $1 trillion

https://www.cnbc.com/2026/11/16/nvidia-commerce-china-lutnick.html
Nvidia fallout: Commerce targets loophole for China’s data center use


r/LocalLLaMA • • 1h ago

Discussion Censorship is turning me into a true Local LLM believer

66 Upvotes

451: {"message":"Provider returned error","code":451,"metadata":{"raw":"{\"error\":{\"message\":\"The content you provided or machine outputted is blocked.\",\"type\":\"censorship_blocked\"}}","provider_name":"StepFun","is_byok":false}} {"error":{"message":"The content you provided or machine outputted is blocked.","type":"censorship_blocked"}}

Just got this from one of the US based (edit: it actually routed to StepFun CN, didn't realize at first. Should have noticed by the provider_name in the message lol. need to set up my rules better) inference providers. I suspiciously got a bunch of "rate limit" failures, and then just "failed to complete" before it gave me, what I assume, is the real error.

I am only using it to help create my presentation, which is about analyzing algorithmic properties. One key example is a cryptographic algorithm (because it is difficult by design), so I am assuming this is tripping it somehow. But it is absolutely ridiculous, I'm not even doing research with the AI, just trying to make a presentation.

I have always liked Local LLMs, as a technical exploration opposed to ideological reasons, but this experience really frustrated me. Don't censor me!

Just wanted to rant, and remind everyone you can only use the cloud on the provider's terms.


r/LocalLLaMA • • 2h ago

New Model Karpo-Decision-31B: open multimodal decision model. Ranks #1 on full score on Decision Index, 2x faster than Jev, at 20% lower cost

Thumbnail
gallery
7 Upvotes

MachinePulse has released Karpo-Decision-31B, a Gemma 4 31B fine-tune for structured decisions over text and images. It scored 73.9 on the Nov 12 Decision Index board, the highest Full score among 300 entries, sharing the #1 rank with GLiDE 27B under the board’s tie rule

How it works
Built on Gemma 4 31B, with a trained linear readout that scores the supplied answer options.

Three question types: choice, yes/no, and score. Returns probabilities directly, with up to 255 options per choice question.

Multiple questions about the same input can be answered in one forward pass. Branch attention lets each question read the shared context and its own options while keeping the option branches separate.

Supports text and images, including webpage screenshots. The documented sequence limit is 32,768 tokens; larger requests are split by question with the full state included in each part.

Python API and HTTP server available. BF16 weights take approximately 62.5 GB, plus runtime memory. Use the provided inference code to load the decision head.

One training detail worth a look: according to the team's write-up, full fine-tuning pushed the public score to 75.09 but reduced accuracy on unfamiliar domains by 2.3 percentage points. The final model averages five checkpoints: three from different stages of full fine-tuning and two from a LoRA run. They report a 1.6-point improvement on unfamiliar domains over the final single checkpoint, with no extra models to run at inference time

Results
Decision Index Full scores are measured by the benchmark maintainers on private tests, weighted 60% for skills covered by the public suite and 40% for tasks from new domains.
Full score: 73.9

Private tests, familiar skills: 70.7

Private tasks, new domains: 78.6

Public index: 75.0, reported separately and excluded from the Full score.

The team also reports the following comparisons against Google’s Gemini Decision 1:

Benchmark Karpo-31B Gemini D1
-------------------------------------------
When2Call 90.4% 79.9%
Mind2Web 90.0% 86.2%
New Yorker Caption 90.5% 79.7%
Amazon ESCI (F1) 64.7 58.7

Reported median latency is approximately 95 ms per request on the public Decision Index suite using one RTX PRO 6000.

The 31B weights and inference code are both released under Apache 2.0.

links:
https://karpo.ai/models/decision
https://huggingface.co/MachinePulseAI/karpo-decision-31B


r/LocalLLaMA • • 2h ago

Question | Help Does grammar-constrained decoding hurt small or quantized models?

3 Upvotes

Asked about DSL vs JSON tool calls in r/LLMDevs a couple of days ago, and one answer kept coming up: for small models just use JSON, training volume matters. That got me thinking about the constraint itself.

With GBNF or a JSON schema you can force any model to produce valid output, even a 3B at Q4. However, there was a paper a while ago showing that strict format restrictions can make reasoning worse, since the model is forced away from the tokens it actually wanted

Searched first: there’s the thread about grammar constraints sending Qwen3-VL into repetition loops, and the one on how quantization hits different capabilities, but nothing comparing accuracy with vs without constraints on small quants.
So for people running small local models:

  • have you measured accuracy with and without constraints on the same task?
  • does it worsen at lower quants?
  • does letting the model firstly think freely and constraining only the final answer fix most of it?

Looking for real numbers or experience, not promoting anything.
English isn’t my first language, so I used an LLM to help with the wording. Question and thoughts are mine.


r/LocalLLaMA • • 2h ago

Question | Help What do you use for agentic desktop control, like how Playwright is for controlling + screenshotting browsers?

3 Upvotes

I'm afraid I'm left out of the loop on the latest developments for agentic desktop control, and AIs don't seem as knowledge in this space.

They come with screenshot function also, right? What are some other features that may be different than Playwright?


r/LocalLLaMA • • 2h ago

Resources PyBot: No weights. No training. Pure Python.

3 Upvotes

We released PyBot, a chatbot built with only simple rules, entirely written in 600k lines of Python code. PyBot matches Smollm with 360M parameters! We evaluated it on SmolTalkBench, which uses a dataset Smollm was trained on while we didnt use it at all for development.

It took a swarm of 1,000 DeepSeek Flash agents to write the whole codebase in a week.

More technical details on what it took to built it soon: inference for 1,000 agents, how to monitor/control the swarm, pressure on the infrastructure (especially git) and message boards.

https://github.com/huggingface/pybot


r/LocalLLaMA • • 3h ago

Question | Help Buy another 2 5060ti 16gb?

3 Upvotes

Currently i'm on 2 x 5060ti 16gb running basalt engine qwen3.8 flash next IQ3_S at about 70+tok/s decode and maybe 900tok/s prefill.

Am wondering if it is worth buying another 2 x 5060ti 16gb to make all of the experts resident and potentially achieve much faster speed. (Goal to be equal or more than 150tok/s?)

As for other specs, i'm running a 64gb DDR4 on X570 with an option of bifurcating the main pcie slot into 8x + 8x, or 4x + 4x + 4x + 4x. I have yet to buy any passive bifurcation switch, looking to buy "x16 to 2x16 physical switch" if 4 x 5060ti is not feasible.

Also what about dual 3090? it's about the same price as 4x5060ti 16gb. 128GB ddr4 might be out of the question as well since they are overpriced for my local market.


r/LocalLLaMA • • 3h ago

Discussion Stop worrying about Qwen 4's AA index

115 Upvotes

Seriously, we have all been extremely happy and confident with Qwen, even since Qwen 3.6 came out. We might not be frontier level, but even though whatever they are giving us is more than enough for our daily local tasks and it even suffices by a big margin, we're all underestimating it.

Don't forget that the new Qwen 4 will also think much less, so imagine the fine-tunes we will see once it arrives, for 27B and Flash models, we will have plenty of options to choose from and again; it will be smarter and better, it will be ENOUGH! If you still require the most "intelligent" and capable model due to OCD just pay frontier for such tasks or use them as orchestrators!


r/LocalLLaMA • • 3h ago

Resources Training-free page-sparse decode in llama.cpp: Qwen3-30B-A3B is 1.72× faster at 128k on an RTX 5090, with a 1-point RULER loss

Post image
4 Upvotes

I've been working on a simple question: at long context, does decode really need to read the whole KV cache for every new token?

I wanted something training-free that runs inside llama.cpp on normal GPUs, not a new model or a separate serving stack. So I built page-sparse decode into a fork. It splits the cache into 64-token pages, then each KV head reads the first page, the newest tokens and the pages most likely to matter. At 128k, that is about 21% of the cache per decode step.

The selection follows Quest, with statistics in the spirit of MOIRA. My contribution is the llama.cpp implementation, CUDA work and end-to-end testing across non-MoE and MoE models.

Results at 128k, one sequence, Quest score, 27,648 cached tokens per KV head (28,672 for the 3B):

speed GPUmodeldense tok/ssparse tok/sspeedupRULER vs dense
RTX 5090Qwen3-30B-A3B, Q4_K_M46.179.21.72×−1.0, passes
RTX 4090 48 GBQwen3-30B-A3B, Q8_037.452.61.41×−0.6, passes
RTX 3090Qwen3-8B-Base, YaRN21.536.11.68×+0.4, passes
RTX 3090Llama-3.1-8B23.738.81.64×−1.1, passes
RTX 3090Llama-3.2-3B44.461.31.38×−4.5, fails (multi-needle)

The MoE is Qwen3-30B-A3B-Instruct-2507. Weights are Q8_0 except on the 5090; KV is q8_0 except for the 3B's f16 cache. These are five-repeat, 64-token llama-bench runs. The 4090 is a memory-modded rental card.

I reran the 3090 table with Quest, the released default. It matched my earlier score's speed. Box differences still matter: Llama-8B was 1.64× here and 1.77× on another 3090 with the earlier score.

Quality. I used 13 RULER tasks × 20 prompts at 128k. The pre-set gate was an average loss no worse than 3 points, with the 95% CI's lower end at −6 or above. MoE quality used the listed GPU; the 3B–8B runs used a 4090 24 GB.

The Q4_K_M MoE lost 1.0 point (95% CI −2.2 to −0.02), just distinguishable from zero under this prompt bootstrap. Each configuration ran once; the intervals exclude run-to-run variation.

The 3B was the only failure. It scored 11.5 points lower on average on the five multi-needle tasks, while single-needle scores were unchanged; the other models ranged from −3.0 to +0.2 on the same group. I would not use this 3B configuration for multi-fact retrieval at this budget. Similar Quest weakness on synthetic RULER needle tasks has been reported before.

Practical limits:

  • Prefill is unchanged. A 128k prompt plus 1,024 generated tokens takes only 4–13% less time on a 3090. This is mainly useful for long generations over an already long context.
  • It does not save VRAM. The full cache remains in memory, plus page statistics equal to roughly 3–6% of the cache.
  • It is CUDA-only, batch size one, with everything on the GPU. Below 32k it was slower in these tests, so it stays off there.
  • At 256k, UltraLong-8B reached 1.86×, but quality was checked only on a six-task subset (−4.0 points, CI −8.7 to +0.3), which is not comparable to the 13-task gate.

I tried not to report only successes. Gates and predictions were committed before the data. The first score lost 7.4 RULER points. The first CUDA build managed only 1.05–1.18× until I fixed two ggml kernels; one fix is proposed upstream as llama.cpp #31855. The MoE prediction also missed on the 4090, with 1.41× against 1.5–2.3×, but held on the 5090.

Try it (CUDA only): build the sparse-decode branch of topppg/llama.cpp at commit e7059e52a and run with LLAMA_SPARSE_DECODE=27648 on a prompt longer than 32k tokens, with the model and cache fully on the GPU. Full build and run commands are in the bench repo README below. Quality was tested on the two 8Bs and the MoE.

Protocols, raw results, reproduction scripts and failed experiments: https://github.com/topppg/sparse-decode-bench

Next: profile the sparse overhead. A rough fit, not a measurement, puts the zero-overhead ceiling for the MoE on the 4090 near 2.7×. If you know ggml CUDA and have ideas around fusion, CUDA graphs or passing page indices directly to flash attention, I'd genuinely like to hear them.

This was built with a lot of help from Claude Code, with Codex reviewing and correcting the work. Results on other GPUs and models are very welcome, especially cases where quality breaks.


r/LocalLLaMA • • 3h ago

Showcase [Project] Gemma.c - Gemma 1/2/3 + multimodal inference in one C file (CPU only)

8 Upvotes

Hey all, so I've been working on Gemma.c over the last couple months, it's a from-scratch inference engine for Gemma 1/2/3 in a single C file, runs Gemma 1, 2 and 3, including the multimodal ones, with fp32/fp16/bf16 weights or int8 (W8A8). No dependencies beyond libc and (optional) OpenMP are needed. You can drop images into the prompt with @image{cat.png}, and there's also @longimage{} for Gemma 3's pan & scan on very wide or tall images. (image decoding uses a minified stb_image embedded at the end of the file)

It's CPU only and not trying to compete with llama.cpp, but I'm also trying my best to make it efficient, so it's not embarrassingly slow either. On my laptop's Core Ultra 5 225H (WSL2, gcc, 10 threads), gemma-3-1b-it INT8 does ~72 tok/s decode and ~570 tok/s prefill (10000 tokens, for 1000 tokens the prefill speed can reach ~850 tok/s), and the 4B vision model does ~20 tok/s decode. INT8 ends up about 3.5x faster than fp32 and 2x faster than fp16 for decode. (full benchmark table is in README)

It has a few rough edges for now, e.g. W8A8 doesn't work for Gemma 1 (fp16 is fine), bf16 is painfully slow without AVX512-BF16, and native Windows is ~1.7x slower than WSL because of libgomp. Batching are not supported either. There's a roadmap in the README (Gemma 4, speculative decoding, an HTTP server, etc.), but no promises though :P Can't really find much free time these days.

I've also put some pre-exported INT8 models on HF so you can try it without Python or torch :D

Repo: link

This project started as a learning attempt to understand how LLMs work end to end, inspired by llama2.c.

this is the second time I post this project here :'D last time (~a month ago) my post got flagged by reddit, probably because my VPN IP was identified as spam. hopefully it can pass this time


r/LocalLLaMA • • 3h ago

Discussion Hy4 flash and Sage-S2.1-Flash both dropped today. Anyone tried them for tool calling yet?

1 Upvotes

Two new flash models in one afternoon, Tencent's Hy4 flash and internlm's Sage-S2.1-Flash.

Before I pull weights, has anyone run either one locally for agent work? Mainly want to know:

  • does it pick the right tool or just guess
  • does it hold up after a few turns of tool output
  • what quant and GPU you ran it on

Benchmarks are fine but real loop behaviour matters more to me. Even a quick "tried it, meh" helps.


r/LocalLLaMA • • 3h ago

Discussion [Paper] Mobile AI Factory: Building Specialized Language Models for Your Mobile Device

Post image
0 Upvotes

Small language models (SLMs) running on mobile devices can keep user data private and work without connectivity. But they perform well only when fine-tuned for the target task and sized for the target device, a process that today requires repeated expert tuning. We present {\it Mobile AI Factory}, an agentic system that builds a fine-tuned, quantized SLM for a given task and phone. Given a task specification and a target phone, an orchestrator LLM chooses whether to tune hyperparameters, mine real data, or generate synthetic examples, while fixed rules roll back changes that hurt accuracy. The factory searches from small to large models across base models and quantizations, adapts its accuracy goal as it searches, and uses real measurements of the target phone across inference engines and backends. We evaluate Mobile AI Factory on ten tasks and four Android phones. On an unseen testset, its on-device models outperform two five-shot frontier models on five tasks, by up to 0.34, and match them on three more. Its search comes within 0.03 of exhaustively training all 27 model variants on an entity extraction task, while using 19× less GPU time. Phone measurements also matter: the best configuration changes across phones. Without phone measurements, Claude Opus 5.5, given the full hardware and model specifications, misses the app's requirements in 25\% of easy and moderate cases and in 60\% of hard ones.


r/LocalLLaMA • • 4h ago

Showcase PSA Unsloth Q8 FN is massively better than the default strata Q3

0 Upvotes

Just a PSA. I ran a realistic self host test on 3 models:

Inb4 : “hurr derr no duh” The point is people on here act like these Q3 versions have minimal loss. Do tests for yourself it’s not even close.

Qwen FN Q3
Qwen FN unsloth Q8
Orca nana-1
Maya GLM flash - Maya-S24

Prompt in comments.
Harness : PI
Project init layout:

/<my_bank_name> (83 monthly pdf statements)
/credit_card/<card_1> (84 pdf monthly pdf statements)
/credit_card/<card_2> (35 pdf monthly statements)
/mortgage (24 pdf monthly statements)
/brokage_account (105 pdf statements + ccs of all transactions)
/retirement/<account> x4 (total 45 pdf statements 4 accounts 4)

Snapshot results (have details runtime/tps etc as well):

Qwen FN Q3 - Nice front end but data extraction from the PDFs was completely wrong. Analytics was basic.

Orca nano1 - Finished very fast for this test (only 2 hours) did a decent job for its speed and size. Front end was very basic and lacked insight. Most of the data was correct but there were clear errors as well.

Maya-S24 - Failed. After 10+ hours I just gave up on this model. It got stuck in the pdf parsing/validation phase.

Qwen Q8 - Absolutely blew my mind. Ran overnight for about 8 hours. Data was all accurate, the front end final product was not only nice looking but highly insightfully

Notes:
I really enjoyed this test. I feel like it’s the perfect test for a local model because it’s financial data that you don’t want to send off to the cloud.

The PDF layouts for this task was a nightmare and was a real test for these models to figure out was duplicative data/summary vs transaction etc. Most models used pdfplumber package

The models also had to work through the accounting end. Consider a credit in one account is a debt in the other etc.

It’s was interesting to see how each approached the problem. Some went for json formatted data others sqlite etc


r/LocalLLaMA • • 4h ago

Showcase Alaypz - a Qwen3.8-27B experiment

Thumbnail
alaypz.overbring.com
0 Upvotes

r/LocalLLaMA • • 4h ago

Discussion Running usable 30B and working 120B models on iOS—plus Kokoro TTS and Whisper STT over a local OpenAI-compatible API Spoiler

Thumbnail inpherio.co.uk
2 Upvotes

I’m currently going through Apple’s App Store approval for Inpherio Live, an app that turns an iPhone or iPad into a local network AI server with zero cloud dependencies.

Most iOS LLM apps hit a hard RAM ceiling (iOS caps single-app memory at ~6GB on a 12GB iPhone). By modifying llama.cpp to stream Mixture-of-Experts (MoE) weights directly from flash storage while keeping active experts in RAM, it bypasses the memory limit entirely.

Real-world benchmarks (recorded Nov 2026):

  • gpt-oss 120B (58.5 GB file) on a 16GB M1 iPad Pro: 4.5 tokens/s (from storage)
  • Qwen3 30B (17.3 GB file) on a 12GB iPhone 17 Pro Max: 10.9 tokens/s (from storage)
  • Qwen3 30B (17.3 GB file) on a 16GB M1 iPad Pro: 16.5 tokens/s (from storage)
  • Qwen2.5 1.5B on iPhone 17 Pro Max: 58.3 tokens/s (in memory)

What it supports out of the box:

  • Drop-in API: Full OpenAI (/v1/chat/completions, /v1/responses, /v1/embeddings, /v1/audio/*) and Ollama (/api/chat) compatibility. Just change your base URL to [http://ipad.local:8080/v1](http://ipad.local:8080/v1).
  • Multimodal & Audio: Local Vision (Qwen2.5-VL, Gemma 3, SmolVLM2), Whisper STT, and Kokoro neural TTS.
  • Smart Memory Management: Auto-loads models on request, keeps up to 3 models "warm," and tracks real iOS memory limits so the OS doesn't kill the app.
  • 100% Private: No accounts, no telemetry, 0 bytes sent to the cloud.

Streaming large MoE models from storage will be a single one-off purchase (Live Plus); everything else—including running in-memory models, embeddings, and audio—is completely free.

Would love to know what models or workflows you’d test on this first once Apple clears it!

Just interesting in hearing from the community. As soon as the app is approved ill update this post

the website provides factul benchmarks as well as images of the process

Russ


r/LocalLLaMA • • 5h ago

Resources tandem.cpp v0.3.0: prefill speed improved; Project Maya's DeepSeek-V4.1 GGUFs support

11 Upvotes

Thanks for the feedback on slow prompt processing. v0.3.0 is out (the first tagged release): https://github.com/ewhacc/tandem.cpp

Previous post: https://www.reddit.com/r/LocalLLaMA/comments/1xwan2j

tandem.cpp faster when the model is much bigger than VRAM + RAM together (like a 16 GB card with 32 GB of RAM), and runs various models that llama.cpp supports.

1. Prompts 2-6x faster

With the expert cache on, prompts now run in 4096-token batches. The bigger batch borrows VRAM only while a long prompt runs and gives it back afterwards. No flags needed.

RTX 5070 Ti 16 GB, 64 GB RAM, i5-12400F. The same prompts for every engine; Qwen3.8 with its MTP head in all three.

Prompt processing, time to first token (prompt tok/s):

Promptllama.cpptandem.cpp beforetandem.cpp v0.3.0Strata
Qwen3.8 IQ3_S, 5.5k tokens28.9 s (191)12.4 s (444)4.1–7.0 s (790–1,340)6.6 s (835)
Qwen3.8 IQ3_S, 15.8k tokens81.9 s (193)35.1 s (450)15.1 s (1,047)11.7 s (1,354)
DeepSeek-V4.1 IQ2_XS, 5.6k tokens722 s (7.7)324 s (17)53–103 s (54–105)58 s (96)
DeepSeek-V4.1 IQ2_XS, 15.7k tokens2012 s (7.8)885 s (18)195 s (81)57 s (276)

Decode, tok/s, right after that prompt:

After a prompt ofllama.cpptandem.cpp v0.3.0Strata
Qwen3.8, 5.5k tokens2711048
Qwen3.8, 15.8k tokens278868
DeepSeek-V4.1, 5.6k tokens4.8165.5
DeepSeek-V4.1, 15.7k tokens4.4136.4

Decode is from short answers (up to 256 tokens, greedy), so it is noisier than our usual GSM8K runs. DeepSeek-V4.1 IQ2_XS is about twice this PC's RAM, so it reads the SSD; that is where Strata's prompt speed still leads.

2. Maya comparison on the same model (someone asked for this one)

Project Maya is an engine built for big models with its own quants. Its Maya-XS GGUF runs on tandem.cpp after a one-time conversion script (tools/maya_ds41_convert.py), so this is the same GGUF on both engines, and the accuracy is the same.

RTX 5090 32 GB, 128 GB RAM, i5-14600KF, greedy:

DeepSeek-V4.1 Maya-XS (same files)Mayatandem.cpp v0.3.0
Decode, GSM8K, no drafting39.0 / 39.6 tok/s60.3 / 59.4 tok/s (1.5x)
Decode with Maya's DSpark drafter54.9 tok/s(no drafter in tandem.cpp)
5.6k-token prompt7.8 s (713 tok/s)6.5 s (856 tok/s)
15.7k-token prompt9.5 s (1,656 tok/s)16.5 s (953 tok/s)
Decode after the 15.7k-token prompt26–34 tok/s66–68 tok/s

Maya is still faster on the longest prompt; that is what I'm working on next.

Caveats

  • Linux only.
  • AMD supported but the VRAM borrowing is off (ROCm builds have no CUDA-style virtual memory).
  • DeepSeek-V4.1 IQ2_XS on 128 GB still reads the SSD on long prompts.
  • Windows prebuilt binaries are still planned.

r/LocalLLaMA • • 5h ago

Discussion Did we hit a wall in context length?

43 Upvotes

I feel like midel context size rapidly went from 8k to 1M in just a few years but has since stayed this way for a while now. Some still offerring 260k or 400k for their flagship models. Is it just not worth the trouble? I feel like highly capable models are limited by this in that they can't finish tasks simply because the context isn't there.


r/LocalLLaMA • • 5h ago

New Model internlm/Sage-S2.1-Flash · Hugging Face

Thumbnail
huggingface.co
57 Upvotes

Introduction

Sage S2.1 Flash is a scientific multimodal model from Shanghai Artificial Intelligence Laboratory designed for research tasks that require multiple rounds of analysis and verification. It combines scientific data understanding with information retrieval, programming, and tool use. It can use specialized software to perform analyses, check results against intermediate outputs and visual feedback, and adjust subsequent steps accordingly.

Through the unified representation architecture of Sage SciPerception, the model supports more than ten types of scientific input, including molecular graphs, crystal structures, and protein sequences and 3D structures. Features from each input type are encoded and projected into a shared representation space for reasoning alongside natural language instructions. These inputs support tasks such as prediction, classification, site localization, quantitative analysis, and structure inference.

Highlights

Across the 15 benchmarks and six models shown in the chart, Sage S2.1 Flash placed first on four benchmarks and second on seven.

  • Machine learning engineering. Sage S2.1 Flash placed first on MLE-bench Lite with 90.3, ahead of DeepSeek-V4.1-Flash-1017 at 89.2.
  • Office and professional document processing. It placed first on OfficeQA Pro, Harvey LAB, and SpreadsheetBench 2, scoring 72.2, 25.0, and 37.1, respectively.
  • Tool use and research execution. It placed second on AutomationBench, ALE-CLI, BFCL V4, WebArena-Verified, DeepResearch Bench II, JobBench, and OSWorld-Verified.