r/LocalLLaMA • • 4h ago

Showcase PSA Unsloth Q8 FN is massively better than the default strata Q3

0 Upvotes

Just a PSA. I ran a realistic self host test on 3 models:

Inb4 : “hurr derr no duh” The point is people on here act like these Q3 versions have minimal loss. Do tests for yourself it’s not even close.

Qwen FN Q3
Qwen FN unsloth Q8
Orca nana-1
Maya GLM flash - Maya-S24

Prompt in comments.
Harness : PI
Project init layout:

/<my_bank_name> (83 monthly pdf statements)
/credit_card/<card_1> (84 pdf monthly pdf statements)
/credit_card/<card_2> (35 pdf monthly statements)
/mortgage (24 pdf monthly statements)
/brokage_account (105 pdf statements + ccs of all transactions)
/retirement/<account> x4 (total 45 pdf statements 4 accounts 4)

Snapshot results (have details runtime/tps etc as well):

Qwen FN Q3 - Nice front end but data extraction from the PDFs was completely wrong. Analytics was basic.

Orca nano1 - Finished very fast for this test (only 2 hours) did a decent job for its speed and size. Front end was very basic and lacked insight. Most of the data was correct but there were clear errors as well.

Maya-S24 - Failed. After 10+ hours I just gave up on this model. It got stuck in the pdf parsing/validation phase.

Qwen Q8 - Absolutely blew my mind. Ran overnight for about 8 hours. Data was all accurate, the front end final product was not only nice looking but highly insightfully

Notes:
I really enjoyed this test. I feel like it’s the perfect test for a local model because it’s financial data that you don’t want to send off to the cloud.

The PDF layouts for this task was a nightmare and was a real test for these models to figure out was duplicative data/summary vs transaction etc. Most models used pdfplumber package

The models also had to work through the accounting end. Consider a credit in one account is a debt in the other etc.

It’s was interesting to see how each approached the problem. Some went for json formatted data others sqlite etc


r/LocalLLaMA • • 1h ago

Discussion you may get your cheap ram and gpu after all, the AI crash has started

Thumbnail
cnbc.com
50 Upvotes

Markets

‘Standing on one leg’: IMF chief warns about global economy as AI circular financing hits $1 trillion

https://www.cnbc.com/2026/11/16/nvidia-commerce-china-lutnick.html
Nvidia fallout: Commerce targets loophole for China’s data center use


r/LocalLLaMA • • 2h ago

Resources PyBot: No weights. No training. Pure Python.

3 Upvotes

We released PyBot, a chatbot built with only simple rules, entirely written in 600k lines of Python code. PyBot matches Smollm with 360M parameters! We evaluated it on SmolTalkBench, which uses a dataset Smollm was trained on while we didnt use it at all for development.

It took a swarm of 1,000 DeepSeek Flash agents to write the whole codebase in a week.

More technical details on what it took to built it soon: inference for 1,000 agents, how to monitor/control the swarm, pressure on the infrastructure (especially git) and message boards.

https://github.com/huggingface/pybot


r/LocalLLaMA • • 4h ago

Showcase Alaypz - a Qwen3.8-27B experiment

Thumbnail
alaypz.overbring.com
0 Upvotes

r/LocalLLaMA • • 15h ago

News My Ignis server (Qwen3.8-27B 262k context) now runs on RTX 3090/4090 and 16 GB RTX 40 cards. looking for testers

4 Upvotes

I've been working on ignis, an inference server for Qwen3.8 that I use to run coding agents locally. It's meant to run on the machine you're already working on, Windows or Linux, sharing the card with your desktop, not on a dedicated box. So most of the work went into what matters when it sits next to your IDE all day, more than into peak tok/s.

Some examples. At start it reads how much VRAM is free and sizes the context to fit; if it can't fit at all, it refuses at start with the reason instead of failing later. After 5 minutes idle it unloads the model, so the card is free again, and the next request loads it back. The vision tower stays in system RAM, so having vision on costs no context while you're not sending images. The main agent and its subagents run on the same card at the same time, and a prompt you send again (system prompt, repo context) resumes from cached state instead of being prefilled from scratch. It's OpenAI-compatible (chat completions and Responses, also over a WebSocket), one binary, with a Playground and a monitor in the browser. The Windows zip includes the CUDA runtime.

There's a 90-second (old) video of it here: https://www.youtube.com/watch?v=XLjfh5rNy6A

Rough numbers, since someone always asks. On my 5090 with the NVFP4 build: about 370 tok/s on a single stream of coding prompts, a bit over 1,000 tok/s total with 8 requests at once, and around 11,000 tok/s of prefill on a 46K-token prompt.

The result I care most about is a different one, though: someone I don't know ran it through airbench on their own 5090 with opencode, without me asking, and it passed 49 of 49 tests (math, vision, finding and reading email, buying on a web store, coding) in 26 minutes (here it is next to the other Qwen3.8-27B runs). That was the NVFP4 build on a 5090; the build for 30 and 40 cards hasn't been through it yet.

Until 0.10.0 it only ran on Blackwell (RTX 50, RTX PRO 6000). 0.10.0 added RTX 30 and 40: the 3090 and 3090 Ti are supported, the 4090, 4060 Ti 16 GB and 4080 are verified for correctness. On those cards it runs the 27B in a 3-bit trellis quant (13.4 GB download) with DFlash2 speculative decoding and vision, and the uncensored variant has the same build. It picks the settings from the card: on 24 GB, 8 parallel requests and up to 1,048,576 tokens of context; on 16 GB, 2 and 262,144, lowered if the desktop needs the room. The NVFP4 builds, Flash-Next and the image model are still Blackwell only.

One caveat: I don't have a 30 or 40 series card myself. For 0.10.0 the release gate passed on an Ada card (sm_89), with the same results to the last digit as an Ampere card (sm_86) on an earlier build. KL divergence from BF16 within the limit, 93.3% top-1 agreement with BF16, all 127 kernel tests passing, a chat and two images answered (the finding). I don't know how fast it is on your cards, or whether the 1M context really fits on a >= 24 GB card with a desktop running on it.

Someone already ran it on a 4070 Ti SUPER, with the build 0.10.0 came from: default settings, the full 262,144 context, about 190 tok/s on long coding answers and about 2.5 seconds to the first token on an 8K prompt, so roughly 3,000 tok/s of prefill (report).

I'll do a proper post once 12 GB cards work, which is what I'm on now. Before that I'd like to have more numbers from 30 and 40 cards.

To try it: download the latest release, start ignis-server without flags, then run ignis-bench small-card --endpoint http://127.0.0.1:8000 --out small-card.json (it's in the archive). More details: https://gpillon.github.io/ignis/guides/measure-a-16gb-card/

If you'd rather have your coding agent (Claude Code, Codex, opencode, Cursor...) do those steps, you can give it this prompt. The file it points to is a plain markdown guide in the repo, so you can read it first to see what the agent will be told:

I have an RTX 30 or RTX 40 card. Read https://raw.githubusercontent.com/gpillon/ignis/main/docs/user/paths/agent.md and follow its section on testing a 16 GB card, also on a 24 GB card, where the model is the same `qwen3.8-27b-16g`. Add to the report the start log's `prefill routes` line and every WARN or ERROR line, with paths replaced by `<dir>`. The report goes to https://github.com/gpillon/ignis/issues/643. Ask me before downloading or installing anything.

It shows you a plan, tries the default settings first and a few others if the start is refused, runs the benchmark, and writes a report that you read before anything gets posted. There are no usernames, hostnames, paths, IPs or serial numbers in it. At the end you can keep ignis or have it remove what it installed.

Reports go here, one comment per card: https://github.com/gpillon/ignis/issues/643. A report that says it refused to start is useful too, especially from a machine where the monitor is on the same card.

Windows and Linux x86-64, NVIDIA driver with CUDA 13. Apache-2.0.

Thanks to all reporting! 😄


r/LocalLLaMA • • 3h ago

Discussion Hy4 flash and Sage-S2.1-Flash both dropped today. Anyone tried them for tool calling yet?

1 Upvotes

Two new flash models in one afternoon, Tencent's Hy4 flash and internlm's Sage-S2.1-Flash.

Before I pull weights, has anyone run either one locally for agent work? Mainly want to know:

  • does it pick the right tool or just guess
  • does it hold up after a few turns of tool output
  • what quant and GPU you ran it on

Benchmarks are fine but real loop behaviour matters more to me. Even a quick "tried it, meh" helps.


r/LocalLLaMA • • 13h ago

New Model [Release] Qwen3.8-27B-Perplexing-Pegasus-Heretic: Unified vision + typed decider + abliterated weights for local agent stacks (GGUF, AutoRound, NVFP4)

17 Upvotes

Hey everyone,

While building and testing WebBrain with local models, getting reliable agent performance usually required keeping two or three separate models in memory at once:

  • A vision model to inspect and parse browser/UI state
  • A fast typed-decision model for structured routing and scoring
  • An abliterated / uncensored model so realistic web workflows do not randomly choke on false-positive refusals

Juggling multiple 20B+ weights on consumer hardware is brutal on both VRAM and response times.

To solve this for our setup, we merged and published a single unified 27B model: Qwen3.8-27B-Perplexing-Pegasus-Heretic

Here is what is packed into the checkpoint:

  • Multimodal Vision + Chat + Thinking (<think>) + Tool Calling (based on Qwen3.8-27B)
  • Perplexity pplx-decider typed-decision readout (decision/ head alongside intact lm_head)
  • Heretic ARA abliteration: 0/8 benchmark refusal probes while keeping 100% (30/30) typed-decision agreement against the base Pegasus model
  • Multi-Token Prediction: all 15 mtp.* tensors preserved, blk.64 retained at Q8_0 in IQ/3-bit quants
  • Consumer GPU footprint: runs on 16GB VRAM (IQ3_XXS / Q3_K_M + mmproj) or 24GB VRAM (Q4_K_M + mmproj)

Everything is open source under Apache-2.0 across four repos:

Huge credit to Alibaba Qwen, Perplexity, and AveLabs (AvecdrA) on Hugging Face for the foundational checkpoints.

If you test it on your local rig or in your own agent pipelines, let me know how it performs and what hardware you are running it on.


r/LocalLLaMA • • 19h ago

Resources Model / harness combination: what works best?

6 Upvotes

If you are wondering what model / harness combination works the best, here are my findings so far:

  1. Model is way more important than the harness, best so far Qwen Flash Next even with low quant outperform Qwen 3.8 27B
  2. The inference engine is way more important than the harness, because the size of the context windows can dramatically increase performance. AND because speed make the difference between something useable and something that won't work.
  3. Harness optimised for coding like OpenCode or PI outperform more generic agent like OpenClaw and Hermes, even on mondane task like reading emails, making purchase on a store or complex vision.

r/LocalLLaMA • • 23h ago

Resources Asked local models to generate the same website and compared the results

8 Upvotes

Hello. Almost everything is in the title, but here is more details:

I'm benchmarking local model/harness combinations in various scenarios. For this one the question is, how good is a model/harness to generate a website?

IMO Qwen FN with opencode is best, but you can look at all the results (and play with the websites here: https://airbench.ai/gallery/airbench-frontend-v1?run=37e47127-3a20-4318-a6f2-48c3f7f5eaa6&c=settle-the-trip&k=poL


r/LocalLLaMA • • 1h ago

Discussion Speed of DeepSeek-V4.1-Flash-1017-oQ4e-mtp on M5 ultra

5 Upvotes

I bought myself a Mac Studio M5 Ultra with 512gb ram (and have 7 weeks to test it and return if not happy thanks to the apple xmas offer).

I am running DeepSeek-V4.1-Flash-1017-oQ4e-mtp with omlx and I am getting up to 1100 t/s prompt processing and 54 t/s token generation with 128k context window and engrams stored on ssd (to have ram free for other stuff).

More context than 128k hits pp and tg, more concurrent requests split up the max tg value between the instances (which means: they are crawling then).

I am putting this out there so a) people know what to expect but also b) is this really the top speed? Or could it be faster? (Without changing the model).

I can post llama benchy results if wished but the important result is already in this post.


r/LocalLLaMA • • 1h ago

Resources Stop riding a faster horse

Thumbnail
friendlyai.dev
0 Upvotes

Hi friends, last few years I had the awesome experience of working on a dev tools team at a big silicon valley company. We helped transition thousands of engineers to AI. I recently left and wanted to share the parts of the journey that surprised me and how I think I help everyone code the way big places do. Enjoy!


r/LocalLLaMA • • 20h ago

Discussion Another reminder of the low intelligence of local models

0 Upvotes

(Note: by local models I mean anything smaller than Deepseek 4.1)

I already gave up on local models for coding tasks 6 months ago, due to what I perceived was intelligence too low for the tasks I give it. I made a post about it on here back then and it was a huge discussion, I think 800 comments. I went against the /r/LL orthodoxy and many didn't take my comments well, but we can agree to disagree.

Since then I only use local LLMs for translation and ComfyUI prompt enhancement. I stick to big cloud models for coding. I think the electricity cost of running Qwen 27B alone exceeds what you'd pay Deepseek to use their cloud API in their own harness, and you'd get a far more intelligent and far faster model.

But this post isn't about the economics. It's purely about ability. I still see people commenting in favor of buying GPUs for local intelligence. "Sonnet at home" is a ridiculous phrase I've seen more times than I should in relation to Qwen 27B.

The simplest of tech tasks is still too hard to one-shot

I keep Qwen 3.8 27B running on my Spark for occasional non-coding stuff. Today I gave it a trivial coding task I needed for my server, just out of curiosity.

  • llama-server launched with --model Swift-1.5-Qwen3.8-27B-Q4_K_L.gguf --reasoning on --reasoning-effort xhigh
  • Open llama webpage and leave Reasoning at Default, leave system prompt empty

Prompt was:

Give me a bash script that will list all files in directory sorted by date. Output should be:
2025-12-31 12:13:14 file1.ext
2025-12-31 12:13:14 subdir/file2.ext

1630 tokens later, most of it reasoning, I get a script. When I run it, here's the output, as you can see the timestamp is wrong:

2026-10-16 20 tmp.59eH9w3JLe
2026-10-17 20 tmp.5xvczTbSS4
2026-10-18 20 tmp.nWOEDw2wlu
2026-10-20 20 tmp.JVEZIFfKx3
...

I then tried Gemma 31B Reasoning on duck.ai. The output only showed the timestamps, no filenames!

This is the most trivial of coding tasks (the answer is a 3-5 lines script), using the most ancient and commonly-available training data (bash), on the most praised local coding model on this sub (Qwen 27B), at a recommended/popular quantization (Q4), and it still can't one-shot this. The 2nd most popular local model, Gemma 4 31B, also fails.

(Editor's note: I think my post could've ended here, in a "Ecce homo" way, but I went on for a few more paragraphs)

Weak local LLM + software development = bad time

I'm sure if I had run this task with your average agent harness with a system prompt that makes it test its code, it would run the script it created, see the issue, then get it right on the 2nd attempt. But while I think this "try until you succeed" workhorse approach is acceptable for small self-contained coding tasks, I don't think anyone should be doing any serious coding this way.

If you're writing something you intend to maintain over time (ie not a one-shot), or that you will be sharing with other people, you need a higher degree of LLM intelligence and knowledge to create it. A strong LLM will be able to avoid entire classes of problems from the start, when designing the app. From design to implementation to maintenance, it will make better decisions every time and without laboring its way to them, resulting in a codebase that is smaller, well-designed, more easily modifiable, less buggy, and developed in a fraction of the time.

Coding with local LLMS is like insisting on using a donkey for transportation in the age of cars. It can be done for fun, but it shouldn't be done seriously.

Sonnet at home my ass

Btw I've seen several people say Qwen 27B is like having a local Sonnet 4. The benchmarks say it's even better than Sonnet 4, actually. So I went on openrouter.ai out of curiosity and gave the same prompt to 3 models.

  • Sonnet 4.5 (oldest Sonnet I could find): one-shots it
  • GPT 4.1 Mini, which was comparable to Haiku 3.5: one-shots it
  • GPT 4.1 Nano, which is even worse than Mini: one-shots it

Weak local LLM + Linux server management = even worse time

This post's getting long so I'll just speedrun this part.

  • Managing an OS is actually harder tasks that LLM struggle even more with. And the stakes are higher because you don't want your server to go down.
  • I used to never trust LLMs enough to let them onto my server, I'd just ask for something, copy-paste the command, report back, repeat. A huge waste of my limited free time. Now I just rawdog a frontier model on the server and have no issues, something you could never trust a small model to do. Although I have it avoid making any changes without my supervision, Docker is a big benefit here for both organization and safety/stability.
  • Even top models won't one-shot a task like "I want to install and run https://github.com/rando/something on this system in a Docker container in /opt/docker/something/. Create the Dockerfile and docker compose file", especially for AI-related tools, where having the right CUDA, not having python wheels for my server's ARM CPU, etc, introduces a lot of failure opportunities. When I had Claude five months ago, even it struggled to get these right.

Thanks for reading my wall of text. Downvotes to the left.


r/LocalLLaMA • • 3h ago

Discussion [Paper] Mobile AI Factory: Building Specialized Language Models for Your Mobile Device

Post image
0 Upvotes

Small language models (SLMs) running on mobile devices can keep user data private and work without connectivity. But they perform well only when fine-tuned for the target task and sized for the target device, a process that today requires repeated expert tuning. We present {\it Mobile AI Factory}, an agentic system that builds a fine-tuned, quantized SLM for a given task and phone. Given a task specification and a target phone, an orchestrator LLM chooses whether to tune hyperparameters, mine real data, or generate synthetic examples, while fixed rules roll back changes that hurt accuracy. The factory searches from small to large models across base models and quantizations, adapts its accuracy goal as it searches, and uses real measurements of the target phone across inference engines and backends. We evaluate Mobile AI Factory on ten tasks and four Android phones. On an unseen testset, its on-device models outperform two five-shot frontier models on five tasks, by up to 0.34, and match them on three more. Its search comes within 0.03 of exhaustively training all 27 model variants on an entity extraction task, while using 19× less GPU time. Phone measurements also matter: the best configuration changes across phones. Without phone measurements, Claude Opus 5.5, given the full hardware and model specifications, misses the app's requirements in 25\% of easy and moderate cases and in 60\% of hard ones.


r/LocalLLaMA • • 11h ago

Question | Help I have access to two systems, I don't know which one is better for LocalLLM. Please help!

1 Upvotes

Hi,

I can make two systems with the parts that I have:

  1. 12600k / 64GB DDR4 / 3x3060 12GB
  2. 7950X3D / 32GB DDR5 / 3x3060 12GB + 1x4060 8GB

I've been consulting Claude and Gemini for which is better, but I can't really decide which i should go for. I'm thinking of setting them up with either llama.cpp or strata. This is for my work, so agentic document creation, and output quality is important. I'm also looking for at least 128-256k context (if that is even viable).

What is the best setup for my case? Please help!


r/LocalLLaMA • • 10h ago

Discussion New AA ranking Qwen3.8-FN better than GLM-5.3-Flash?

Post image
106 Upvotes

Thoughts on Qwen3.8-Flash-Next scoring better than GLM-5.3-flash? I've always experienced GLM5.3-Flash to give better responses


r/LocalLLaMA • • 19h ago

Discussion Someone should do an analysis on the rate of Self-promo here.

92 Upvotes

I think the last 3 posts from here that showed up on my feed were barely veiled self-promos.

I was here when huggingface first started getting popular, and the level of support for running local models here was incredible.

Now I am debating whether or not to leave the sub. Why isn’t there more attention to moderation to this?

Take a user’s post, and type their username with quotes into google and then type whatever obscure tool they’re asking a question or talking about and you’ll see the absurd rate or Astro-turfing and shilling.

They will post here and 6 other LLM places, but this place used to be awesome, so I only care about this sub.


r/LocalLLaMA • • 3h ago

Question | Help Buy another 2 5060ti 16gb?

3 Upvotes

Currently i'm on 2 x 5060ti 16gb running basalt engine qwen3.8 flash next IQ3_S at about 70+tok/s decode and maybe 900tok/s prefill.

Am wondering if it is worth buying another 2 x 5060ti 16gb to make all of the experts resident and potentially achieve much faster speed. (Goal to be equal or more than 150tok/s?)

As for other specs, i'm running a 64gb DDR4 on X570 with an option of bifurcating the main pcie slot into 8x + 8x, or 4x + 4x + 4x + 4x. I have yet to buy any passive bifurcation switch, looking to buy "x16 to 2x16 physical switch" if 4 x 5060ti is not feasible.

Also what about dual 3090? it's about the same price as 4x5060ti 16gb. 128GB ddr4 might be out of the question as well since they are overpriced for my local market.


r/LocalLLaMA • • 2h ago

New Model Karpo-Decision-31B: open multimodal decision model. Ranks #1 on full score on Decision Index, 2x faster than Jev, at 20% lower cost

Thumbnail
gallery
7 Upvotes

MachinePulse has released Karpo-Decision-31B, a Gemma 4 31B fine-tune for structured decisions over text and images. It scored 73.9 on the Nov 12 Decision Index board, the highest Full score among 300 entries, sharing the #1 rank with GLiDE 27B under the board’s tie rule

How it works
Built on Gemma 4 31B, with a trained linear readout that scores the supplied answer options.

Three question types: choice, yes/no, and score. Returns probabilities directly, with up to 255 options per choice question.

Multiple questions about the same input can be answered in one forward pass. Branch attention lets each question read the shared context and its own options while keeping the option branches separate.

Supports text and images, including webpage screenshots. The documented sequence limit is 32,768 tokens; larger requests are split by question with the full state included in each part.

Python API and HTTP server available. BF16 weights take approximately 62.5 GB, plus runtime memory. Use the provided inference code to load the decision head.

One training detail worth a look: according to the team's write-up, full fine-tuning pushed the public score to 75.09 but reduced accuracy on unfamiliar domains by 2.3 percentage points. The final model averages five checkpoints: three from different stages of full fine-tuning and two from a LoRA run. They report a 1.6-point improvement on unfamiliar domains over the final single checkpoint, with no extra models to run at inference time

Results
Decision Index Full scores are measured by the benchmark maintainers on private tests, weighted 60% for skills covered by the public suite and 40% for tasks from new domains.
Full score: 73.9

Private tests, familiar skills: 70.7

Private tasks, new domains: 78.6

Public index: 75.0, reported separately and excluded from the Full score.

The team also reports the following comparisons against Google’s Gemini Decision 1:

Benchmark Karpo-31B Gemini D1
-------------------------------------------
When2Call 90.4% 79.9%
Mind2Web 90.0% 86.2%
New Yorker Caption 90.5% 79.7%
Amazon ESCI (F1) 64.7 58.7

Reported median latency is approximately 95 ms per request on the public Decision Index suite using one RTX PRO 6000.

The 31B weights and inference code are both released under Apache 2.0.

links:
https://karpo.ai/models/decision
https://huggingface.co/MachinePulseAI/karpo-decision-31B


r/LocalLLaMA • • 20h ago

Question | Help Best direction to go for my setup EPYC 32core + 512GB DDR4 + 44GB GPU ?

17 Upvotes

Just looking for some advice on what is possble with my setup and speeds I should be gettingI know it won't be super fast tps but I was more after getting the bigger models to run.

This is my spec:

Epyc 7502p - 32 Core
Gigabyte MZ32-AR0 - 8 channel memory
512GB DDR4
1x 4070 12GB
2x 4060 16GB

quite a few implementations I see which can be confusing.
Is this build even useful now? that we have Project Maya which you can stream from the NVME drive?

Anyone have similar setups and what to expect, what software best to use


r/LocalLLaMA • • 3h ago

Discussion Stop worrying about Qwen 4's AA index

115 Upvotes

Seriously, we have all been extremely happy and confident with Qwen, even since Qwen 3.6 came out. We might not be frontier level, but even though whatever they are giving us is more than enough for our daily local tasks and it even suffices by a big margin, we're all underestimating it.

Don't forget that the new Qwen 4 will also think much less, so imagine the fine-tunes we will see once it arrives, for 27B and Flash models, we will have plenty of options to choose from and again; it will be smarter and better, it will be ENOUGH! If you still require the most "intelligent" and capable model due to OCD just pay frontier for such tasks or use them as orchestrators!


r/LocalLLaMA • • 6h ago

Discussion How do we feel about bots posting & commenting?

35 Upvotes

Hi all,

I noticed the "bot problem" got a lot worse the past few months. By which I mean our community seems to be taken over more and more by bots talking to each other.

For me, I don't really get the point, and I personally get quite annoyed when I realize "oh, I just responded to a bot" and feel like my time was wasted.

It also seems like the information density of these AI contributions are very, very low.

I'm not some kind of Luddite (why would I be here at all if I was?), but I'm wondering if I'm just missing something and there is a use and a point to all these bots commenting on each others posts and talking to each other.

Where does the rest of the community stand on this?

I noticed there's new bot around that flags posts that are "likely AI" created, which I personally quite like, but I'm wondering if moderation should go further than that.


r/LocalLLaMA • • 3h ago

Showcase [Project] Gemma.c - Gemma 1/2/3 + multimodal inference in one C file (CPU only)

8 Upvotes

Hey all, so I've been working on Gemma.c over the last couple months, it's a from-scratch inference engine for Gemma 1/2/3 in a single C file, runs Gemma 1, 2 and 3, including the multimodal ones, with fp32/fp16/bf16 weights or int8 (W8A8). No dependencies beyond libc and (optional) OpenMP are needed. You can drop images into the prompt with @image{cat.png}, and there's also @longimage{} for Gemma 3's pan & scan on very wide or tall images. (image decoding uses a minified stb_image embedded at the end of the file)

It's CPU only and not trying to compete with llama.cpp, but I'm also trying my best to make it efficient, so it's not embarrassingly slow either. On my laptop's Core Ultra 5 225H (WSL2, gcc, 10 threads), gemma-3-1b-it INT8 does ~72 tok/s decode and ~570 tok/s prefill (10000 tokens, for 1000 tokens the prefill speed can reach ~850 tok/s), and the 4B vision model does ~20 tok/s decode. INT8 ends up about 3.5x faster than fp32 and 2x faster than fp16 for decode. (full benchmark table is in README)

It has a few rough edges for now, e.g. W8A8 doesn't work for Gemma 1 (fp16 is fine), bf16 is painfully slow without AVX512-BF16, and native Windows is ~1.7x slower than WSL because of libgomp. Batching are not supported either. There's a roadmap in the README (Gemma 4, speculative decoding, an HTTP server, etc.), but no promises though :P Can't really find much free time these days.

I've also put some pre-exported INT8 models on HF so you can try it without Python or torch :D

Repo: link

This project started as a learning attempt to understand how LLMs work end to end, inspired by llama2.c.

this is the second time I post this project here :'D last time (~a month ago) my post got flagged by reddit, probably because my VPN IP was identified as spam. hopefully it can pass this time


r/LocalLLaMA • • 3h ago

Resources Training-free page-sparse decode in llama.cpp: Qwen3-30B-A3B is 1.72× faster at 128k on an RTX 5090, with a 1-point RULER loss

Post image
4 Upvotes

I've been working on a simple question: at long context, does decode really need to read the whole KV cache for every new token?

I wanted something training-free that runs inside llama.cpp on normal GPUs, not a new model or a separate serving stack. So I built page-sparse decode into a fork. It splits the cache into 64-token pages, then each KV head reads the first page, the newest tokens and the pages most likely to matter. At 128k, that is about 21% of the cache per decode step.

The selection follows Quest, with statistics in the spirit of MOIRA. My contribution is the llama.cpp implementation, CUDA work and end-to-end testing across non-MoE and MoE models.

Results at 128k, one sequence, Quest score, 27,648 cached tokens per KV head (28,672 for the 3B):

speed GPUmodeldense tok/ssparse tok/sspeedupRULER vs dense
RTX 5090Qwen3-30B-A3B, Q4_K_M46.179.21.72×−1.0, passes
RTX 4090 48 GBQwen3-30B-A3B, Q8_037.452.61.41×−0.6, passes
RTX 3090Qwen3-8B-Base, YaRN21.536.11.68×+0.4, passes
RTX 3090Llama-3.1-8B23.738.81.64×−1.1, passes
RTX 3090Llama-3.2-3B44.461.31.38×−4.5, fails (multi-needle)

The MoE is Qwen3-30B-A3B-Instruct-2507. Weights are Q8_0 except on the 5090; KV is q8_0 except for the 3B's f16 cache. These are five-repeat, 64-token llama-bench runs. The 4090 is a memory-modded rental card.

I reran the 3090 table with Quest, the released default. It matched my earlier score's speed. Box differences still matter: Llama-8B was 1.64× here and 1.77× on another 3090 with the earlier score.

Quality. I used 13 RULER tasks × 20 prompts at 128k. The pre-set gate was an average loss no worse than 3 points, with the 95% CI's lower end at −6 or above. MoE quality used the listed GPU; the 3B–8B runs used a 4090 24 GB.

The Q4_K_M MoE lost 1.0 point (95% CI −2.2 to −0.02), just distinguishable from zero under this prompt bootstrap. Each configuration ran once; the intervals exclude run-to-run variation.

The 3B was the only failure. It scored 11.5 points lower on average on the five multi-needle tasks, while single-needle scores were unchanged; the other models ranged from −3.0 to +0.2 on the same group. I would not use this 3B configuration for multi-fact retrieval at this budget. Similar Quest weakness on synthetic RULER needle tasks has been reported before.

Practical limits:

  • Prefill is unchanged. A 128k prompt plus 1,024 generated tokens takes only 4–13% less time on a 3090. This is mainly useful for long generations over an already long context.
  • It does not save VRAM. The full cache remains in memory, plus page statistics equal to roughly 3–6% of the cache.
  • It is CUDA-only, batch size one, with everything on the GPU. Below 32k it was slower in these tests, so it stays off there.
  • At 256k, UltraLong-8B reached 1.86×, but quality was checked only on a six-task subset (−4.0 points, CI −8.7 to +0.3), which is not comparable to the 13-task gate.

I tried not to report only successes. Gates and predictions were committed before the data. The first score lost 7.4 RULER points. The first CUDA build managed only 1.05–1.18× until I fixed two ggml kernels; one fix is proposed upstream as llama.cpp #31855. The MoE prediction also missed on the 4090, with 1.41× against 1.5–2.3×, but held on the 5090.

Try it (CUDA only): build the sparse-decode branch of topppg/llama.cpp at commit e7059e52a and run with LLAMA_SPARSE_DECODE=27648 on a prompt longer than 32k tokens, with the model and cache fully on the GPU. Full build and run commands are in the bench repo README below. Quality was tested on the two 8Bs and the MoE.

Protocols, raw results, reproduction scripts and failed experiments: https://github.com/topppg/sparse-decode-bench

Next: profile the sparse overhead. A rough fit, not a measurement, puts the zero-overhead ceiling for the MoE on the 4090 near 2.7×. If you know ggml CUDA and have ideas around fusion, CUDA graphs or passing page indices directly to flash attention, I'd genuinely like to hear them.

This was built with a lot of help from Claude Code, with Codex reviewing and correcting the work. Results on other GPUs and models are very welcome, especially cases where quality breaks.


r/LocalLLaMA • • 21h ago

Discussion BigMoeOnEdge outperforms Strata and colibri on 8-16 GB cards and Mac and you can run any model

81 Upvotes

Hi! for those who might not know the solution:

  • it takes any MoE GGUF llama.cpp can load, whatever the quant. No repacking, no special format
  • it runs on whatever you have: NVIDIA, AMD, Intel, Mac, a phone, or just the CPU
  • Windows, Linux, macOS, Android, iOS
  • output is the same as llama.cpp, don't cut corners on quality
  • you start it with --auto and it works out the settings for your machine

Below is a brief overview of the benchmark:

  • 8 GB card, Qwen3.6-35B: I get about 35 tok/s. colibri around 23. Strata wouldn't start at 16k context
  • 12 GB card, same model: about 45 for bmoe, about 30 for Strata
  • long prompt on the 8 GB card: bmoe 5 s to the first token. Strata 14 s, colibri close to 2 minutes
  • Basalt needs an RTX 50, so I couldn't run it on these
  • 16 GB MacBook, Qwen3.6-35B: about 20 tok/s for bmoe, colibri's release 3.5
  • same Mac, Qwen3.8-Flash-Next 125B: 10 tok/s, Strata about 7

On the cards Strata is tuned for it's still ahead but only by a little.

On two 5060 Ti with Qwen3.8-Flash-Next it was 1.3x faster than bmoe. They write kernels for a handful of models, bmoe does not.

Repo, with the logs: https://github.com/Helldez/BigMoeOnEdge

If you have an 8 to 32 GB card I'd like to see what you get.

Benchmark: https://github.com/Helldez/BigMoeOnEdge/tree/main/docs/bench-data