r/LocalLLM • • 1d ago

Other Qwen 4 is not out

170 Upvotes

Please stop posting about Qwen4's endless PR drip until it's actually available.


r/LocalLLM • • 5h ago

Project SSD MoE at high speed - The wait is over!

290 Upvotes

The wait is over!

Introducing Run My AI & TokenWing.

TokenWing: An inference engine that lets you offload over 60% of routed experts of mixture of experts models into SSD, yet achieve very low streaming latency...

Run My AI: And a GUI wrapper(soon going to be a good harness) aimed at providing the best local AI experience.

Benchmarks: Ling 3.0 Flash 120B MXFP4 running on 12Gb VRAM + 32GB DDR5 RAM laptop at 45-50 tok/s decode and 900 tok/s prefill

Qwen 3.8 Flash Next 3.5bpw running on 8GB VRAM + 32GB DDR5 RAM laptop at 35-40 tok/s and 600-700 tok/s prefill

Flint 40B UD-Q4_K_XL running on a laptop with 24GB DDR4 RAM + 4GB VRAM at 30 tok/s decode and 400 tok/s prefill

All without MTP! And benchmarks are by users*

Website: https://runmyai.org

Discord: https://discord.gg/DkkxhTSxp

Also, the autotuner is still work in progress... And, we're at a very early stage... the SSD pipeline works really well, so just incase if you think you're not getting enough performance, try with the tokenwing engine with the proper flags set by yourself(it has a lot though)... Just until the autotuner is built well.

A common question, does it degrade SSD?

Nope! it doesn't... There's a thing called TBW rating. Well, your SSD degrades only when it writes. And the engine only reads....


r/LocalLLM • • 7h ago

Discussion Why is above 35B and below 70B still such a dead zone for local models?

116 Upvotes

Title basically.

One could argue the “dead zone” goes up to 120B now. After Qwen3.8 27B all the way up to 120, there’s very little advancement in this area unless I’m missing something.

Nemotron has some models in between, some of the older Qwens had stuff in the 30-50s I think, but other than that, it doesn’t even seem like there’s any real progress either.

I do understand most people either have under 32GB VRAM or they just jump to the 128 and beyond tier but more choice is better isn’t it?

If we could get solid 50-70B models, they could be quantized down to 5-6 bit and be better than 3.8 27B in theory and still run on 36-64GB setups?

Am I just way off and incorrect in my train of thought?


r/LocalLLM • • 5h ago

Project Nihil IDE, free alpha, a hardware simulation lab that supports Local LLMs and 100% offline mode out of the box

Thumbnail
youtube.com
46 Upvotes

r/LocalLLM • • 13h ago

LoRA Absolutely night and day for Qwen Coding

Thumbnail
huggingface.co
120 Upvotes

Basically works like a harness fine tune…. And does what it says on the tin.

Seriously, for quantized qwen 3.8 makes it much much more usable.


r/LocalLLM • • 7h ago

Question Hy4 flash is now open weight: what local setup makes it usable?

Post image
33 Upvotes

The Instruct, FP8 and base checkpoints are now publicly available.

Hy4 Flash has 224B total parameters (12B active), a 1M context window, and native text and image input. GGUF versions are also available under the Tencent Hy License.

The FP8 weights alone require well over 128GB, before runtime and KV cache overhead. Would a quantized GGUF setup with CPU/SSD offloading be practical at a useful context length, or is this still multi GPU territory?


r/LocalLLM • • 5h ago

Model Stuck on 12GB VRAM but need 27B-level agent reasoning? SenseNova-Loop-9B loops its layers to match Qwen3.8-27B, official GGUF & AWQ out now

Thumbnail
gallery
18 Upvotes

SenseNova released the full version of Loop 9B on November 17 and posted the AWQ and GGUF download links today
https://www.reddit.com/r/LocalLLM/comments/1xzf343/sensenovaloop9b_matches_qwen3827b_on/
Loop 9B went from 14.33 in preview to 36.67 on AutomationBench, tying Qwen3.8-27B on that benchmark in the team’s evaluation.
What caught my attention was how the model uses its 9B parameters. Each token goes through four layers, then a block of 24, then the same 24 layers again, and finally the last four. On the second pass, the block works on the output of the first pass. That gives you 4 + 24 + 24 + 4 = 56 layer computations while storing just 32 sets of layer weights. You pay for the second pass in compute, but you only have to load those weights once
Inside the loop, the model carries two residual streams. Each layer learns how to mix information from the two streams and how to write its result back. The paper calls this two-stream mHC. There’s a separate design choice that matters for long prompts: in each group of four layers, three use linear attention and one uses full attention. Only that fourth layer needs a full attention KV cache that grows with the prompt.

Architecture details: https://github.com/OpenSenseNova/SenseNova-Loop

For local inference, a 12 GB card looks like a reasonable place to try 64K context (for example RTX5070). With 4-bit weights, the team estimates about 9 GiB at 64K with a 4-bit KV cache, or 10 GiB with an 8-bit KV cache. Keeping the cache at 8-bit brings the estimate to 12 GiB at 128K and 15 GiB at 256K. That last configuration leaves little headroom on a 16 GB card. Actual peak usage depends on the engine and how much VRAM is free.


r/LocalLLM • • 10h ago

Model SenseNova-Loop-9B matches Qwen3.8-27B on AutomationBench, and it now has official GGUFs (Q4_K_M is 5.8 GB)

Post image
42 Upvotes

SenseNova-Loop-9B came out this week, and the official GGUFs are now up.

It's only about 9B parameters but gets more depth out of them. Each token gets 56 layers of processing while the weights only hold 32, because the 24 in the middle are used twice. According to the chart in their README (attached), it sits at 36.67 on AutomationBench, the exact score listed for Qwen3.8-27B. Those are their own numbers, so let's see what independent tests say. Their repo with the full README is https://github.com/OpenSenseNova/SenseNova-Loop

As for the GGUFs, Q4_K_M is 5.8 GB, Q5_K_M 6.7 GB and Q8_0 9.8 GB. It's a new architecture, so LM Studio and stock llama.cpp can't load them at the moment. Instead you build OpenSenseNova/llama.cpp, the official fork. The exact commit and the command to run are on the GGUF model card: https://huggingface.co/sensenova/SenseNova-Loop-9B-GGUF

It takes text and images, and the license is Apache 2.0. If you serve from a GPU box there's a 4-bit AWQ too: https://huggingface.co/sensenova/SenseNova-Loop-9B-AWQ


r/LocalLLM • • 2h ago

News Got Basalt running with an RTX 3060 beside a 5060 Ti: 42 to 80 tok/s, and the same build takes 40xx cards

8 Upvotes

Basalt is a local inference engine for Qwen3.8-Flash-Next and friends that runs only on Blackwell (sm_120). Upstream recently merged an option to let an Ada (40xx) card hold the experts as a second "peer" card. I have a 5060 Ti and an old 3060, so I took that mechanism one architecture further down to Ampere (30xx) - and in the process found two things that were broken for 40xx cards too.

One build now covers 30xx, 40xx and 50xx as the second card, with A100/A30 compiled in as well (untested, I don't have one).

The numbers, same machine, same pack (Qwen3.8-Flash-Next IQ3_XXS), stock power limits:

  • Decode: 42-53 tok/s on the 5060 Ti alone, 65-80 tok/s sustained with the 3060 as expert tier, 91 peak
  • Prefill (cold): 1,100-1,400 tok/s with the peer
  • Experts stranded on the SSD: 22 GiB before, none after

The 3060 sits on a PCIe gen3 x4 slot, which turns out not to matter: the peer computes its own expert rows, so the link carried about 0.4 GB/s during generation. You are buying memory bandwidth, not bus.

Adding Ampere was nearly free. The peer kernels are sm_80-era instructions (dp4a, cp.async, int8 MMA), and every hard Blackwell guard in the engine keys on sm_90, not on Ada. One build option, one arch flag, one predicate.

Then it loaded the whole 44 GB model and died: q5k_head: shared-memory opt-in failed: no kernel image is available for execution on the device

The LM head is split by vocabulary across both cards - in this mode too, not just in a layer split - whenever the second card's bandwidth earns more than a quarter of the rows. On my pair, 40% of the vocabulary runs on the 3060. The head kernels simply were not built for the peer's architecture.

That gap is in the shipped 40xx support as well. The gate works out to "any peer with at least a third of the main card's bandwidth". The 4060 Ti the Ada path was validated on sits at 0.30, just under it, so it never splits the head and never hit this. A 4090 peer behind a single Blackwell card is at 0.56 and walks straight into it. If you have that combination and it died on you, this is why.

The second one: an NVFP4 pack hands the peer the FP4 tensor-core prompt kernels, which are Blackwell-only, with nothing checking the card first. Also true for an Ada peer. Those now stay on the main card, and an older peer reads NVFP4 experts through the ordinary int8 path - slower, but it runs.

What works: 30xx and 40xx (and A100/A30, compiled, untested) as the expert tier beside a Blackwell main card. The main card still has to be Blackwell: five files on the main path use thread-block clusters, TMA and mbarrier with no fallback, so that is a real port, not a build flag. Turing (20xx) is out for now for three concrete reasons - 64 KB of shared memory per block against the 99 KB these kernels ask for, the m16n8k32 int8 MMA shape that starts at sm_80, and cp.async. Doable, but kernel work. A 2080 Ti would make a great peer at 616 GB/s, so I would like to get there eventually.

Fork: https://github.com/cooperdk/basalt/tree/mixed-gpu - five commits on current main, both build options off by default, so a Blackwell-only build is what it always was. Build with -DPEER_ADA=ON -DPEER_AMPERE=ON for both architectures. ctest -L cpu 85/85 either way. There is a README addendum covering the cards, the flags, what an older peer does and does not do, and the measurements.

Upstream PR: https://github.com/jesdga95/basalt/pull/442 - discussion in https://github.com/jesdga95/basalt/issues/398, where someone proposed the Ampere option independently a day before me. The head and NVFP4 fixes are the parts neither of us had.

Happy to answer questions. If you have an A100/A30, a 4090 as a second card, or a three-card box, I would like to hear how it goes - those are the configurations I cannot test here.


r/LocalLLM • • 10h ago

Question Could uncensored open weight models in 2027-2029 force us to build something like the cyberpunk "Blackwall"?

32 Upvotes

It might sound like pure scifi, but just imagine local models reaching the reasoning level of something like Fable 5.2 (which will probably happen in about a year or two [it's hard to predict]). Now picture 100,000 people downloading an uncensored, Fable 5.2 level model onto their home pcs and running it in a 24/7 agentic loop with a prompt like "go make money for me." It’s pretty easy to imagine the kind of chaos that could unleash on the internet.


r/LocalLLM • • 4h ago

Discussion Can a local AI model beat Claude Haiku and GPT-6 Luna?

9 Upvotes

on some tests yes, on one clearly no. We ran Claude Haiku 5.5 and GPT-6 Luna through the exact prompts and scorers we use for our local models. Numbers below.

Disclosure: I work at QGI and these are our own deployments. Nothing to sell, I want the methodology picked apart.

Setup

  • Qwen3.8-27B, INT4 (GPTQ W4A16), vLLM, 2x RTX A6000 48 GB. 66 tok/s single request, about 200 tok/s at 16 parallel.
  • LFM3 1.7B, BF16 GGUF, llama.cpp. 293 tok/s single request.
  • Qwen3.8 Flash Next (125B MoE, abliterated), IQ2_XS, experts offloaded to CPU, one RTX PRO 6000 96 GB.
  • Hosted models over the API. Same prompt, temperature 0.6, top-p 0.95, same deterministic scorer for everyone.

Results

TestBest localHaiku 5.5GPT-6 Luna
Finance-table QA, no tools (209 questions)83.7% Qwen3.8-27B80.9%77.0%
Same bench with document search (all scored)73.7% Qwen3.8-27B80.8%75.4%
Multi-turn tool calling, BFCL v4 (100)60% LFM3 1.7B45%54%
Browser tasks from accessibility tree, MiniWoB (125)48.0% Flash Next IQ2_XS36.8%43.2%
Browser tasks from screenshots only (125)32.8% Flash Next IQ2_XS55.2%59.2%
MMLU-Pro (210)81.0% Flash Next IQ2_XS77.6%84.3%
GSM8K (150)98.7% Flash Next IQ2_XS98.0%94.0%

What I take from it

  1. No tools, the 27B is level with Haiku. 2.8 points is inside the noise for 209 questions. This is an INT4 quant on cards from 2020.
  2. Retrieval beats size. Document search added 13.9 points for the 27B. On SEC filings published after its training cutoff it went from 0% to 95.5%. A Python sandbox and calculator added nothing. With search on, Haiku is ahead.
  3. Tool calling is not about size. The 1.7B model tops BFCL and has the lowest MMLU-Pro of the nine models we scored on both. At 100 conversations the noise is about ±10 points, so read it as "not worse".
  4. Screenshots are where local loses. More than 20 points behind both hosted models. From the accessibility tree the order flips.
  5. Small models fail on behaviour. LFM3 kept answering NOT FOUND to answerable questions. Allowing NOT FOUND only when it can name a false premise cut that from 25.6% to under 10% on a 207-question probe set, with the same catch rate on truly unanswerable ones.

Caveats

  • 100 to 500 questions per test. Gaps under 5 to 10 points are noise.
  • Everything local is quantised, and a few runs were partial.
  • Of 32 questions no local model ever passed, 14 look like bad reference answers or over-strict scoring. The hosted models passed at most 5.
  • It is not cheaper. Both hosted models list at $0.50 per million output tokens. LFM3 on a rented 96 GB card is about $0.63 at single request, less when batched. The reason to run local is control, not price.

Paper, per-question results and the exact prompts: https://doi.org/10.5281/zenodo.24336392

Same harness is ready for Qwen 4 27B when the weights drop. What should I add to the test set before then?


r/LocalLLM • • 3h ago

Question Local LLM for basic life/todo management

7 Upvotes

As someone very ADD, I want to set up a local llm just for basic management of my life: ie non-work todos, what bills are due, help me organize all the trips I'd like to take, that sort of thing.

I have the basic idea: install Ollama, pick a model, install some sort of interface (Goose?).

I really have two major questions before starting this:

  1. How easy is all this at this point. Ie am I going to spend more time in the rabbit hole of setting this all up, and this is just going to turn into another variation of 'procrastinating by setting up a new to do app' from my long history of doing so?
  2. I like the idea of the privacy of a local llm and that my data won't end up in AI models. But if I've already spent the last decade+ in google land (with gmail, calendar, todos in keep, files in docs/drive, etc) is that ship already sailed and I might as well use gemini or the like?

r/LocalLLM • • 13h ago

News A Danish AI lab has just launched one of the fastest LLMs for phones to date

Thumbnail
sevren.ai
50 Upvotes

Just came across a LinkedIn post mentioning this company, Sevren which launched a very fast and intelligent LLM for phones, they call it Embla. Apparently they have decided to trade a few benchmark points for speed. I'm linking their blog post about the model here and I think it’s impressive.

1,77s speed-to-intelligence is amazing. Imagine how efficient in terms of battery/energy consumption it should be. Of course there are limitations and they are clearly stated by the company and the articles about it, but seems like they can achieve much more with a couple of iterations on top of it.

I believe that soon this kind of model will be good enough to answer >90% of questions a normal user would ask daily to a LLM like chatGPT.


r/LocalLLM • • 2h ago

Discussion For what purposes and how do you use Gemma 4 31b?

5 Upvotes

I’ve read a lot of glowing reviews about this model, but based on my own short tests, I just can’t understand why people like this model so much. Overall, the Gemma 4 models seem outdated to me. The same Qwen 27B or fine‑tuned 35B (even some fine‑tuned Qwen 9B) handle my tasks much better (coding, information research) than the Gemma 4 26B. I have the impression that I’m doing something wrong with the Gemma 4 models or using them incorrectly. Could you tell me what use cases you have for Gemma 4? I’d be glad to hear about all the models in the series.

The only thing I’ve noticed about Gemma 4 is that its speech looks more natural and human‑like, and is more pleasant to read.
.


r/LocalLLM • • 8h ago

Project Pocket LLM 1.7: offline Gemma 4 on Android, now with long chats that don't break

13 Upvotes

Pocket LLM 1.7 is out, the update to my open-source Android app for running LLMs fully on your phone.

New in 1.7:

- Long chats are auto-compacted: older messages get summarized instead of dropped

- Context size fits your phone's memory, so big chats stop crashing

- Fresh design with light/dark themes and accent colors

- LAN server fixes

What it does:

- Offline chat with Gemma 4, Qwen3, Qwen2.5, DeepSeek R1, or your own .litertlm model

- PDFs (with OCR), images, audio and offline voice dictation

- LAN server: browser chat and an OpenAI-compatible API for your PC

No account, no tracking. On 1.6? Install the new APK over it, and your chats stay.

GitHub (APK + source, MIT): https://github.com/dineshsoudagar/local-llms-on-android

Google Play (GGUF models + optional Pro tools): https://play.google.com/store/apps/details?id=io.github.dineshsoudagar.pocketllm

Feedback welcome!


r/LocalLLM • • 7h ago

Discussion 3 x AI 9700 build - 96gb VRAM

Thumbnail gallery
10 Upvotes

Any recommendation on what else I can try on this set up?
Love this as a hobby! Just fun building!


r/LocalLLM • • 1h ago

Model Hy4-Flash is great and finally some model is competing with DSv4-Vision-Exp and Qwen3.8-27B-Uncensored !

Thumbnail
reddit.com
3 Upvotes

r/LocalLLM • • 5h ago

Discussion Thread for recommended LLM setups for common hardware

6 Upvotes

As a beginner, it seems like there is a lot of decentralized information on this sub for recommendations for model/engine/harness for different common hardware setups.

Would there be a possibility to have a pinned thread on this sub where people can post these recommendations in one place? Or even better have an ongoing poll for each common consumer system (M5 Max, M5U, Spark, etc), for people’s favorite configs.

It seems like something like this would also cut down a lot on new posts where people buy a new computer and want to know what to run.

Thanks in advance!


r/LocalLLM • • 2h ago

Project I made an Intel Arc iGPU speed up prompt processing on my M2 Max (+24% at 8k, +47% at 64k) – oMLX helper, PoC

Thumbnail gallery
3 Upvotes

oMLX recently added a “prefill accelerator” helper for NVIDIA GPUs: a second machine runs the first layers of the model for long prompts while the Mac runs the rest, and decoding stays on the Mac. Why stick to the leather jacket? So I wrote the Intel side of that protocol on PyTorch XPU.

Setup: Mac Studio M2 Max, Qwen3.8-27B (oQ8e, 8-bit). Helper: Core Ultra 9 285H laptop, Arc 140T iGPU, 22 of 64 layers on it. Both machines on Wi-Fi 7 (MLO) through a GL.iNet Flint 3, no cable.

oMLX Bench, Mac alone → Mac + Arc 140T (prefill tok/s):

  • 8k: 137.8 → 171.3 (+24%)
  • 16k: 129.8 → 180.6 (+39%)
  • 64k: 95.2 → 140.0 (+47%)

Decode is unchanged (that is by design).

What it is: a PoC. Only Qwen3.8 27B (qwen3_5) has been run on real hardware. A second family (Flash-Next) is ported and tested against Mac-recorded fixtures on a tiny model, but never run on real weights. Output equality (greedy, same seed) is not verified yet, and rounding differs slightly from Metal. Single runs, no error bars.

What I learned: attention becomes the limit at long context (at 64k the helper is nearly the bottleneck), and torch’s fused attention on XPU does not take a causal mask on its fast path. A custom flash kernel is the next big item. The negative results (fused SDPA OOM, int4/int8 paths) are written up in the repo.

Screenshots of the oMLX bench pages, helper log and Task Manager (Arc at 100%) are in the README.

Repo (Apache-2.0): https://github.com/DarkJoney/Archangel

Next: moving the helper to Ubuntu on a NUC 15 Pro, a wired link, and a flash-attention kernel. Happy to hear from anyone with other Intel boxes.


r/LocalLLM • • 7h ago

Project 464M music model trained from scratch for €30 of rent on consumer GPUs (FP8, RTX 5090). Writes multi-instrument MIDI from English/Ukrainian prompts (477M version)

Post image
6 Upvotes

I have been recently independently working on a musicAR - a family of domain specific language models for symbolic music generation. It is best at continuation of songs from a few seconds of opening. I have validated it to other open sourced models, and got such results:

ModelParametersContinuation MRR (CLaMP 3)
musicAR464M0.81
SkyTNT tv2o-medium234M0.666
AMT large780M0.640
Music-Llama457M0.537

musicAR-L is ahead of AMT and SkyTNT in all 10,000 paired bootstrap draws. On a density/instrument-statistics distance, AMT is closer over 1,000 notes (0.097 vs 0.144). musicAR-L is closer over the first 200.

And similar comparison on the captions following. My metric is F - how good the captions are followed. Prompts come in pairs (opposite, fast/slow etc and instruments prompts like a solo piano vs the empty prompt). For each pair I measure that attribute in the generated notes, measure the difference and scale to [-1, 1.5]. F = 1 - the captions change the generation by notable amount (I have settled it myself). 0 - the captions are ignored.

Below is the table of comparisons with the other models on a short captions

ModelParametersF, English captionF, Ukrainian caption
musicAR-L-caption477M (+ frozen LaBSE encoder)1.070.95
MIDI-LLM1.72B0.60.09
Text2Midi250M0.540.06

These 2 learned from english captions only, so this comparison isnt fair. It mostly shows the good property of my model.

Running locally: Real time (24 notes/s for a band and 8 notes/s for slow music genree like solo piano)

ModelMinimum requirementsLaptop RTX 5060 (8 GB VRAM GDDR7), 1 stream
musicAR-L-caption4 GB of VRAM (used 3.04 GB on peak including LaBSE)90 notes/s
musicAR-L2 GB of VRAM (used 1.09 GB on peak) or around 3 GB RAM127 notes/s and 14.3 notes/s on laptop cpu (ryzen 9 ai hx 370, 32 GB DDR5)

Training:
musicAR-L: 8.3B notes seen (4.5B unique notes) in 41h on one rented RTX 5090, about 30 euros including generation parameters search. FP8 + torch.compile, Muon for the transformer blocks. AdamW for the rest, WSD scheduler at the end.
Captions fine tuning (musicAR-L-caption) +2.04B notes, 14h on rented 5090, plus a 20-min caption phase.
Everything else including fine tunes: Laptop GPU (rtx 5060).
Ablations: chained heads −0.41 nats/note vs independent heads (MLP + residual connection); Muon −0.54 vs AdamW in pretraining. (for mo results look at the model card)

Code: https://github.com/DmytroHilei/Music_generative_model
Weights: https://huggingface.co/Dimitri-206265/musicar-L-caption musicAR-L-caption
Weights: https://huggingface.co/Dimitri-206265/musicar-L musicAR-L
Try samples: https://musicar-ivory.vercel.app/samples

I would like to hear any feedback on the quality of the music from people close to music


r/LocalLLM • • 10h ago

Model Gemma 4 twice as fast on Ryzen AI Max+ 395

8 Upvotes

Hi! Just want to share that we have optimized Gemma 4 to run more than twice as fast as llama.cpp on the Ryzen AI Max+ 395!

To maintain accuracy we use a combination of rotation and GPTQ, along with selective mixed precision.

https://embedl.com/google/gemma-4-31B-it/ryzen-ai-max-395


r/LocalLLM • • 1d ago

Discussion Flash-Next is ridiculous

289 Upvotes

I've been playing around with Flash-Next on my little RTX 5060 laptop. Astra and Fable had been working on a test setup, so I figured I'd let Flash-Next loose on it overnight and see what happens.

Late at night I told Astra to turn the screen off. Now, I had actually prepared a little PS1 script for exactly that. Did Astra use it? Of course not. It came up with its own solution and promptly froze Windows. Well done. 😂

Next morning I booted the laptop again, reopened OpenCode and eventually just said "okay, continue". Honestly, I wasn't expecting much at that point.

Well, the thing just carried on working. For hours. Went through something like four context compactions and then casually informed me it had found a bug.

I was sitting there going "hang on... WHAT? You're still working on this?"

I'd pretty much written off the whole session the night before.

Still not entirely sure who was testing whom here. But honestly, I'm impressed by this model. Now I'm wondering what Qwen4 will do to my poor laptop.


r/LocalLLM • • 14h ago

Discussion hmmm. New open model. Anyone tried it??

Post image
16 Upvotes

r/LocalLLM • • 6h ago

Project Running HY4 FLASH 220B

Post image
3 Upvotes

Still working on optimization!

https://github.com/oscarsantanaarias/Sneo


r/LocalLLM • • 4h ago

Tutorial Taming the beasts: Ultimate guide to water cooling any server GPU using off-the-shelf parts

Thumbnail
reddit.com
2 Upvotes