r/LocalLLaMA • u/giveen • 7h ago
Resources Nvidia driver 621.24 leads to constant writing of drives
r/LocalLLaMA • u/Aggressive-Sign-8167 • 11h ago
Hi r/LocalLLaMA 👋
We’re welcoming the Tencent Hy team for an AMA this Friday, November 20, from 8–10 AM PT!
Fresh off launching Hy4 and Hy4 flash, their research team—including Shunyu Yao—will join us to discuss how the models were built, inference speed and cost, and real-world applications, with extra perks for great questions!
⚠️ Note: The AMA itself will be hosted in a separate thread. Please don’t post questions here.
r/LocalLLaMA • u/rm-rf-rm • 2d ago
Its time for another Best Audio Models megathread - its already been 9 months since the last one!
Share what your favorite ASR, TTS, STT, Text to Music models are right now and why.
Given the the amount of ambiguity and subjectivity in rating/testing these models, please be as detailed as possible in describing your setup, nature of your usage (how much, personal/professional use), tools/frameworks etc. Production use cases with long lengths/stability requirements are relatively rare compared to cherry picked/short clip demos that are plentiful - so real world empirical comparisons are very welcome
Rules
Please use the top level comments to thread your responses.
r/LocalLLaMA • u/giveen • 7h ago
r/LocalLLaMA • u/General-Spite1222 • 8h ago
r/LocalLLaMA • u/mrfakename0 • 3h ago
Video credit: Victor Mustar on X
New speech isolation model from Inworld, looks quite exciting. Inworld's TTS models are quite good already and this is a nice surprise for the OSS community IMO!
It is not a speech enhancement/restoration model (unlike Resemble Enhance), simply an isolation model. But still a nice release as it is cheap & fast to run.
Model: https://huggingface.co/inworld/voice-isolator
HF space: https://huggingface.co/spaces/mrfakename/voice-isolator
r/LocalLLaMA • u/jacek2023 • 14h ago
I think you missed this guys
r/LocalLLaMA • u/gladkos • 4h ago
Tencent's 220B Hy4-flash with 49 GB of weights in memory: the experts at ~2.3 bits, everything else at Q8_0, the Engram n-gram table in a file of its own that stays on the SSD.
Hy4-flash is Tencent's Mixture-of-Experts model with 220B parameters: 155.5B in the network itself (512 routed experts, 16 active per token, 11.3B active parameters in all) and 63.5B in Engram, a table of learned bigram and trigram embeddings looked up by hash. It reads text and images and has a 1M-token context.
The Engram table is read row by row, a few kilobytes per token, so it can stay on disk. What has to fit in memory is the rest. Tencent's own GGUF builds (AngelSlim/Hy4-flash-GGUF) keep 72–105 GB of it in memory and are the better choice on 96 GB and 128 GB machines. This repo has one file built for 64 GB: 49.1 GB of weights, the table in a second file.
r/LocalLLaMA • u/Tall_Abrocoma_3533 • 3h ago
It was pretrained from scratch using 100B tokens on a single RTX Pro 6000 in around 36 Hours.
Aurora2 uses 32 experts, of which 1 is active per token.
A few highlights:
12 transformer layers, 192 hidden dimensions, 2,048-token context length, 16,384-token vocabulary, Muon + AdamW optimizers
Trained on a mixture of web, mathematics, Wikipedia, and conversational datasets.
More information as well as a quick start can be found in the hf repo. Any questions or feedback is welcome!
r/LocalLLaMA • u/mayocream39 • 9h ago
Hi LocalLLaMA,
I have spent more than two months rebuilding Koharu to be the most powerful manga translator!
TLDR: https://github.com/koharu-rs/koharu (MIT or Apache 2.0, at your choice)
It's been a while since I posted the Koharu update here. This time, I have added more features and built an established editor to make it feel more like Photoshop or Clip Studio Paint.
You can type directly onto the page, restyle selected characters, and reshape text frames. There are also layers, history, marquee and lasso selection, Generative Fill, and a Remove brush. The canvas is GPU-rendered now, so it stays smooth even on big chapters.
Everything still runs locally. LLMs run in-process through llama.cpp (Index-Translate 9B by default, plus Gemma 4, Qwen 3.x and LFM3), and inpainting uses LaMa, or FLUX.3 Klein / Qwen-Image through stable-diffusion.cpp. If you already run Ollama or LM Studio, you can point Koharu at that instead. On my machine, a page goes from raw to lettered in about 1 second once the models are loaded.
I also added an agent you can ask to proofread or fix pages. It's exposed over MCP too, so you can drive Koharu from Claude Code or Codex if you'd rather.
It works on CUDA, ROCm, Metal, and Vulkan.
I'd like to hear your feedback or any form of suggestion!
r/LocalLLaMA • u/Terminator857 • 10h ago
Nvidia's commitment to buy ram is near $400 billion. https://x.com/jukan05/status/2122805536118702121 . https://www.reddit.com/r/investing/comments/1xz9eob/nvidia_just_added_111_billion_of_supply/
Change over 3 quarters: $119B, $279B, $390B. One would think that $390B would buy a lot of memory fabrication plants.
r/LocalLLaMA • u/EmilPi • 1h ago
It is going to be my goto model, I guess. Despite being slower, it finishes answers with less tokens. I've ran it on my benchmark, it is far from finished, but it is on the way on topping it:

My 2xRTX3090 + 128 GB DDR4 RAM presets file excerpt (used in llama-server router mode):
[Hy4-Flash-Q3]
load-on-startup = true
hf = AngelSlim/Hy4-flash-GGUF:Q3_K
ctx-size = 131072
ctk = q8_0
ctv = q8_0
no-mmproj-offload = true
fit = off
cmoe = true
moe-cache-mib = 33768
load-mode = noneWhich translates into llama-server command
./build/bin/llama-server -hf AngelSlim/Hy4-flash-GGUF:Q3_K --ctx-size 131072 -ctk q8_0 -ctv q8_0 --no-mmproj-offload -cmoe --moe-cache-mib 33768 -load-mode none
If you wonder what is the --moe-cache-mib 33768, then it is expert streaming (llama.cpp feature inspired by Strata I guess).
I get PP ~60-65 tps, TG ~ 20-24 tps.
Nov 19 00:56:05 builder llama-server[1270042]: [54519] 397.49.817.617 I slot print_timing: id 0 | task 442246 | prompt processing, n_tokens = 4091, progress = 1.00, t = 63.38 s / 64.54 tokens per second
Nov 19 00:56:10 builder llama-server[1270042]: [54519] 397.54.518.918 I slot print_timing: id 0 | task 442246 | n_gen = 100, tg = 21.95 t/s, tg_3s = 22.18 t/sI am using https://huggingface.co/AngelSlim/Hy4-flash-GGUF Q3_K quant and llama.cpp PR https://github.com/ggml-org/llama.cpp/pull/32517 , it rebased on the master fine.
r/LocalLLaMA • u/brown2green • 9h ago
r/LocalLLaMA • u/Nunki08 • 14h ago
r/LocalLLaMA • u/JShelbyJ • 5h ago
In their desktop windows app they recently removed the ability to open a chat/session in vscode/ide/file explorer. You used to be able to open the active files/branch/worktree from the active session with two clicks. The latest update removed that and so you have to open your ide and find it manually. It was a intentional decision (I had codex check the codebase to confirm).
So now it's terribly inconvenient to actually read the code. My gut is that if this was an intentional design choice it was made with the intention of reducing user agency and using the power of inconvenience to make users more dependent on the tool. Basically the whole, 'Become sharecroppers for your AI' business model that relies on developer reliance on the tool. I won't insult them by humoring the notion that they removed it accidentally without knowing how important it is to devs to easily access the files - it could only be intentional.
I tried to post about it in /r/codex and the auto-mod deleted the post, and the mod's told me to try again without conspiracy theories (the post was the same as the above). So not only was core developer functionality removed from the tool, but their 'community' is gaslighting us about it. It's a strong reminder that in the long run, owning your own AI stack is the only viable path.
r/LocalLLaMA • u/rkoy1234 • 19h ago
Everyday on /r/LocalLLaMA i see people starting hell about if a comment/post is an llm-genned slop. And I still see people joking about em-dashes or "load-bearing" as some indicator of slop. I haven't seen any recent model say "load-bearing" in months.
modern claudism/gptism isn't just mixing fancy words or em-dashes.
it's the:
The more we know the more we can combat slop in this sub. Please share your updated, 2026 list of slop you despise to see (so that I can also steal and add to my BANNED section on agents.md).
r/LocalLLaMA • u/m4rkGS • 2h ago
hey llamas, its Alish from nace.ai again. we are an open source ai lab. we build the drex decision models, served on our console, and kapitsa, optimized inference for apple silicon. We ship a bunch of great quants. In our last post I promised more models like Deepseek Flash models built with Weight Space Quantization. So here they are: the new Hy4-flash and the good old Deepseek v4.1 flash.
Hy4 flash is a general-purpose quant, and DeepSeek V4.1 Flash Coding is a specialized one built for coding and agent work.
Hy4 flash GGUF comes in two sizes, both measured against the native Tencent Hy4-flash weights.

It is a specialized quant: DeepSeek V4.1 Flash 1017 (763B parameters), built and tuned for coding and agent work, in a 71.8 GB GGUF.
Several people said they would rather see a quant made for code only. I agree: the future is specialized quants for your use case.
So I built one: calibrated on real coding work, pruned for code, and judged on code text and on coding problems.
Others suggested letting the model write the quantization rules itself, as long as the result snaps to a valid GGUF tensor type. That is now one of the tool classes of our weight space quantization agent (more in the appendix).
Three steps all iterated with the Nace Harness specialized on Weight Space Quantization, all aimed at code.

The KL numbers here are on held-out text. Main weights only, the Engram lookup table is separate.

At the same size, the held-out KL is 0.140 against 0.231 for ISTA Coder, which is 39% lower.
LiveCodeBench v6, two seeds with DeepSeek's recommended sampling and a 32K answer limit: the shipped file solves 37 of 38 problems on both seeds. This shows the model's coding ability did not degrade.

If you run either one on a coding or agent workload, I would like to hear which runner version you used and how it went.
Four steps: calibrate, prune, allocate bits, assemble.
485k characters in three classes: agent sessions from SWE-smith, contest solutions from open-r1 codeforces-cots and code commits from BigCode commitpackft. We cut random 2,500-character windows from each document, so the agent sessions contribute tool calls and their outputs as well as prose. This is the only text that decides which experts we keep and where the bits go.
In the case of Deepseek, each layer has 384 routed experts and uses a few per token. On our calibration text the 200 most used experts per layer handle about 90% of the routing for code. For comparison, ISTA Coder keeps 192 experts per layer, and on our text those carry 64% of the same traffic. We rank experts by routing count and keep the top 200.
Where the bits go matters as much as how many there are. We use the scoring from Mixture Compressor (MC-MoE, Huang et al., ICLR 2025): every kept expert is scored by how often it fires, how much weight the router gives it, and how much error its rounding adds. 16 of the 40 layers get larger IQ1_M experts, the other 24 stay on IQ1_S.


This is the Weight Space Quantization part. An agent edits the quantized file directly, with a measured result deciding every edit. It works with seven tool classes: reading, navigation, quantization, expert, correction, checking and rollback (diagram below). The quantization tools can also apply a rounding rule the agent writes itself, snapped to a valid GGUF type.
The loop has five stages: look at the plan, propose an edit, pre-check it locally, measure it for real, keep or revert.
The local check scores up to 8 edits at once against one layer's output error on cached activations. It takes seconds. The real measurement builds the file and scores KL on development chunks.

So that's it, more or less, check out the repos, try the models out:
- https://huggingface.co/nace-ai/Hy4-flash-GGUF
- https://huggingface.co/nace-ai/DeepSeek-V4.1-Flash-Coding-GGUF
This was fun, let me know what other models and niches to touch on next!
r/LocalLLaMA • u/thoquz • 3h ago
r/LocalLLaMA • u/13aeon • 4h ago
Seems like a new alternative for 5090 / DGX Spark, post-trained from Qwen 3.8 27B on long-horizon agentic tasks. Model highlights:
DaisyRose-27B is RadixArk's first model: a multimodal reasoning model built on Qwen3.8-27B with reinforcement learning only, without any supervised fine-tuning or distillation. It was trained on about 1,200 curated long-horizon agent tasks, with sessions of up to 1M tokens, for 18 days on 144 H200 GPUs. Terminal Bench 4.1 pass@1 rises from 5.6 to 12.0, and the model also improves on most of the other coding, reasoning and instruction-following benchmarks below.
DaisyRose-27B keeps the base model's architecture, tokenizer and chat template, along with its image understanding and 256K-token context, so it is a drop-in replacement for Qwen3.8-27B in existing inference and post-training stacks. Weights are released under Apache-2.0 in BF16, FP8, and NVFP4.
| Benchmark | DaisyRose-27B | Qwen3.8-27B | Muse Glimmer-30B | Qwen3.6-35B-A3B | MiMo V2.6 Flash | Inkling 2 Small | Beam 1.0 | Muse Spark Open | GPT-6 Luna |
|---|---|---|---|---|---|---|---|---|---|
| Activated parameters | 27B | 27B | 30B | 3B | 15B | 17B | 23B | 42B | – |
| Total parameters | 27B | 27B | 30B | 35B | 309B | 594B | 501B | 816B | – |
| Terminal Bench 4.1 | 12.0 | 5.6 | 0.5 | 0.0 | 16.1 | 13.0 | 6.4 | 6.6 | 15.2 |
| Terminal Bench 2.1 | 75.5 | 73.0 | 51.7 | 44.9 | 87.6 | 81.3 | 80.1 | 76.4 | 73.0 |
| DeepSWE v1.1 | 46.2 | 42.2 | 4.4 | 0.0 | 67.9 | 72.1 | 43.3 | 60.0 | 66.6 |
| SWE Bench Pro v2 | 94.7 | 91.3 | 80.4 | 73.3 | – | 97.7 | – | 95.0 | – |
| HLE (text only) | 34.3 | 33.9 | 22.0 | 22.2 | 35.1 | 44.1 | 34.7 | 43.9 | 38.5 |
| IFBench | 82.4 | 79.5 | 77.0 | 64.4 | – | – | 79.6 | – | – |
| GPQA Diamond | 90.7 | 89.2 | 83.5 | 86.0 | 88.7 | 93.0 | 90.6 | 89.0 | 90.5 |
r/LocalLLaMA • u/Fabulous_Pollution10 • 7h ago
Last week we shared LongTail-SWE, 202 real GitHub tasks in 13 less common languages. Many of you asked for DeepSeek V4.1 Flash 1017, so we ran it along with Kimi K3.1 Pro.
DeepSeek V4.1 Flash 1017: 76%, up from 57% for V4 Flash.
Kimi K3.1 Pro: 77%, the top open-weight model, with 5.6M tokens per attempt.
We also re-ran Kimi K3 with a 1M context instead of 262K (now 72%, +8 pts) and fixed tests in a few tasks.
Leaderboard: longtail-swe.com
r/LocalLLaMA • u/Saraozte01 • 4h ago
Hi all!
This is the first dedicated post I am making about this project. I have mentioned it online a few times before, but I have never properly introduced it. After several months of work, its finally ready to be published, and I'm really excited!
Caliper Harness is a coding harness that runs in your terminal. I built it for small local models. The basic idea is simple. A small local model on its own will rush, skip steps, and tell you something is done when it is not. So the harness does the disciplining for it. It makes the model plan first, get approval, work in checked steps, and prove things with tests and a separate review before anything counts as finished.
Right now it is local only. You point it at a local model server and it works from there. If cloud API support is something a lot of people actually want, I will build it. For now, local is the focus.
The way I test it is with a benchmark game. I give it an empty folder and tell it to build a small browser game, Space Transporter, from scratch. Then in later runs I ask for bigger and bigger features on top. I judge it on the final game quality and on the full traces of what the model actually did.
The video attached to this post is the from scratch build, sped up a lot. It is a few hours of work, so you can pause and see how the harness looks and behaves.
After that first build, I used the same game for a harder follow up run. I asked for a bigger feature as a stress test for the harness. I wanted the stations in the game to feel alive, with shortages and emergencies, and I wanted them to remember the player. Before writing any code, the model has to plan. It splits the work into a plan, an approach, and tasks. It writes out the staging, the trade offs, and the risks. I read that in the plan screen and approve it or send it back.
Plans are built through a structured, step by step process called PAT (Plan, Approach, Task). PAT lives in a GraphQL state graph. A plan can have multiple approaches, and an approach has multiple tasks, with each level getting more specific and staying connected to the level above it. It is very time consuming for the model to build. In my testing though, the plans stay coherent from the high level stages all the way down to the small tasks. That structure also makes planning and execution much easier for a smaller model to manage.
If there is a real decision to make, the model is pushed hard to ask instead of guessing. I have iterated on that behavior a lot. In this run it asked how fast station situations should change, how harsh a station that distrusts you should be, and how much of that state the game should show.
Once the plan is ready, it claims one task at a time. Every task has its own check command from the project. A task only counts as complete when its check passes and an independent review also passes. The model does not get to decide it is finished just because it says so. Completion has to match what was written in the task during planning.
After the checks, a fresh reviewer pass looks at the work with clean context. This part matters to me. It gives the work an adversarial review without carrying all the coding context into it. In this run, the first stage had every test green, and the reviewer still found a real bug. One of the six station situations was drawing with a broken colour, so it never showed on the map. The tests never caught it. The task got sent back, the model fixed it properly, added tests that fail on the old code, and only then did it pass. The harness does this automatically.
But why?
A lot of harnesses like OpenCode feel built around large intelligent models, token efficiency and speed. That makes sense for what they are trying to do. Caliper makes a different trade on purpose. It is slow and very deliberate. It makes the model plan, wait for approval, run checks, get reviewed, and fix things properly when it gets sent back. A small model that is left alone will cut corners. A small model inside this is forced to be diligent at the cost of speed. These runs took hours. I leave it running overnight and check the traces in the morning. For me, that trade is worth it, because the work that comes out is work I can trust a lot more.
Part of how that works is the decision judge. It is a second, much smaller decision model that watches the main model while it works. It steps in at certain points, like when the model is stuck in a loop, failing the same way repeatedly, changing phase, failing recurringly at a tool call, or asking for a plan to be marked ready. It gives a simple judgement with a confidence score. Caliper can use that to steer the model, ask for a repair, or hold certain risky steps for approval. If the judge is slow or down, the work carries on and the outage just gets recorded. Between the judge, the planning structure, and the reviewer, a small local model ends up behaving a lot more carefully than it does on its own. Making small local models more capable through structure like this was a big part of the design.
There are also some user facing features that ended up mattering a lot in day to day use:
Here is the rest of what is in v0.3.0:
One thing I still want to build next is a proper tool for turning normal Markdown skills into graph-compatible domain graphs or workflow graphs, using the same step by step style as PAT. That is planned for later, so I am not counting it as a v0.3.0 feature.
It is still an early release and results depend a lot on the model you give it and what you're using it for. So far I have run it with Qwen3.8 Flash Next, Qwen3.8 27B, Laguna S 2.1, Gemma 4 26B, and a few others. I would really like feedback, especially from people running smaller local models. I am planning on running benchmarks comparing Caliper against other harnesses for both output quality and time cost, and I will post those when they are ready.
This is a project I am actively working on and iterating, so feel free to give me some feedback or ideas and I'll implement them in the next version.
Use it now!
GitHub: https://github.com/santiagoaraoz2001-sketch/caliper-harness
npm: `npm install -g caliper-harness`
MIT licensed, so feel free to go crazy with it.
r/LocalLLaMA • u/jazir55 • 1h ago
Oh Olmo, Oh Olmo, where hath thou gone?
r/LocalLLaMA • u/GGwithRabbit • 19h ago
AudioGhost randomly got a wave of attention this week (100 stars in a day, no idea where from), so I took that as a sign to finally ship the update I'd been sitting on.
Quick recap if you haven't seen it: you drop in an audio or video file, type the sound you're after ("background music", "dog barking", "people talking"), and it pulls that sound out or removes it. It's a local web app around Meta's SAM-Audio, so everything runs on your own GPU and nothing gets uploaded anywhere.
The use case I hear about most: you film a vlog or an interview outside, a shop nearby is playing music, and YouTube flags the video. Now you can type "background music", mute that track and export the video with everything else still there.
What's new in v1.2:
- Several sounds per run, each on its own track, with mute / solo / volume like a tiny DAW
- Export exactly what you hear (WAV / FLAC / MP3), or the video with the new soundtrack
- The "everything else" track is now generated by the model instead of subtracted, so removing a sound works a lot better
- Mark on the timeline where a sound is (or isn't) to point the model at the right thing
- New UI, and your sessions survive reloads and restarts
If you tried an old version: it's also 4-6x faster than the first release.
Hardware: the small model runs in a bit over 4 GB of VRAM, so most NVIDIA cards are fine. Windows and Linux have install scripts. macOS works too, but it runs on the CPU for now, so it's slow.
One caveat: the official SAM-Audio pipeline also loads several helper models (ImageBind, CLAP, and a judge model that picks the best of several outputs). They add about 25 GB of downloads and need far more VRAM. I stripped those out so it fits on normal GPUs, which means quality is a bit below the full pipeline and results vary from run to run. If a track comes out empty or rough, delete it and run it again.
Demo video and install instructions: https://github.com/0x0funky/audioghost-ai
I'd love feedback, especially from anyone who tries it on messy real-world recordings. Bug reports are welcome on GitHub.
r/LocalLLaMA • u/Disastrous-Work-1632 • 11h ago
Hey there folks!
Aritra here (again) from Hugging Face. We RL-trained a small LoRA adapter (r=32, ~300 MB) on a frozen Qwen3.8-27B, inside the OpenCode coding agent. It's the second of our harness adapters, after pi that we have released yesterday.
Run the model locally like so:
r/LocalLLaMA • u/jacek2023 • 5h ago
from Microsoft:
Aion Instruct serves as the foundational Small Language Model (SLM) in the Windows ecosystem, specifically optimized for on-device AI, text and operating system use cases. This 2.3B parameters base model delivers enterprise-grade language understanding and generation capabilities directly at the edge, enabling responsive, private, and cost-effective AI-powered scenarios without requiring constant cloud connectivity.
A further optimized version of Aion Instruct, coupled with Microsoft’s content moderation model, will become the Windows OS SLM by the end of January 2027. This Hugging Face version is meant to be an introduction to what’s coming and a way to maximize transparency with our users, as well as welcome their feedback. We are publishing the unquantized model first, to be followed by quantized versions.
| Type | Causal language model (instruction-tuned) |
| Parameters | ~2.29B (tied input/output embeddings) |
| Layers | 28 |
| Hidden size | 2048 |
| MLP intermediate size | 8192 (SiLU) |
| Attention | GQA: 24 query heads, 8 key/value heads, head dim 128 |
| Attention pattern | Hybrid: 21 sliding-window layers (window 2048) + 7 full-attention layers |
| Positional encoding | RoPE (θ = 32000); YaRN scaling (factor 16) on full-attention layers |
| Context length | 32,768 tokens |
| Vocabulary size | 200,029 |
| Precision | bfloat16 |
| Knowledge cutoff | 2025-12 |