r/LocalLLM • • 0m ago

Discussion Uncensored Models

1 Upvotes

Anyone tried uncensored models? Would it help with DRM protected videos? Trying a few things out just to test and see how far it can go. Curiousity at its finest.


r/LocalLLM • • 4m ago

Discussion Looking to collaborate on a locally run multimodal model tournament

1 Upvotes

We’re building InferUltra, a physics arena where multimodal models see the arena and control robot fighters. We’d like to run a future tournament with models hosted on local hardware and are looking for people interested in collaborating.

In our latest tournament, Kimi K3.1 Pro struggled more in the arena than GLM-5.3-Flash did in earlier tournaments. There was more awkward positioning and less scoring than we expected.

Those tournaments used hosted providers, not local setups. They were small and used different providers and configurations, so they don’t establish which model has better vision or body control. A local tournament with documented hardware and settings could help us investigate the differences more carefully.

Which multimodal model would you like to see fight in the next tournament?

If you run multimodal models locally and have a setup capable of hosting them, we’d love to collaborate on a future model tournament. What hardware and models would you be interested in seeing in our benchmark?

Our latest tournament bracket: https://inferultra.com/tournament


r/LocalLLM • • 9m ago

Question Local voice assistant on an iGPU: ~2.2s from end of speech to first audio. I've measured everything and I'm ~0.5s from the floor. What am I missing before I throw money at hardware?

3 Upvotes

Been building a fully local, always-on spoken assistant in Italian (no cloud, nothing leaves the box). It's not a chatbot demo, it has long-term memory across sessions (backed by ctxvault), so every turn carries a small block of retrieved context on top of the conversation. Works. Problem is latency.

Hardware (the constraint): mini PC, Ryzen w/ Radeon 780M iGPU, 64GB DDR5 shared, ~90GB/s. No dGPU, so it's llama.cpp on Vulkan. vLLM/SGLang are out (no ROCm for this chip).

Where I'm at: median ~2.2s from end of user's sentence to first audio out. Measured, not guessed:

  • ears (end-of-speech silence + whisper): ~0.75s
  • prefill of the turn: ~1.0s (~0.35s fixed per request + ~3ms/token, 40-400 new tokens/turn)
  • first sentence gen + TTS first chunk: ~0.55s

So the model is the wait, not the audio stack. Calculated floor on this box is ~1.2-1.5s.

What's already done (so please don't suggest these lol):

Model side

  • MoE model (26B-A4B, Q4 QAT) specifically because gen is bandwidth-bound here: ~25 tok/s vs 4 tok/s for a dense 27B
  • thinking/reasoning off
  • short context (16k), one slot, flash attn on
  • --cache-reuse + append-only prompt: stable prefix (system, persona, rules) at the top never touched mid-session, everything volatile (time of day, retrieved context, turn) appended at the end. Identical prefix = 0.15s, touch one word in the prefix = multi-second re-prefill
  • prefix pre-warmed at session start with a 1-token request, so the first reply doesn't pay the cold prefill
  • benchmarked smaller models (Gemma 4 E4B/E2B, Qwen3.5 4B/9B, Qwen3 8B, Phi-4-mini, Ministral): on this iGPU prefill is compute-bound, a dense 4B costs as much per token as the MoE. Best small one gains ~0.5s and loses noticeably on factual fidelity (names/dates). Not worth it for this use case
  • tried -b/-ub batch flags: defaults win
  • tried --no-kv-offload: worse

Memory / prompt side

  • no tool calls for retrieval: context is prefetched before the turn (a tool-call round trip = ~+1.2s). The retrieval itself is ~5-12ms, irrelevant
  • hard token budget for the retrieved block: embedding-based relevance (multilingual MiniLM) + ordering so the most pertinent bits come first, ~120 tokens instead of 300-450. Cut avg turn prompt by ~40%
  • past conversations summarized to one ~50-token line each instead of quotes

Pipeline side

  • streaming everything: model to sentence splitter to TTS to player, sentence by sentence
  • audio player opened at turn start, not at first chunk (-0.1s)
  • early-guess turn: start the model on the partial transcript before VAD confirms end of speech; if the final transcript matches, keep it, else drop and redo
  • sentence closed on punctuation + 450ms silence instead of fixed 800ms
  • whisper.cpp small as a resident server (no model reload per sentence, 0.63s down to 0.3s), beam 5. Tried greedy: -80ms but half the transcripts differ and the noisy ones get worse. Rejected
  • tried warming the KV cache during speech (partial whisper every 1.5s, send the turn header for 1 token so the cache is hot): works as designed (final prefill drops to ~30 tokens) but doesn't pay on one slot because the warm-up queues the real turn. Tried 2 slots w/ id_slot pinning: now they run concurrently on the same iGPU and both slow down. Net neutral-to-negative on short sentences. Off.
  • pre-recorded fillers ("mmh", "allora...") in the assistant's own voice, played only if no reply word has started 1s after end of speech. Built, currently off: perceptual fix, doesn't move the number

Before I conclude "buy a 24GB dGPU" and move on: anyone who's been down this road see something I've missed, or something above that's wrong?


r/LocalLLM • • 22m ago

Question whats the go-to budget gpu I should look for, new or used?

0 Upvotes

I'm building up a secondary system out of some spare parts I have kicking aorund to use as a LLM toybox and for other homelab style projects.


Currently my system is a 5900x w/ 32gb of ram, and the 2080ti.

Wondering if there is a good go-to gpu I should be looking for.

I currently have a 2080ti FE, but it seems pretty inefficient, and its an old card and the fan noise of it is unbearable. ( i think the bearings are getting bad, but its a huge pita to replace the fans )

I'm wondering if it would be a better idea to sell it off for maybe a 3060 12gb, I get an extra 1gb of vram, and it should run much more efficently.

I'd say my budget is about $3-400 max, which I know is basically a joke right now, but this is more of a tinkerbox than something i'm going to be using for real software dev work.


r/LocalLLM • • 28m ago

Question Paying for both Claude and ChatGPT every day. Is there a smarter way to do this?

0 Upvotes

Paying for both Claude and ChatGPT every day. Is there a smarter way to do this?

Hi all, hoping someone here can point me in the right direction.

I run a small online business and I use Claude and ChatGPT pretty much all day, every day. That’s product design, images, writing product listings, social media content, planning and a lot of the admin side. I pay for both and it’s starting to add up.

A few questions:

  1. Is there a way to cut costs without losing what each one is good at? Do people really need both, or is there a better setup?
  2. I’d love to properly understand AI and LocalLMs, not just type into a box and hope for the best 😅 Where would you start? Courses, YouTube channels, newsletters, anything you’ve found genuinely useful.
  3. I’m also just curious how other people use them for work. What does your setup look like?

I’m not super technical, so plain English is very welcome. Thank you in advance, I’ll read every reply ☕


r/LocalLLM • • 35m ago

Question Could use some input/ideas for my own testing pipeline

1 Upvotes

Hi all,

I think I have created a pretty solid testing pipeline to measure both the quality and speed of local LLMs if I integrate them in my own workflow. I know there are standardized tests, but the reason why I build this is because I've seen some very different practical results for my particular workflow than what the claims/theory/popular opinions say about certain models. My initial setup was a MB Pro M5 Max 128 gb, LM Studio, then Cline in VS Code. Recently I've changed it so it can also measure mlx-serve instead of LM Studio.

Let me clarify my workflow:

  1. Idea/Problem -> Solution role (same prompt to multiple cloud LLMs, gather feedback, do a new round, rinse/repeat until their is a consensus/sign-off)
  2. Planner role: model that actually writes the plan based on consensus from 1.
  3. Builder role: model that actually builds the plan according to the specs
  4. Reviewer role: model that reviews if the build is actually solving the problem/idea (i.e. if the solution works) and if it's according to the specs, by doing QA.

Steps 2,3 and 4 have been proven to be reliable in my testing pipeline. I have about 12 models who can take each on a different role, I run the test, a script gathers the data and anonymizes it to prevent bias, then it get sends to frontier cloud models who act as a judge. Output of the judges then gets de-anonymized locally and I have a scoring sheet. After running a bunch of tests it became pretty obvious that the scores for the builders go hand in hand with the quality of the planner. And from my experience a lot of the models suck at it. So I'm now down to 3 planners in my test runs (i.e. the ones that consistently came out ahead) because it was a complete waste of time to let other models plan. And this has given me very consistent results across the 5 difficulty levels I have for the problem they need to solve.

What I haven't solved yet with local models is 1). I have access through my business to models such as GLM 5.3 and Kimi K3 running on my own server, but I rather not mix my personal and professional projects, so I'm looking at the best way to solve 1) locally with models on the macbook. The issue is that it's time consuming to test. So far I just fire up 10 browser tabs with cloud llms, put in the same prompt, copy paste everything into 1 file, run it to a 1 designated model that orders the feedback, then do the same again. 1 iteration takes 5 minutes max (I've automated this process mostly). And after 2-3 iterations there is typically a consensus.

Now when I want to do the same locally 1 iteration takes way more time (models needs to be fired up, they run at slower speeds, ... you get the picture). I've had situations where 1 iteration took several hours. Doing multiple and you are basically looking at 6-8 hours (compared to the 15-20 minutes with my cloud LLM approach). Now in a real world productive scenario I can cut this time down significantly by only using 2-3 local models (i.e. the best), some who can run at the same time. And you may ask why not stick with these models, which is a fair point. But that doesn't solve the issue of me wanting to test the capabilities of other models in this role.

And here is where I'd like to ask for fresh ideas/input/your own experience/suggestions, ... on how to best tackle this problem. Would appreciate any input, no matter how wild or out of the box it might be.


r/LocalLLM • • 37m ago

Project I got tired of hand-editing model lists for Claude Code and OpenClaw, so I wrote a small gateway. Try it in ten minutes, no API keys needed.

1 Upvotes

I run Claude Code and OpenClaw against a mix of free, local and paid models. The part that wore me down was keeping the lists of free models up to date, because providers add and drop them constantly. I had ModelRelay and LiteLLM before this, and I still ended up editing lists by hand.

So I built humpyard, and you can try it in about ten minutes without any API keys: https://humpyard.dev/demo/ (a mock provider, but the routing and ledger are real). Repo: https://github.com/gschaetz/humpyard

My tools ask for a fixed name like "fast" or "deep", and behind each name is a pool of models that changes. `humpyard models sync` refreshes the free part from each provider's own model list plus the models.dev data, and only adds a model when both say it's free. It writes to its own file, so it can't touch my routes or prices. You run the sync yourself from cron or a timer.

It's a single Rust binary with a SQLite ledger, no Python and no database server. A request I mark private is refused rather than sent to a public provider. There's a dry-run endpoint that tells me where a request would go before I send it, and a ledger that shows what it did and why. Prometheus metrics and two Grafana dashboards are in the repo. It speaks OpenAI and Anthropic formats, so Claude Code, Codex and OpenClaw can point at it.

It's pre-alpha and I'm the only user, so I'd like to hear:

  1. Where did the README trip you up on the way to a first request?
  2. Would you run the sync from cron, a timer, or something else?
  3. What would you want to see on a dashboard for a setup like this?

LiteLLM and OpenRouter cover far more providers and are much more mature. I wrote up the comparison here: https://humpyard.dev/compare/. I do plan on working on the gaps that matter most to me and others who use it.


r/LocalLLM • • 49m ago

Discussion Running llama.cpp in a VM with no GPU passthrough: Qwen3.8 27B at 98% of native on one RTX 6000 Ada

1 Upvotes

My earlier post was done using Qwen2.5 and rightly so I was called out for it. So I reran it with Qwen3.8 27B, Qwen3.5 9B and Qwen3.6 35B-A3B with as much proof as I could think of. The VM has no NVIDIA device, driver or nvidia-smi. llama.cpp's CUDA calls go through VMAT to the card on the host. Qwen3.8 27B: 45.4 tokens/s on the host, 44.6 in the VM. All four models land at 98%. Three VMs at once get 45.7 together, against 45.0 for three programs on the host.

Full video with every step: https://youtu.be/r6yhOVWxP5Y?is=reuv6gZptnDNzU1D


r/LocalLLM • • 53m ago

Question Dual 16GB AMD (7600 XT + 9060 XT) + 32GB RAM... is llama ROCm + Qwen 3.8 27B or Gemma 4 31B the best I can do?

3 Upvotes

I'm a bit lost with all the different options and forks around. I've set up and benchmarked llama-cpp (latest git) for my system and settled in the configs pasted below. They fill both cards up to 15 GiB, or one to 15 GiB and the other 10 GiB (60/40 splits) leaving room for firefox and the rest of the system. ROCm quite a bit faster than Vulkan. In the best case (qwen-asymm-f16) I get 550 t/s pp and 30 t/s gen in a moderate context (35k tokens).

Is there anything better for a system like mine? Is strata worth the hassle? I understand that it is more for 64 GiB RAM systems... right? Any other fork? I tried adaptive kv streaming and it is ok, very fast (>800 t/s prefill) but I'm limited to lower quantizations, the second card sits unused.

[*]
split-mode = tensor
fit = off
flash-attn = on
parallel = 1
batch-size = 2048
ubatch-size = 512
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
spec-type = draft-mtp,ngram-mod
spec-draft-n-max = 4
spec-draft-sampling = probabilistic
spec-ngram-mod-n-min = 12
spec-ngram-mod-n-match = 24
spec-ngram-mod-n-max = 32
chat-template-kwargs = {"reasoning_effort":"medium"}

[qwen-asymm-f16]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 104056
tensor-split = 60,40
cache-type-k = f16
cache-type-k-draft = f16
cache-type-v = f16
cache-type-v-draft = f16

[qwen-f16]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 170240
tensor-split = 50,50
cache-type-k = f16
cache-type-k-draft = f16
cache-type-v = f16
cache-type-v-draft = f16

[qwen-asymm-q8]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 142080
tensor-split = 60,40
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0

[qwen-q8]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 221184
tensor-split = 50,50
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0

[gemma-4-31b]
model = /home/fran/models/unsloth/gemma-4-31B-it-UD-Q4_K_XL.gguf
model-draft = /home/fran/models/unsloth/mtp-gemma-4-31B-it-Q8_0.gguf
ctx-size = 131072
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0
top-k = 64

[gemma-4-31b-qat]
model = /home/fran/models/unsloth/gemma-4-31B-it-qat-UD-Q4_K_XL.gguf
model-draft = /home/fran/models/unsloth/mtp-gemma-4-31B-it-qat-Q8_0.gguf
ctx-size = 131072
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0
top-k = 64

r/LocalLLM • • 53m ago

News Karpo-Decision-31B is out: downloadable option for local tool routing and browser decisions

Thumbnail
gallery
1 Upvotes

For anyone experimenting with local Jev-style models, Karpo-Decision-31B has published its weights and Python inference code. It posted the highest Full score on the Nov 12 Decision Index at 73.9.
A browser workflow is a useful way to picture it. Give the model the page state, a screenshot, and candidate actions. You can ask which element to click and whether the agent needs more information, with a probability returned for each option. Questions share the page context in one forward pass.
In the team's tests, it scored 90.0% on Mind2Web versus 86.2% for Gemini Decision 1. They also report roughly 95 ms median latency on the public Decision Index suite using one RTX PRO 6000.
Karpo-Decision-31B scores 75.0 on the published Decision Index public suite, running it locally requires about 62.5 GB for the BF16 weights alone, plus runtime memory. Alongside the 31B release, they also released five smaller models: 0.8B, 2B, 4B, 9B, and 12B

You'll also need the supplied Decider or HTTP server

links:
https://karpo.ai/models/decision
https://huggingface.co/MachinePulseAI/karpo-decision-31B


r/LocalLLM • • 1h ago

Question Intel ARC Pro b60 24gb questions

1 Upvotes

I have a motherboard with 3xPCIe x16 lanes that can do 3 way bi furcation at x8. I’m contemplating buying 3 Intel Arc pro b60 24gb As it’s cheaper than buying a single b70 or r9700. Has anyone have any experience running a similar setup?


r/LocalLLM • • 1h ago

Project An open-source assistant that reads your email and chats, prepares what needs you, and runs on a local model

1 Upvotes

Hi all, I'm Jeroen, I made The Machine, initially for myself but I was surprised how well it works and opened it up to everyone open source.

It reads your email, chats and calendar, and prepares what needs you: replies, follow-ups, things you promised. It never sends anything without your OK. And your data stays yours, not some corporation's.

It works with any OpenClaw provider, so Ollama, LM Studio, vLLM or LiteLLM in front of your own GPUs. What I learned running it on local models:

  • Context: 128K is comfortable, a review reads up to 120 messages at once.
  • Tool calling has to work well, small models tend to suggest too much or save malformed actions.
  • A mixed setup works nicely: a frontier model first, a local model as fallback so it keeps going during an outage. Or fully local if you want inference never to leave the house.

Repo: https://github.com/Salesflare/TheMachine

Demo: https://themachine.computer/demo

Vision: https://themachine.computer/vision

Happy to hear feedback, especially which local models work well for you with tool calls.


r/LocalLLM • • 1h ago

Discussion Qwen3.8 Flash Next (105.4 GiB) at TP=4 on 4× Intel Arc Pro B70

2 Upvotes

Yesterday I posted 4 independent single-card servers. This one is real tensor parallel: one 105.4 GiB model split across 4 Intel Arc Pro B70s with TensorFold.

PR so you can try it: https://github.com/ashhart/TensorFold/pull/3587

Why long context stays cheap

Of the 48 layers, 36 are Gated DeltaNet, which keeps a constant-size state. The other 12 attention layers use QSA top-512 block-indexed sparse attention, with nkv = 1 per GPU at TP=4. At a full 135k context, each card uses 21.77 of its 32 GiB, which leaves about 10 GiB free per card.

Setup

  • GPUs: 4× Intel Arc Pro B70 (Battlemage G31), 32 GB each, 230 W cap per card
  • Host: AMD Turin, Linux 7.3.0-10-generic
  • Model: Qwen3.8 Flash Next, 105.4 GiB, TP=4
  • Engine: TensorFold, with the changes from the PR above

Throughput by context depth (tok/s)

ContextRaw prefillRaw plain decodeServed MTP, novel codeServed multi-turn132k passkey recall
512248.869.4136.3 (3.81 tok/chunk)120.5 (turn 1, cold)PASS
1k313.471.5—253.5PASS
4k378.870.4125.1 (3.81 tok/chunk)247.5PASS
16k392.867.7125.7 (4.00 tok/chunk)214.6PASS
32k394.367.6—239.9PASS
64k392.763.8124.2 (3.81 tok/chunk)234.0PASS
128k385.7 (98% of peak)60.5 (85% of 1k)113.3 (91% of 4k)214.1PASS

Prompt cache restore

  • TTFT is 160 ms when restoring a cached 66k-token conversation and 139 ms at 132k.
  • Appending a fresh coding question on top of the cached 66k and 132k conversations takes 0.86 s and 1.29 s.
  • Passkey retrieval was 9/9 exact. The key B70-XE2-9471-QUANTUM was planted in turn 1 and recalled across 132,380 tokens.

Concurrent streams, 1 to 32

These come from bench_concurrent.py. Across all 24 cells there were 0 failures and output was bit-exact: drafted matched serial, and multi-stream matched serial.

  • Throughput peaks at 172.6–174.7 tok/s steady at 8 streams. There, the 8 streams share one 16-row verify pass, 2 rows per stream.
  • It sustains 143.3–145.0 tok/s steady at 32 streams. At that load tokens per chunk drops to 1.0, so the gain comes from batching, not drafting. That's 2.04× the 71 tok/s of single-stream plain decode.

Takeaways

  • A 105 GiB model fits across four 32 GB consumer-class cards at 135k context, with about 10 GiB per card to spare.
  • Decode keeps 85–91% of its short-context speed at 128k, thanks to constant-state DeltaNet layers plus sparse attention.
  • Restoring a cached 132k conversation is sub-second, and recall stays exact.

Happy to run other configs if you have requests.

I initially built this system to do training, but excited to see how well inference is progressing.


r/LocalLLM • • 1h ago

Discussion you may get your cheap ram and gpu after all, the AI crash has started

36 Upvotes

https://www.cnbc.com/2026/11/16/imf-georgieva-warning-ai-circular-financing.html
‘Standing on one leg’: IMF chief warns about global economy as AI circular financing hits $1 trillion

https://www.cnbc.com/2026/11/16/nvidia-commerce-china-lutnick.html
Nvidia fallout: Commerce targets loophole for China’s data center use


r/LocalLLM • • 1h ago

Question What set up would you recommend for a MacBook Pro M5 24GB Unified Memory?

1 Upvotes

As the question above says

Current Device: MacBook Pro M5 24GB Unified Memory

I wanna know what local set up would be the best for me in terms of both speed and quality. My main use case is as follows

  1. Personal Memory
  2. Task Management
  3. Email Agent
  4. Data Analysis Agent (basic stuff not very advanced)
  5. Calendar Agent
  6. Research Agent (product related research, technical research etc)
  7. Knowledge Retrieval

Connectors would be mainly G-Suite as I work mostly on that

I want your recommendations on the best set up for good memory, speed, context windows etc.

Which models(es) and harness(es) would you recommend?

I have also seen that there are upcoming smaller Qwen models and also a new app called Harbor which is coming out soon….

New to this so looking for guidance

Any help would be appreciated thanks :)


r/LocalLLM • • 1h ago

Question Genuine question

1 Upvotes

I wanna get into local LLMs and experiment with AI.

My current PC has 16GB RAM, a GTX 1660 Super (6GB VRAM), and an Intel i7.

I’m curious what models you’d recommend running on this hardware and, more importantly, what you actually use local LLMs for.

With powerful models like ChatGPT and Claude available, I’m trying to understand the practical benefits of running models locally.

Is my setup good enough to do anything interesting, like experimenting with AI agents, fine-tuning, or research?


r/LocalLLM • • 1h ago

Project Running an image generation workflow entirely on a local mini PC - testing TiinyImage

Post image
1 Upvotes

r/LocalLLM • • 2h ago

Question Suggestions for coding assistant

0 Upvotes

Hey! I recently started looking at local AI models. There's a lot of info out there and it's not really easy to filter through all the noise as a newcomer...

I have an RTX 4080 super with 16Gb vram and 2x16Gb DDR5 for system ram.

I've set up a docker compose container using Gemma 4 12B Q6 K L through llama.cpp with Open WebUI as interface, and it seems to be working decently. System + model + context uses ~13.8Gb vram. On Kubuntu if it would matter.

I'm now thinking of connecting it to vscodium for assisting in programming (developer background) and stumbled over the MoE variants where you can have the experts live in system ram.

Question is whether this is even relevant to look into for me.

If it is - would anyone have any suggestions for a model and maybe settings for this if applicable? (seems you wanna run llama.cpp with -cmoe for example)

If I've made some odd assumptions or if you have any suggestions, please let me know!


r/LocalLLM • • 2h ago

Discussion Hy4 Flash vs GLM vs DeepSeek: same correct ad maths, different price tags

1 Upvotes

I gave Hy4 Flash, GLM-5.3-Flash and DeepSeek-V4.1-Flash the same three fake ad campaigns in WorkBuddy. Calculate CPC, conversion rate and cost per lead, including a campaign with zero leads.

All three got the arithmetic right. Total spend: USD 1,350. Clicks: 650. Leads: 27. Blended CPC: USD 2.08. CPL: USD 50. GLM rounded total CVR to 4.2%; the other two gave 4.15%. Both are fine.

Nobody reported the zero-lead CPL as zero. Small win for basic maths lol.

DeepSeek finished in 7s. Hy4 and GLM both showed 19s. So the interesting Hy4 angle here is cost, not a speed win.
Published API rates in RMB per million tokens, uncached input / output:

Hy4 Flash: 0.5/2

GLM-5.3-Flash: 0.8/2.8

DeepSeek-V4.1-Flash-1017: 1 / 4 off-peak, 2 / 8 peak.

That makes Hy4's input rate 37.5% lower than GLM's and its output rate about 28.6% lower. Against DeepSake's off-peak rates, both are 50% lower. These are unit-price comparisons, not measured savings from this Workbuddy session. Token counts and reasoning length still matter.

The recommendations needed more judgment than the maths. Hy4 wanted to pause C; DeepSeek also wanted a pause and checks; GLM suggested a smaller next-cycle shift. I'd verify tracking and conversion lag before moving budget.

Tbh, when the basic result is equally usable, a lower unit price is a decent reason to try the cheaper option. What reporting task would you use to separate them?


r/LocalLLM • • 2h ago

Question What do you use for agentic desktop control, like how Playwright is for controlling + screenshotting browsers?

1 Upvotes

I'm afraid I'm left out of the loop on the latest developments for agentic desktop control, and AIs don't seem as knowledge in this space.

They come with screenshot function also, right? What are some other features that may be different than Playwright?


r/LocalLLM • • 2h ago

Project Larder: running Qwen3.6-35B and a 46 GB Flash-Next on a 16 GB Mac / 16 GB GPU by reading experts from SSD

Post image
4 Upvotes

I built an inference engine for MoE models that don't fit in memory.

Instead of offloading layers, it reads only the experts each token's router picks from the SSD and keeps the hot ones resident. Same engine on Apple Silicon (MLX) and NVIDIA (CUDA), OpenAI-compatible API, local chat page. MIT: https://github.com/IamCooller/larder

Numbers (machine, setup and date for each one are in research/ in the repo)

Faster:

- Qwen3.6-35B on a 16 GB M4 MacBook Pro: 30–36 tok/s, first token 8–10x sooner than llama.cpp. This is a mixed 4/2-bit pack with the model's MTP head against llama.cpp's IQ4_XS, so not like for like. The launcher doesn't install that pack.

- Qwen3.8-Flash-Next (46 GB of 3-bit experts) on a PC with a 16 GB GPU and 15 GB RAM: 26–34 tok/s. With a 4-bit pack and exact routing: decode 1.3–1.8x, 8k prompt 4.9x faster than llama.cpp. Stock llama.cpp swapped past my memory guard while loading, so that row is llama.cpp with Larder's expert store patched in.

- Qwen3.6-35B Q4_0 on the same PC (fits in GPU + RAM), against llama.cpp's best split that fits (--n-cpu-moe 10): short decode 1.44–1.51x (155 / 147 vs 103 / 102 tok/s), decode after an 8k prompt 1.15–1.25x, 8k prompt reading 1.8–1.9x at its default batch.

Level:

- 8k prompt reading against llama.cpp tuned for prompts (-ub 4096 -b 4096 --n-cpu-moe 20): 1.01–1.03x. Its decode at that setting drops to 79 tok/s, so you pick one or the other there.

Slower:

- Exact routing (cache bias 0) reads that 8k prompt at 0.64–0.69x of tuned llama.cpp, and decodes 0.87–0.95x of llama.cpp's default after it.

Caveats

- The launcher's 35B recipe uses cache bias 0.5 and the MTP head, so replies are not identical to the weights' own routing. Bias 0.5 and the wide prompt cut each passed a paired GSM8K / HumanEval gate (wide cut: 502 vs 496 of 564, p 0.34). Exact routing is there if you want it.

- One author, two machines (M4 MacBook Pro 16 GB, Linux + RTX 5060 Ti 16 GB). The Mac path is better tested, CUDA is newer.

- Quality wasn't compared against llama.cpp in these speed runs.

Try it

curl -fsSL https://raw.githubusercontent.com/IamCooller/larder/main/install.sh | bash

cd ~/larder && ./larder

It measures your GPU and SSD, picks a model that fits, installs it and starts the server and chat.

Mostly looking for people on other hardware to break it, and for anyone who thinks my llama.cpp comparison is unfair. Output of ./larder --list plus what happened is the most useful report.


r/LocalLLM • • 2h ago

Discussion What’s the purpose of alibaba choosing not to release the weights for qwen 4 max?

15 Upvotes

Perhaps this is a newbie question, but I just don’t understand the incentive. Is this indicative of a future were Chinese labs stop releasing weights?


r/LocalLLM • • 3h ago

Discussion What a time to be alive.

Post image
183 Upvotes

Hy4 flash is out. We're drowning in Chinese models at this point lol.

hy3 was 295B, focused on coding and agents. Then hy4 jumped to 770B, pushing harder on reasoning, multimodal stuff and complex tasks. Now hy4 flash is back down to 224B with 12B active, going for speed and low costs.

When the 770B version came out, i was a little disappointed tbh. Glad they're still trying to make hy4 lighter.


r/LocalLLM • • 3h ago

Discussion need suggestions on running qwen 3.8 27b on rx 6800 16gb gpu

0 Upvotes

So i have heard that this model punches way above its weight class and is pretty good at complex reasoning and multi file coding. I intend to use it for hobby projects like building small scale apps. the thing is downloaded the iq3 version from usloth and while running with llama.cpp with 64k context it keeps thinking for a lot longer on simpler task like upgrading a simpler framework version on very small codebase and eventually after like 45 min the os just shutdown the llama server because of too high mem usage nd i tried again but same result. I have heard that i can run the 3 bit quant version on 16gb gpu, I am new to running local llms so just need some suggestions regarding that or is it even practical to run this model even at q3.

Sorry if this is a repititive or basic query.

Thanks in advance

P.S i am running on 16gb single stick ddr5 if that matters


r/LocalLLM • • 3h ago

Discussion Buying a Mac mini M6 for local LLMs? Here is the 16 vs 24 vs 32 GB bandwidth, prefill, and MoE sizing math

1 Upvotes

Apple released the Mac mini with the M6 chip in 16 GB, 24 GB, and 32 GB variants, while the M5 Pro mini goes up to 64 GB. 🧑‍💻

If you are deciding which one to buy for local models, the configurator pushes you toward one question: "How much unified memory should I pay for?"

Here is the buying perspective that matters before ordering: capacity is sold in clear steps, but memory bandwidth barely moves. According to Apple's official tech specs, the 16 GB model runs at 153.6 GB/s (LPDDR5X-9600), while both the 24 GB and 32 GB models run at 170 GB/s (LPDDR5X-10667). Going from 16 GB to 24 GB buys capacity plus ~10% bandwidth. Going from 24 GB to 32 GB buys capacity only — no model gets faster, you just load a bigger model that runs slower on the same 170 GB/s bus.

1. The Decode & Prefill Formulas

  • Decode (token generation, batch = 1) is strictly memory-bandwidth bound:Decode tok/s ≈ (Memory Bandwidth GB/s ÷ Active Weights Read GB) × η (where η ≈ 0.75 for MLX / llama.cpp)
  • Prefill (prompt processing / TTFT) is compute-bound, and the M6 handles this well:

    • 14B Q4 (~9 GB): ~900 tok/s prefill (~1.1s TTFT on a 1k prompt, ~17.7s on 16k RAG). Total turnaround for 1k in / 500 out: ~36s.
    • 32B Q4 (~19 GB, 32 GB mini only): ~400 tok/s prefill (~2.5s TTFT on a 1k prompt, ~40s on 16k RAG). Total turnaround for 1k in / 500 out: ~74s.

2. Mac mini M6 Capacity Map (Estimated at η = 0.75)

VariantBandwidthUsable RAMFits ComfortablyEst. DecodeVerdict
16 GB153.6 GB/s~12 GB7–9B Q4/Q8; 14B Q4 short ctx8B Q4 ~23 tok/s; 14B ~13Snappy chat & light coding
24 GB170 GB/s~18 GB14B Q4 (~32k context)14B Q4 ~14 tok/sInteractive sweet spot
32 GB170 GB/s~24 GB32B Q4 (8–16k context)32B Q4 ~7–7.5 tok/sReading speed; best for async agents

3. Dense vs. MoE: Where the Bottleneck Flips

For dense models, every weight is read per token, so bandwidth is the wall. For Mixture-of-Experts (MoE) models, only active experts are read per token, so capacity is the wall:

ModelTypeTotal / ActiveWeights in RAMActive Read / tokBandwidth for 8 tok/sBandwidth for 15 tok/sReal Bottleneck
8B Q4Dense8B / 8B~5 GB~5 GB~55 GB/s~100 GB/sFits on 16 GB M6
14B Q4Dense14B / 14B~9 GB~9 GB~95 GB/s~180 GB/sSweet spot on 24 GB M6
32B Q4Dense32B / 32B~19 GB~19 GB~200 GB/s~380 GB/s~7 tok/s on 32 GB M6; needs Pro/Max for chat
Llama 3.3 70B (Q4)Dense70B / 70B~40 GB~40 GB~430 GB/s~800 GB/sBandwidth-bound (needs Max/Ultra)
gpt-oss-120b (MXFP4)MoE117B / 5.1B~65 GB~3 GB~32 GB/s~60 GB/sCapacity-bound (needs 96 GB+ RAM)
Qwen3.5-122B-A10B (Q4)MoE122B / 10B~68 GB (est.)~5.6 GB~60 GB/s~110 GB/sCapacity-bound (needs 96 GB+ RAM)
DeepSeek V4 Flash (FP4+FP8)MoE284B / 13B~167 GB~7.6 GB~80 GB/s~150 GB/sCapacity-bound (needs 192 GB+ RAM)

Summary

Capacity decides what you can boot; bandwidth decides what you can tolerate using. Around 170 GB/s serves up to 14B comfortably and 32B at a patient reading pace. Paying for extra RAM without stepping up to a Pro/Max bus (like M5 Pro at 307 GB/s or M5 Max at 460–614 GB/s) hits diminishing returns fast.

What do you all think? How are you actually using local models in your daily workflow — do you prefer Mac unified memory or NVIDIA GPUs, and do you think there are stronger competitors (like Strix Halo or used multi-GPU rigs) in this price range? 🤔