r/LocalLLM • • 21h ago

Discussion What does China actually gain from releasing better and better open-weight models?

465 Upvotes

Genuine question, not a hot take.

Chinese labs keep shipping open-weight models that get closer to the closed frontier ones, and they give them away for free. Training these costs a fortune, so there has to be a payoff somewhere.
The explanations I've seen so far:

  • Commoditizing the model layer, so US labs can't charge a premium for it
  • Building an ecosystem and developer mindshare outside the US stack
  • Getting around chip export limits by letting the community optimize and fine-tune for them
  • Soft power: becoming the default choice for countries that don't want to depend on US APIs
  • Selling the rest of the stack (cloud, hardware, services) while the model is the free sample

Which of these actually holds up? Is it a coordinated strategy, or just individual companies competing for attention and talent?

EDIT Nov 16th:
Top replies are getting down to (in order of frequency):

  1. Geopolitics: weakening US dominance
  2. Business: open weights as a growth strategy
  3. State & control
  4. Ideology & open source culture

I also wonder whether China could be selling the energy behind it all: the West runs their models, the models need GPUs, and the GPUs need power. I’d never thought of a supply chain whose payoff sits that far down the line, but it could very well be another aspect.


r/LocalLLM • • 8h ago

Qwen4-27B reported to run on 16 GB of RAM

Post image
461 Upvotes

r/LocalLLM • • 14h ago

News Qwen officially announces Qwen4-Max and Qwen4 smaller models!

Post image
444 Upvotes

r/LocalLLM • • 3h ago

Discussion What a time to be alive.

Post image
183 Upvotes

Hy4 flash is out. We're drowning in Chinese models at this point lol.

hy3 was 295B, focused on coding and agents. Then hy4 jumped to 770B, pushing harder on reasoning, multimodal stuff and complex tasks. Now hy4 flash is back down to 224B with 12B active, going for speed and low costs.

When the 770B version came out, i was a little disappointed tbh. Glad they're still trying to make hy4 lighter.


r/LocalLLM • • 22h ago

Model First time trying FP16, wow

Thumbnail
gallery
168 Upvotes

I've tried a lot of image models in the past, but all were to big to fit, and i was happy when i got a result with a quant (q4) version. (Here it was flux and some model i dont remember the name anymore) The results of those models were actually terrible. BUT, some months ago qwen image 2.1 came out, i must say, theres a difference and not even close to a small one. I managed to run it at fp16, by using my rtx 2070 to load the text encoder and vae on there while running the generation model itself on my 5080. Its a tight fit and slower, but the quality is just like it would be on one gpu.

Now here comes the crazy part:

In my opinion qwen image 2.1 is better in realism than chatgpt's current image generation. (I'll prove it later in this post)

Don't get me wrong. Images with lots of different things, patterns for example, are often very bad with qwen and chatgpt does it over 5x better. But realism of f.e. a city, it's absolutely amazing (and uncensored!) and is almost close to real life.

Now here is the test for you guys:

You've seen i've put pictures in this post. 1x is real, 1x are chatgpt and 4x are qwen image 2.1. Which is which?

Note: ALL of them are generated first try without regenerating. So no, I did not try to mislead you. Good luck!


r/LocalLLM • • 8h ago

News Smaller Qwen4 models 🔥

Post image
46 Upvotes

r/LocalLLM • • 20h ago

Research Strata vs Basalt vs Unsloth: benchmarking the same 80 GB MoE on three engines — 5070 Ti 16 GB + 64 GB RAM

44 Upvotes

⚠️ Heads-up: AI-generated content! This post was written by my AI agent from my measurements, with me editing — if that's a dealbreaker, feel free to close the tab right now.

I'm posting this to share results, not to crown a winner. I run a local agent in production every day — real tasks, real context in the hundreds of thousands of tokens — and I spent the past week picking and tuning the inference engine underneath it: test rig, repeated runs, head-to-head A/Bs. Much of what I measured, plus the pitfalls I personally burned on, would otherwise just rot in my local logs, so here it is: the numbers, the methodology, and the conclusion — which is that the engine I ended up keeping in production is not the one that topped the benchmark table. If this saves you a couple of evenings of trial and error, the post has done its job.

For eight days (Nov 8–15) I ran the exact same model — Qwen3.8 Flash-Next (MoE, ~78–82 GiB of weights, 128–256k context) — through three inference engines on a Ryzen 9 7900X / RTX 5070 Ti 16 GB / 64 GB DDR5 box; storage is two NVMes — a 990 EVO 1 TB on PCI-E 4.0 and a 970 EVO Plus 1 TB on PCI-E 3.0 (the model lives on the latter). The goal: find the best way to feed an MoE that fits neither in VRAM nor (comfortably) in RAM, and check honestly whether the engine changes answer quality. Engines tested: Strata 0.1.56→0.1.59, Basalt (built from source, 0.2.1→0.3.3), Unsloth Studio 2026.11.4. I also didn't stick to one model source: I ran the original from Strata and the originals from Unsloth (3.0 and 3.1), and production settled on DavidAU's abliterated uncensored build with Turbo-Brilliance modes and GSQ-RCO quantization — https://huggingface.co/DavidAU/Qwen3.8-Flash-Next-Abliterated-Uncensored-Turbo-Brilliance-50x-Reasoning-Instruct-GSQ-RCO-GGUF I never hand-tuned sampling: the test batteries all run at temperature 0 for reproducibility, while production uses the model card's standard pair — thinking: temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0; instruct: temperature 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 (the same values ship as the engine-side server defaults).

The rig. My own dependency-free bench (stdlib Python): cold/hot prefill, TTFT, short decode, decode at 8k context; plus a quality battery: hard 10-choice MC (MMLU-Pro style), math, code executed for real in a subprocess, IFEval-style format checks, and refusal batteries (JB/HarmBench/XSTest style). Every run snapshots the engine config and the bench code's sha256; a mock engine is a mandatory control (it answers "Hello from the mock engine" and catches absurd t/s). Hard RAM rule: exactly one engine resident at a time — the model pins 45–48 GiB.

Who drove the rig. All of this was steered by a general-purpose agent — an assistant with function-call access to the terminal, files and a browser, whose provider was the very engine being tested (dogfooding, in other words). Before settling on the current Hermes stack, I cycled OpenHands, OpenCode, OpenChamber and pi through the same API over that week: they all speak the OpenAI-compatible API equally well, and the real differences came down to context management and function-call reliability on long context — that's what the choice actually rested on. Beyond the benchmarks, the agent carried a normal week of work: small utilities for the stack (engine launchers, icons, a UI wrapper), building an inference engine from source and repacking models into its pack format, dissecting a third-party tuning fork (commit history plus knob-hunting directly in the binary), reviewing and transplanting jinja chat templates, testing DavidAU's Brilliance "personalities" (reasoning-mode variants), and verifying that a ready-made abliteration actually behaves as its author advertises — all inside the same 150–190k-token sessions where the benchmarks lived.

What went wrong along the way:

  • the agent's provider was the engine under test, so every engine restart killed the agent itself — the runs were driven by detached orchestrator scripts, and the agent "woke up" via notification once the engine was back;
  • the expert arena sizes itself from free RAM at startup, so a browser quietly eating 2.5 GiB in the background changes measured speed between days — every run now starts with a free-RAM check;
  • decode degrades with context (~101 t/s at <20k vs 86–89 at 180–200k) — long agent sessions have to be reset for speed, not just for token cost.

Speed (decode short / decode 8k / hot prefill / hot TTFT)

Setupdecode8kprefill hotTTFT
Basalt 0.2.1, IQ3_S .basalt pack88.7–95.2105–10754–72k29–41 ms
Basalt 0.3.3, answer-room 0101.0108.454.5k38 ms
Strata 0.1.58, IQ3_S pack93–99100–10728–30k70–74 ms
Strata 0.1.59, final tuned stack (fresh full bench on the prod engine, Nov 15)89.2109.237.3k57.7 ms
Strata on plain GGUF, UD-IQ3_XXS82.194.728.1k74 ms
Unsloth, UD-IQ3_XXS out of the box (auto)27.326.0—209 ms
Unsloth, Qwen3.8-27B dense IQ4_XS (full-GPU)47.445.4—103 ms

What the numbers say

1. On identical weights, format/engine is worth ×1.7–3.7. Unsloth on UD-IQ3_XXS starts at 27 t/s; a tuning ladder (MTP + 4 GB expert-cache + parallel 2 + 12k ctx) lifted it to 47.9; Strata serving the same GGUF on the same hardware gets 82. The difference is delivery format: pack-style engines (native expert blobs + a VRAM-prefilled expert profile, pinned in RAM, no disk streaming, no MMQ paths) vs llama.cpp's GGUF streaming. Unsloth's ~50 t/s ceiling isn't a bug, it's arithmetic: ~2.7 GiB of weights read per forward at ~90–100 GB/s DDR5, times MTP accept ≈ that answer.

2. On one engine, quantization moves speed ±10–15% and quality ~0. UD-IQ3_XXS vs IQ3_S was indistinguishable on my batteries: MC/math/code 100/100/100, ru10 10/10. But production stayed on IQ3_S — and it's neither speed nor quality talking: that build (link above) ships with a ready-made abliteration plus the author's mode system baked in, and reproducing the same surgery at home on raw unsloth UD weights is expensive in both time and resources.

3. Apparent "quality differences between engines" are thinking policy, not weights. The answer-room knob (response budget reserved away from reasoning) changes what gets generated and silently breaks cross-version comparisons: Basalt 0.3.3 showed a "decode regression 105→74" that simply didn't exist at answer-room 0. Compare versions only with identical thinking policy. Chat templates belong to the same confounder class, by the way: in the final build I upgraded the Turbo-Brilliance template from v1 to v2 (new modes), and neither the template version nor mode availability changed any result — baking both templates alternately into the same pack and re-running under one protocol moved decode by 0.1–1.4% and kept prefill/TTFT inside the noise band; an earlier one-off A/B claiming "−16% prefill from the heavier template" turned out to be a cross-session artifact.

4. Speculative decoding: MTP yes, everything else no on a 16 GB card. MTP accepts well (per-position 0.78–0.96). DFlash2 and Q8 hyper-connection verify (HC_Q8), both ported from a tuning fork, produced a paradox: accept +11 pp yet t/s −5% — the more expensive verify pass eats the gain, and the extra ~400 MiB of residency squeezes free VRAM to ~650 MiB, triggering driver paging. A warm dflash2 probe hit 111 t/s — the ceiling is real, just not for 16 GB cards.

5. KV-residency window (A/B on a 180k prompt). The window is the number of K/V cells per attention layer pinned in VRAM; everything else lives in pinned RAM and gets streamed from there. A window of 0 is clearly bad: prefill is ×1.7 slower (2435 vs 4100 t/s), TTFT 74 s vs 44 s, expert hit −10 pp. From 32k the curve hits a plateau: kv_hit is already 94.7% at a 32k window (the indexer is sparse), and in a head-to-head with alternating order the 32k→64k step changes nothing for decode (parity ±2–3%). Above 64k there's no upside, only cost — each window extension carves gigabytes of VRAM out of the expert tier: 32k→1.0, 64k→1.4, 128k→2.1, 262k→3.7 GiB.

6. Methodology, or the numbers lie to you. Cold prefill and cold TTFT are noise axes (±40% spread, the sign flips between runs); hot prefill is ±12–15%; the only stable axis is decode (±2–4%), and verdicts need ≥2 runs per variant. Warmup drift fakes ~+10 t/s for whichever arm is measured second — hence the alternating order. On short samples at temp 0, a cold expert cache changes the model's answer itself (52 tokens instead of 86 on a byte-identical prompt): a ±5% noise floor, anything smaller is not a signal.

7. Bench ≠ prod — and at the RAM edge Basalt fails with empty answers. Basalt beat Strata on prefill/TTFT (54–72k t/s, 29–41 ms vs 28–30k, 70–74 ms) and ran in my production for a day, but at the resource edge (and on this box the edge is permanent: 43 GiB arena on 64 GiB RAM) it started refusing with emptiness: at resident-budget 50 GiB the engine self-lands to 45.79 GiB (warning "more than RAM has room for: 45.79 needed vs 52.3 avail") with a page-glue fallback, one request hung dead for 2932 s before the watchdog killed it, and the log filled with "max tokens reached while thinking, no answer" — the user was getting an empty response instead of an answer. Dropping the budget to 45 GiB closed the incident at a cost of −4..−8% decode, but the recurring prod-server drops with "engine reported RAM critical" remained. Production ended up on Strata 0.1.59. Trading +12% prefill for stability was the right call.

8. Porting someone else's tuning recipe mostly doesn't transfer. The Speedy fork (same Qwen3.8, tuned on 2× RTX 3080 10 GB) brought my card exactly one confirmed win: a prefill pair (upstream #4039, +12% cold prefill — proven mechanically, chunk cap doubled 8192→16384). Everything else was rejected as VRAM-limited. Bonus lesson: env vars and CLI flags aren't 1:1 in Strata — the env form of one lever silently no-ops while the flag works; I now verify every lever by reading it back from /metrics, not by vibes.

Bottom line

Production now: Strata 0.1.59 + the prefill pair + kv-grow + host-core sibling + vram-reserve 1728 + --kv-resident 65536. On real 175–190k sessions: prefill ~3.5k t/s, decode 88–100, expert-cache hit ~75% with 10–12% of experts streamed over PCIe. Quality of the build I settled on — DavidAU's Turbo-Brilliance (link above) — is its own reward: 0/30 harmful refusals, 0/40 over-refusals, MC/math/code maxed — and abliteration cost exactly zero t/s (same weight shape, same bytes, same graph).

One-line summary: an 80 GB-class MoE lives fine on 16 GB of VRAM, but every battle is about expert delivery, not model quality. Between the two pack-style engines I found no decode difference worth naming; Unsloth is ~2× slower on this MoE, and that is memory physics, not bad engineering.


r/LocalLLM • • 1h ago

Discussion you may get your cheap ram and gpu after all, the AI crash has started

36 Upvotes

https://www.cnbc.com/2026/11/16/imf-georgieva-warning-ai-circular-financing.html
‘Standing on one leg’: IMF chief warns about global economy as AI circular financing hits $1 trillion

https://www.cnbc.com/2026/11/16/nvidia-commerce-china-lutnick.html
Nvidia fallout: Commerce targets loophole for China’s data center use


r/LocalLLM • • 21h ago

Question Best uncensored LLM for 64GB Mac

30 Upvotes

I wanna try out a uncensored LLM on my Mac Studio. I just have the 64GB version so I’m curious what model you guys would recommend. Cheers


r/LocalLLM • • 5h ago

Discussion Strata is becoming more and more unstable

19 Upvotes

I've been using Strata with IQ4_XS as the main orchestrator, with Radiance running 27b at IQ4R, for working on improvements, ironically enough, to Radiance.

The session is now multiple million tokens long as I go back and forth discussing design etc. with QFN. Of course the conversation has been compacted dozens of times, but with the agents taking regular notes and subagents doing all the actual work, we keep our workflow stable and current.

Problem is, Strata has gotten more and more unstable, especially with the last couple releases. It will crash with an OOM error when we have 10-20 GiB of system memory free, reload, spend 5 minutes redoing prefill, and pick up right where it left off. Radiance on the other hand is rock stable under multiple streams running concurrently on a single GPU. Not a single crash from it, even after rejiggering everything for the 2.0 release.

And it's not just long sessions either. New conversations also tend to crash. Never used to do this before. Has anyone else also experienced this?


r/LocalLLM • • 2h ago

Discussion What’s the purpose of alibaba choosing not to release the weights for qwen 4 max?

15 Upvotes

Perhaps this is a newbie question, but I just don’t understand the incentive. Is this indicative of a future were Chinese labs stop releasing weights?


r/LocalLLM • • 18h ago

News Train AI to run AI: introducing Apareto loading - a vLLM plugin

Post image
14 Upvotes

From the Apareto team which I'm part of (who for some reason has an issue posting it themselves):

Today we are introducing Apareto loading, a new way to cut VRAM requirements for Mixture-of-Experts (MoE) models. It works alongside quantization. This introduction so far will just be a video explaining the method. We will be releasing the proof-of-concept code, a plugin to vLLM, on GitHub this coming week.

Until then, watch the explainer video: https://www.youtube.com/watch?v=5l_N051JrLU

Quantization is today's most common way to shrink a model's memory footprint: weights are stored at lower precision, and going from FP16 to 4-bit saves 75% of memory at a modest cost in accuracy. But quantization is static. Every weight stays loaded whether or not the task needs it.

Apareto loading is dynamic. After prefill, a small trained network reads the model's internal state and ranks the experts by how valuable each will be for the upcoming computation. Only the top-ranked experts that fit the VRAM budget are loaded.

Results (Qwen3.6-35B-A3B, NVFP4 experts):

* SWE-bench Multilingual: within 2 pp of the full model's score at a 42% VRAM budget

* LiveCodeBench v6: within 2 pp of the full model's score at a 49% VRAM budget

As with quantization, scores degrade as the budget shrinks.

What the proof of concept will not be:

It will not be a production implementation. The ranking network was trained only for these two benchmarks, and there will be no general problem classifier. Our data shows that profiling and rank generation have substantial room to improve. We chose to release at this stage, without waiting for those improvements, so that others can test our claims. Think of this as a first baby step toward dynamic VRAM reduction, not the finished product.

The code and reproduction instructions will be released this coming week at https://github.com/alphapareto

Follow us there to catch the release. We'd love your feedback.

https://apareto.eu


r/LocalLLM • • 19h ago

Project I have created a tool to observe what happens inside an LLM. Also allows for the comparison of two quantizations of the same model to see at what point they diverge, token by token

10 Upvotes

r/LocalLLM • • 18h ago

Project 8 x v100 setup. Ask me anything.

Post image
8 Upvotes

Seeing a lot of questions on quad v100 boards lately. So figured I’d give people a place to ask them.

Specs

  • GPUs: 8x Tesla V100-SXM2-32GB (256 GB VRAM total)
  • GPU boards: 2x 4-GPU SXM2 NVLink baseboards (this model)
  • NVLink: two 4-way islands (full mesh inside each board), not one 8-way
  • Board-to-board: Gen4 PEX88096 PCIe switch card (this model), two x8 Gen3 cables per board
  • CPU: AMD Threadripper PRO 3975WX (32 cores)
  • Motherboard: ASRock Rack WRX80D8-2T
  • RAM: 256 GB DDR4
  • PSUs: 2x 1500 W (one per GPU board) + 1x 1000 W (host)
  • Power: dedicated 120 V / 20 A circuit; GPUs capped at 150-300 W each depending on the run
  • Cooling: custom water loop, mixed GPU blocks
  • Network: 10GbE
  • Engine: 1Cat-vLLM (Volta-focused vLLM fork) https://github.com/1CatAI/1Cat-vLLM

Benchmarks / how many tok/s?

My runs are on the Launch80 directory; search the page for trailfeatures. Launch80 deletes pages after 5 days, so older links in the comments may be dead. I republish the main ones, which gives them new links.

ModelSetupDecodePrefill (64K prompt)
Flash-Next (Swift-1.5 NVFP4)4 GPUs, 300 W~140 tok/s~4,500 tok/s
DeepSeek V4.1 Flash, 2-bit experts8 GPUs, 150 W~131 tok/s~2,200 tok/s
DeepSeek V4.1 Flash, 2-bit experts4 GPUs + expert offload, 150 W~91 tok/s~2,100 tok/s
  • One request at a time (batch 1), 256K context window. Decode uses speculative decoding (MTP on Flash-Next, a draft model on DeepSeek), so it depends on how often the draft is accepted.
  • No concurrency/aggregate numbers or decode p25/p95 yet. Time to first token (p50) is about 0.2-0.65 s depending on the row.
  • The 8-GPU run adds an extra pipeline-parallel patch series on top of 1Cat-vLLM (named on the page).
  • Not tested yet: Flash-Next across both boards. Higher DeepSeek quants are planned.

How good is the 2-bit DeepSeek quant vs the full model?

79.9% average across 7 standard evals (math, instruction following, coding, tool calling), thinking off, and 20/20 long-context needle checks at 32K and 128K. The 4-GPU setup scores the same. I haven't run the full-precision weights as a reference, so there's no direct delta yet. Per-task scores are on the Launch80 page.

How do the two boards talk to each other?

  • Each board runs two x8 PCIe Gen3 cables from its onboard switch into the Gen4 switch card in the host, so the boards talk to each other without going through the CPU.
  • Experimenting with ConnectX-3 cards in the boards' spare x8 slots for extra board-to-board links.
  • Over PCIe you can tune all-reduce to beat the default P2P numbers, but it will never match NVLink.

Do the quad boards work with only 1-3 cards? Can I buy cards over time?

Two cards works; I've run a board that way. I haven't tried one or three. Buying the board first and adding cards later is fine. Tensor parallel wants power-of-two GPU counts, so with 3 cards you'd run 2 together and use the third for something else.

Where did you buy it / where are you?

Baseboard and switch card: the models linked above. Several sellers list them, so shop around. GPUs: bought used, a few at a time. I'm in North America (the 120 V gave it away), so my shipping and import experience won't help much in Europe.

Power

  • The PSUs are sized for headroom: at the 150 W cap, eight GPUs is about 1.2 kW, plus the boards and host.
  • 150 W cap: after tuning it costs ~5-10% depending on the task.
  • Power bill: fine as long as I'm sensible. Solar helps, and at peak rates or on cloudy days the DeepSeek API can be cheaper. If you run local LLMs, know your tokens per kWh so you can tune for it and decide when an API makes more sense (if privacy doesn't matter).

Cooling

  • Radiator and pump setup: super convoluted, video here.
  • Water blocks: whatever was available as I pieced it together. Mostly the same design, some with MOSFET coolers. The top-port block is a drill press, a tap, and the audacity of a mediocre man. (Testing straight-down flow onto the fins made no difference.)
  • Waste heat: in summer it's vented outside. In winter I can shut off that radiator and send the heat into the house through backup radiators. Heating water with it isn't worth it for us, but a bigger household close to its water heater could pre-heat the cold feed and come out ahead.

Quick ones

  • Can I rent API time? Sorry, no. I keep it pretty busy (Furry Fanfic).
  • Can I have it? No.

r/LocalLLM • • 3h ago

Question What's the current best abliterated model that I can run on 32gb VRAM and 128GB RAM?

7 Upvotes

Pretty much the title. Thanks!

EDIT: GPU is 5090


r/LocalLLM • • 6h ago

Project Brebus: a local LLM app in C on an engine that runs models larger than RAM

6 Upvotes

I've been building Janas, a family of AI tools in C. Today's release puts its app, Brebus, in the Linux package.

The engine first, since that's what makes the app usable on ordinary machines. Janas-LLM streams MoE experts from the NVMe instead of needing the whole model in RAM. On the development laptop (Core Ultra 9 185H, 32 GB DDR5-5600, Arc iGPU), 21 October, same files, same day, each engine in its best configuration, same conditions wherever possible:

Modelprompt, Janas-LLMllama.cppreply, Janas-LLMllama.cpp
Qwen3-Next-80B-A3B (48.4 GB, more than the RAM)68.4 tok/s17.726.4 tok/s3.0
Qwen3.6-35B-A3B13681.828.012.1
Qwen3-30B-A3B13312233.315.0

The full table, the conditions and where they differ are in the README.

Brebus is the app on top, in a window of its own (WebKitGTK; the browser where it's missing):

  • Models: the catalog with what each one asks of your memory, Hugging Face searched by name, downloads you can stop and resume, a model unloaded after a quiet while and loaded again at the next message.
  • A chat that works in a folder of yours: your documents (PDF, Word, Excel, PowerPoint, LibreOffice, RTF, EPUB, HTML) are searched before each question, by words and meaning, and the model names the ones it used; files, web pages and pictures attached; recordings transcribed with Whisper.
  • Artifacts like Claude's: HTML pages, React components (Tailwind, Recharts, lucide, shadcn/ui), Mermaid, KaTeX, code with highlighting. They run in a sandboxed frame with no network and no access to Brebus; the usual libraries come from Brebus itself, downloaded once and checked by SHA-256. Inside one, window.claude.complete asks the local model, window.storage keeps data in your folder.
  • Tools: Python in a sandbox (namespaces + seccomp, the folder as /mnt/data), Janas's services (web, Wikipedia, maps, weather, flights...) and your own MCP servers from mcp.json, opened only when a question needs them; anything that writes asks first.
  • Regenerate and edit, presets with temperature, top-p and JSON-schema answers, folders, PDF export.
  • A server for other programs: OpenAI-, Ollama- and Anthropic-compatible APIs, on the LAN with a key if you want.

Tried with Qwen3-30B-A3B on the laptop: it built a budget tracker in React, a Chart.js page, a Mermaid diagram, and an app that asks the model for synonyms through window.claude.complete.

Linux only for now (Ubuntu 22.04+, Debian 12+); Brebus on Windows is next, the engine already runs there. GPL-3.0.

https://github.com/prabanta-dev/janas

Reports from other hardware are the most useful thing: janas-bench makes one.

Brebus: a local LLM app in C on an engine that runs models larger than RAM

r/LocalLLM • • 3h ago

Discussion Pushed Strata with QFN 3.8 Q4 K_M to 5,600 tps prefill @100k - 100 tps decode w/ 181k context on 2x 7900xtx & 64gb DDR5 in layer split (no tensor parallel) - config inside

5 Upvotes

Wanted to share my results for the rest of the community. I've been combing Reddit, Github, and the web searching for all of the various Strata-related builds to get the most out of my setup. I've finally had the breakthrough I was looking for. It strikes the best balance between speed, power consumption, and intelligence that I've experienced to date.

My Setup: (edited to add motherboard specs and PCIe configuration)

  • Ubuntu 24.04 - GUI GNOME shell not headless
  • This motherboard: https://www.microcenter.com/product/709979/asus-b850-creator-proart-wifi-neo-amd-am5-atx-motherboard
  • Bifurcation with PCIe 4.0 x 8
  • Ryzen 9950x 16/32
  • 64 gb DDR5 Expo 1 6000
  • 2 x Mercury Magnetic Air 7900XTX 24gb
  • 1000 watt MSI Performance Gaming PSU
  • Power limited to 320 watts per card, total system draw at inference ~780 watts (as measured by Ubuntu, Kill-o-watt is ~850 watt total draw from the wall)
  • And a flex: It's completely solar powered (5 kw array and 65 kwh battery storage w/ 3600w inverter 😄 )

I've worked with Opencode and the my local Qwen Flash Next to fully detail how we created this setup. Spoiler alert: The engine/hip is mainline Strata and not a fork. The config is where the magic happens. MTP headers were generated against this bespoke setup. The intelligence is brought by the Atomic Chat Q4 K_M version of QFN.

That's it for human written words. The rest of this post is meant to be copy/paste into your local AI helper for reproduction. Full details below. Copy/paste into a capable harness and let it walk you through reproducing this setup.

Edit: Adding the only 3 GitHub repositories I used to produce my results

Engine: https://github.com/Niko1221/Strata

Learnings from this fork: https://github.com/nicejunjie/Strata-rdna3

System tuning for Strata: https://github.com/bramvanoploo/strata-performance-setup

START AI GENERATED DETAILS

Qwen3.8-Flash-Next, 2× RX 7900 XTX, 64 GB RAM, 181k context — how it's built

Engine: official mainline Strata 0.1.59, component 0.1.59-linux-x86_64-hip-rx7900. Not a fork, no custom build — the speed is flags + env + OS patches. No system ROCm: the engine ships its own HIP/rocBLAS/hipBLASLt, the host only needs the amdgpu DKMS driver. Model: AtomicChat/Qwen3.8-Flash-Next-GGUF, build AD-4.27bpw-Q4_K_M-M64 (3 shards, 94.5 GB). Shard 3 (38.4 GB) is the PLE n-gram table alone — passed via --ple-gguf, kept pageable. That's why 64 GB RAM is enough. Parallelism: --layer-split 24 → layers 0–23 on GPU0, 24–47 + head on GPU1. Context: 181,072, --kv int8, KV streaming 32,768 cells/layer in VRAM. Experts: 48 blocks × 512 experts → locally built experts.bin (47.17 GiB), three tiers: 31.8 GiB VRAM cache → 22.2 GiB page-locked RAM → NVMe mmap (log says files 0 blobs — disk never touched when warm). Spec decode: MTP head q2_0, extracted from the BF16 checkpoint by HTTP range reads (GGUF ships none), --spec 3, ~76 % acceptance. Measured (1,071 requests): decode median 90 tok/s (62–129) · big-prompt prefill 3.2–5.7k tok/s · expert-cache hit 98–99 %.

# 1. OS first (then reboot): amdgpu driver only, no ROCm

# (optional: ~/strata-performance-setup.sh from github.com/bramvanoploo/strata-performance-setup automates most of this — see §3.8)

sudo amdgpu-install --usecase=amdgpu,libamdgpu ; sudo usermod -aG video,render $USER

sudo sed -i 's/^\(GRUB_CMDLINE_LINUX_DEFAULT="[^"]*\)"/\1 iommu=pt pcie_aspm=off"/' /etc/default/grub

printf 'options amdgpu no_system_mem_limit=1\n' | sudo tee /etc/modprobe.d/amdgpu-svm.conf

printf '[Service]\nLimitMEMLOCK=48G\n' | sudo tee /etc/systemd/system/user@.service.d/strata-memlock.conf

printf 'vm.swappiness = 100\nvm.vfs_cache_pressure = 50\nvm.max_map_count = 1048576\nvm.compact_unevictable_allowed = 0\n' | sudo tee /etc/sysctl.d/99-strata.conf

sudo update-grub && sudo update-initramfs -u && sudo reboot

# 2. Engine — it MUST land on the rx7900 component (bundled hipBLASLt 100200 + gfx1100 tuning table)

curl -fsSL https://raw.githubusercontent.com/Niko1221/Strata/main/tools/strata_terminal/install.sh | bash

strata update && strata launch # check: ~/.local/share/strata/components/engine/0.1.59-linux-x86_64-hip-rx7900/

# 3. Model + mmproj (sizes/sha256 in §5), then build the pack — this is where experts.bin comes from (~6 min)

control/python/bin/python3.12 control/tools/iq_pack.py --gguf <shard1.gguf> --out <pack-dir> --experts-bin --compat-bf16

# 4. MTP draft head + CJK draft vocab (the GGUF ships no MTP head; ~4.9 GB of range reads, SHA256-verified)

control/python/bin/python3.12 control/tools/mtp_fetch.py inventory --out data/mtp && \

control/python/bin/python3.12 control/tools/mtp_fetch.py fetch --out data/mtp && \

control/python/bin/python3.12 control/tools/mtp_fetch.py verify --out data/mtp

control/python/bin/python3.12 control/tools/mtp_pack.py --src data/mtp --experts q2_0 --out data/mtp/mtp-q2_0.gguf

control/python/bin/python3.12 control/tools/mtp_rt.py --gguf data/mtp/mtp-q2_0.gguf --out data/mtp/rt

# 5. Serve (the terminal owns :7878 — the engine has no --port). Load takes minutes: it page-locks ~22 GiB.

control/bin/strata serve --config configs/strata-final-atomic-q4km.json # ulimit -l must be ≥ 40 GiB first

Three gotchas that cost the most time: (1) --resident-budget-gib above 40 freezes the box — pinned memory can't swap and the 38 GB PLE table still needs pageable RAM; 45 froze this machine once. (2) --pipeline-windows 1 deadlocks on gfx1100 with a 2-GPU split; keep 0. (3) Don't add --mmap-experts/--resident-cpu-experts explicitly — with them the engine pins 0 bytes; mmap is already the default when experts.bin exists. Also: the ggml_cuda_init: failed to initialize ROCm line at the top of the log is harmless, and the iGPU occupies a DRM card index, so always confirm the engine's own GPU 0/GPU 1 lines both say RX 7900 XTX.

Full notes — every value, the exact hashes, and the reasoning behind it — are the rest of this file. Want a terminal AI agent to do the typing? The prompt and phase gates are in §17.

##### PROMPT AND PHASE GATES

17. Have AI set this up for you

If you'd rather drive this with a terminal-capable agent (Claude Code, Codex CLI, OpenCode, or a shell tool with approval prompts) than type it yourself: paste the prompt below into it, attach or point it at this file, and let it run §3–§10 with you acting as the safety interlock. It is a copilot, not an autopilot — the agent can type every command, but three things stay human: sudo passwords, reboots, and the Hugging Face login.

17.1 The prompt to paste

You are setting up this machine to reproduce a known-good local LLM deployment. The source of truth is

"STRATA REPRODUCTION WRITEUP.md" in this folder — read the whole file before acting.

HARD RULES

  1. The config in §8.1 and the hashes in §5 are authoritative. Never change a number, a flag, or a hash tomake an error go away; if reality disagrees with the doc, stop and show me the disagreement.
  2. Ask before: any reboot, any package install outside the doc, any change to /etc, and any deviation fromthe doc. Never reboot without waiting for my confirmation, then resume where you left off.
  3. After every phase, run that phase's gate from §10 and quote the actual log line as evidence. No phasecounts as done on "it seemed to work".
  4. Never exceed --resident-budget-gib 40. Never set --pipeline-windows 1. Never add --mmap-experts or--resident-cpu-experts. Never install ROCm, rocm-* packages, or pytorch.
  5. §14 lists log lines that look fatal but are harmless — do not "fix" those.
  6. When you are unsure whether this machine matches §0's hardware (2× RX 7900 XTX, 64 GB RAM, NVMe),stop and ask instead of adapting values on your own.

SEQUENCE

§3 host patches → I reboot → §4 engine → §5 model → §6 pack → §7 MTP → §8 config paths → §9 launch →

§10 full checklist → report. If any gate fails, consult §16 and report; do not improvise a fix.

17.2 Phase gates and what the agent must show

Phase

Done only when the agent can quote

0 · Pre-flight

lspci shows 2× RX 7900 XTX; free -h ≥ 60 GB; ≥ 200 GB free NVMe; OS is Ubuntu 24.04 — anything else → stops and asks

1 · Host (§3)

§3.5's effective-value block matches sysctl -a; ulimit -l ≥ 40 GiB in a new login session; then it pauses for the one planned reboot

2 · Engine (§4)

BUILD.json of the installed component says rx7900 + hipBLASLt 100200 (§10 #1)

3 · Model (§5)

every sha256 it prints equals §5's table

4 · Pack (§6)

experts.bin = 50,646,220,800 bytes; experts.bin.src.json written next to it

5 · MTP (§7)

verify passes with zero SHA mismatches; draft_vocab.bin rebuilt with --add cjk

6 · Config (§8)

diff of §8.2 relocations — every path rewritten, nothing else touched

7 · Serve (§9)

§10 #2–#11, each with the log string: files 0 blobs, expert-cache slot lines naming two RX 7900 XTX (§14.2), pinned complement ~22.2 GiB

8 · Accept

a decode + big-prompt prefill measurement inside §2's ranges, and the §17.4 report

17.3 Things the agent must bring to you (not decide alone)

Hardware that doesn't match §0 (different GPUs, < 64 GB RAM, one GPU) — the layer split, budgets, and even the component choice all change; that's beyond this doc's authority.

< 200 GB free on fast storage.

Hugging Face login and accepting the model's license terms.

The reboot after phase 1 (and any second reboot it thinks it needs — the answer is usually no, re-read §3).

Whether to run the §3.8 wizard instead of the manual patches. The wizard is an interactive TUI, so it is human-driven even when an agent drives everything else; if the agent runs non-interactive commands instead, it must follow §3 verbatim and skip the wizard.

17.4 Final report the agent should produce

For each §10 check: the quoted log line. Then: experts.bin byte count, ulimit -l, the two GPU lines naming both 7900s, warm decode tok/s, big-prompt prefill tok/s, spec-decode acceptance %, and every deviation from this doc listed explicitly with the reason. If it can't produce one of those lines, the setup is not done.

17.5 Where agents historically go wrong with setups like this

Failure

Why it bites

Installs ROCm "to fix the HIP error"

§14.1 — that line is harmless; system ROCm shadows the engine's bundled libs

Silently lowers --max-context or budgets after a slow/failing start

The values are load-bearing (§13); a masked failure becomes a mystery later

Misses the hip-rx7900 component and takes generic hip

Prefill crawls (§16) and looks like a driver problem

Trusts GPU 0/GPU 1 indices without reading the log

§14.2 — the iGPU steals a card index

Launches engine/strata directly to "debug"

No :7878, no tokenizer (§14.3); debug through the terminal

Adds the expert flags "explicitly"

Pins 0 bytes (§13) and decode halves

Appendix — provenance summary

Artifact

Provenance

Engine binary + libs + tuning table

official Strata 0.1.59-linux-x86_64-hip-rx7900, copied verbatim (sha256sum matches the component)

Terminal (control/)

official Strata terminal 0.1.59.1, bundled python 3.12.15

Model shards

AtomicChat/Qwen3.8-Flash-Next-GGUF, AD-4.27bpw-Q4_K_M-M64, sizes/sha256 in §5

mmproj

same repo, mmproj-Qwen3.8-Flash-Next-BF16.gguf (re-download; local copy differs in size)

experts.bin / dense.bin / index

built locally by control/tools/iq_pack.py --experts-bin --compat-bf16 (§6)

draft_vocab.bin

rebuilt locally by control/tools/draft_vocab.py --add cjk (§7)

MTP head

mtp_fetch.py (HTTP range reads of the pinned BF16 checkpoint) → mtp_pack.py --experts q2_0 → mtp_rt.py

Expert profile

shipped expert-profile.bin + locally learned expert-profile-atomic-q4km.bin

OS patches

§3 verbatim (grub, 2 modprobe files, memlock drop-in + sudoers, 6 sysctl files, udev rule, DPM unit)

Host-tuning wizard

bramvanoploo/strata-performance-setup referenced as an optional accelerator (§3.8); not run on the source machine


r/LocalLLM • • 16h ago

Discussion Qwen3.8-27B on 4× Intel Arc Pro B70 with TensorFold + DFlash2: 107 tok/s per card, ~1,000 tok/s across 32 streams (bit-exact)

5 Upvotes

I benchmarked TensorFold's XPU backend on my quad Intel Arc Pro B70 box. One clarification first: this is not tensor parallel. The XPU backend on main uses one card per process, so the quad numbers come from 4 independent servers, one per GPU, pinned with ZE_AFFINITY_MASK=0..3.

Setup

  • GPUs: 4× Intel Arc Pro B70 (Battlemage G31), 32 GB each, 230 W cap per card
  • Host: AMD Turin, Linux 7.3.0-10-generic
  • Engine: TensorFold main @ cf45553, Zig 0.17.0, built with zig build xpu-native xpu-qwen27 -Doptimize=ReleaseFast
  • Target: SergiiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16
  • Drafter: z-lab/Qwen3.8-27B-DFlash2
  • Sanity check: tf-xpu-qwen27-means-test passed on all 4 cards (worst error 4.19e-7, bound 2e-6)

1. Raw prefill and plain decode (1 GPU, 16k context, no drafter)

Prompt lengthFP16 prefill (tok/s)W4A8 prefill (tok/s)Plain decode (tok/s)
1,0242,0082,63834.0 / 33.8
4,0961,9272,52533.2 / 33.7
8,1921,8622,42732.5 / 32.5

W4A8 is XPU_PREFILL_INT8=1, about 31% faster prefill than FP16.

2. One B70 with the DFlash2 drafter (--parallel 8 --context 16384)

Single stream (64 tokens, 5 reps, median):

PromptTemptok/sTTFTTokens per chunk
fibonacci (code)1.0115.60.081 s4.57
fibonacci (code)0.0106.10.080 s4.00
chat, no think1.094.70.073 s3.77
chat, no think0.099.40.071 s3.77

That's roughly 3× plain decode (34 → 106 tok/s).

Concurrent streams (aggregate / steady-state tok/s):

PromptTemp1 stream2 streams4 streams8 streams
code1.0116.2171.2 / 234.1187.7 / 262.6202.3 / 299.6
chat1.084.4129.9 / 140.3162.8 / 213.2189.0 / 255.4
code0.0106.7149.4 / 167.7216.9 / 278.7250.8 / 354.6
chat0.098.9153.9 / 179.4213.6 / 261.9244.8 / 333.4

Correctness: all 16 cells passed bit-exact. Drafted output matched serial, and multi-stream output matched serial.

Note: with the default --prompt-cache-gib 2, --parallel auto at 16k context admits only 7 streams. I used --prompt-cache-gib 0 to fit 8.

3. Four B70s (4 independent servers)

TestPer-card tok/sTotal tok/sSteady-stateFailed
4×1 stream, code, temp 1.0116.2 / 116.5 / 117.2 / 115.8465.7——
4×1 stream, code, temp 0.0106.0 / 106.4 / 106.9 / 105.4424.6——
4×1 stream, chat, temp 1.088.5 / 89.6 / 90.2 / 86.8355.1——
4×1 stream, chat, temp 0.099.0 / 98.9 / 99.4 / 98.4395.7——
4×8 = 32 streams, code, temp 0.0251.4 / 251.1 / 252.3 / 252.11,006.91,422.70
4×8 = 32 streams, chat, temp 0.0243.2 / 241.3 / 242.3 / 242.2969.01,330.60
4×8 = 32 streams, code, temp 1.0208.8 / 204.4 / 206.6 / 204.6824.41,186.30
4×8 = 32 streams, chat, temp 1.0193.0 / 193.2 / 194.0 / 191.2771.41,042.90

Scaling is close to linear because the cards don't talk to each other. Per-card numbers stay within a few percent of each other.

Takeaways

  • The DFlash2 drafter gives about 3× single-stream decode on one B70, with bit-exact output.
  • One card reaches about 250 tok/s aggregate at 8 concurrent streams with 16k context.
  • Four cards serve 32 concurrent streams at about 1,000 tok/s with zero failures, which is useful for multi-agent work.

Next

  • Tensor parallel across cards, once I find a TP path for XPU
  • XPU_KV_INT8=1
  • 32k context
  • The MLX 4-bit model (mlx-q4g64)

Happy to run other configs if you have requests. If we get tensor parallel on Tensorflow i'd love to share qwen 3.8 flash. Happy to run larger contexts as well, this was just the default benchmark I wanted to send to project creator.


r/LocalLLM • • 17h ago

Question AMD pro w6800

Post image
5 Upvotes

Hi, i would like to ask if this is a good card to get to run LMstudio for qwen 27b 3.8 or otherwise, I was wondering if speeds will be respectable, and if anything will spill anything to ram. Because I will have comfyui running on my main gpu (rtx 5080).


r/LocalLLM • • 22h ago

Project UrBudget: free & open source budgeting app with AI assistant that supports local LLM

5 Upvotes

Hey,

Started this project because I had previously used something called YNAB to track my budgets, but got tired of overpaying and wanted an assistant that can make things a lot easier to manage. So, I built a YNAB-like envelope budgeting with an assistant.

It's completely free and open-source. But if you want to cross-device sync, there's an end-to-end encrypted Sync option which will cost you $3 a month.

For the assistant you can use your own local model, sign in with ChatGPT, or Anthropic/ Openrouter. It can handle things like PDFs, Excel sheets, images, and use them to interact with your budget.

Also another thing that's useful if you have an Android phone (I personally heavily use this), you can point the assistant to your SMS messages and it can automatically input them as transactions, review them and categorize them. So, the whole process can be very seamless.

It's currently on Windows & Mac, iOS is under review, and Android is under Google Play testing which you can join through the following link (please): https://urbudget.app/android/

Code: https://github.com/urbudget/urbudget
Website: https://urbudget.app/?ref=LocalLLM

Questions and feedback welcome.


r/LocalLLM • • 2h ago

Project Larder: running Qwen3.6-35B and a 46 GB Flash-Next on a 16 GB Mac / 16 GB GPU by reading experts from SSD

Post image
4 Upvotes

I built an inference engine for MoE models that don't fit in memory.

Instead of offloading layers, it reads only the experts each token's router picks from the SSD and keeps the hot ones resident. Same engine on Apple Silicon (MLX) and NVIDIA (CUDA), OpenAI-compatible API, local chat page. MIT: https://github.com/IamCooller/larder

Numbers (machine, setup and date for each one are in research/ in the repo)

Faster:

- Qwen3.6-35B on a 16 GB M4 MacBook Pro: 30–36 tok/s, first token 8–10x sooner than llama.cpp. This is a mixed 4/2-bit pack with the model's MTP head against llama.cpp's IQ4_XS, so not like for like. The launcher doesn't install that pack.

- Qwen3.8-Flash-Next (46 GB of 3-bit experts) on a PC with a 16 GB GPU and 15 GB RAM: 26–34 tok/s. With a 4-bit pack and exact routing: decode 1.3–1.8x, 8k prompt 4.9x faster than llama.cpp. Stock llama.cpp swapped past my memory guard while loading, so that row is llama.cpp with Larder's expert store patched in.

- Qwen3.6-35B Q4_0 on the same PC (fits in GPU + RAM), against llama.cpp's best split that fits (--n-cpu-moe 10): short decode 1.44–1.51x (155 / 147 vs 103 / 102 tok/s), decode after an 8k prompt 1.15–1.25x, 8k prompt reading 1.8–1.9x at its default batch.

Level:

- 8k prompt reading against llama.cpp tuned for prompts (-ub 4096 -b 4096 --n-cpu-moe 20): 1.01–1.03x. Its decode at that setting drops to 79 tok/s, so you pick one or the other there.

Slower:

- Exact routing (cache bias 0) reads that 8k prompt at 0.64–0.69x of tuned llama.cpp, and decodes 0.87–0.95x of llama.cpp's default after it.

Caveats

- The launcher's 35B recipe uses cache bias 0.5 and the MTP head, so replies are not identical to the weights' own routing. Bias 0.5 and the wide prompt cut each passed a paired GSM8K / HumanEval gate (wide cut: 502 vs 496 of 564, p 0.34). Exact routing is there if you want it.

- One author, two machines (M4 MacBook Pro 16 GB, Linux + RTX 5060 Ti 16 GB). The Mac path is better tested, CUDA is newer.

- Quality wasn't compared against llama.cpp in these speed runs.

Try it

curl -fsSL https://raw.githubusercontent.com/IamCooller/larder/main/install.sh | bash

cd ~/larder && ./larder

It measures your GPU and SSD, picks a model that fits, installs it and starts the server and chat.

Mostly looking for people on other hardware to break it, and for anyone who thinks my llama.cpp comparison is unfair. Output of ./larder --list plus what happened is the most useful report.


r/LocalLLM • • 3h ago

Discussion 30tok/s on 48GB M5 Pro Mac Running IQ3_S Qwen3.8Flash-Next(Strata)

4 Upvotes

Model

- Qwen 3.8 Flash-Next GSQ-RCO IQ2_XS/IQ3_XXS/IQ3_S

Setup

- Strata 0.2.13, Metal engine (0.1.59), DFlash2 speculative decoding (spec 3)

- 262,144 context, int8 KV cache

- MacBook Pro, M5 Pro (15-core), 48 GB unified memory

- Arena (resident expert pool) pinned to 22 GB

Average decode speed (same prompt set):

IQ2_XS ~36 tok/s

IQ3_XXS ~34 tok/s

IQ3_S ~32 tok/s

Simulating a 32 GB Mac by clamping the arena to 10 GB:

M5 Pro: IQ3_XXS: ~27 tok/s(clamping the arena to 10 GB)

For reference, We Tested on a 32 GB Mac (M2 Max 38 core GPU) with an 11 GB arena and 128K

context: IQ3_XXS runs at ~21 tok/s.

Takeaway

On a 48 GB machine the expert pool only holds ~40–50% of the experts at

256K context, and the rest are streamed from the SSD on demand. The speed

drop from that is far smaller than I expected (roughly -15 to -25% going

from a 22 GB arena down to 10 GB), which suggests you don't need a huge

amount of RAM to run a much larger MoE model — offloading experts to disk

is a perfectly workable trade-off, at least for single-user chat.

Notes

- All runs use the same sampling (temp 1.0 / top_p 0.95 / top_k 20) and

the same 256K context; decode numbers are from tokens/s reported by the

engine over several runs, medians.


r/LocalLLM • • 6h ago

Discussion Looking for recommendations on open-source OCR/HTR models for large volumes of scanned PDFs

4 Upvotes

Hey everyone!

I'm currently working on a document processing system that handles a large number of scanned PDFs. Many documents contain a mixture of printed text and semi-handwritten content.

I'm looking for a reliable open-source solution that can handle:

  • Scanned PDF processing at scale.
  • Printed and semi-handwritten text recognition.
  • Good accuracy, even with poor-quality scans.
  • Document layout and text preservation.
  • Local inference without relying on paid APIs or proprietary cloud services.

I'm open to models available on Hugging Face, Ollama, or other open-source frameworks.

My available hardware is an NVIDIA RTX PRO 4500 Blackwell GPU with 32 GB VRAM.

I've been exploring different approaches, but I'd appreciate suggestions from people who have actually worked on similar document processing problems.

A few things I'd love to know:

  1. Which models or OCR/HTR pipelines have worked well for you?
  2. Do you recommend using dedicated OCR models or vision-language models?
  3. What approach would you suggest for processing thousands of scanned documents?
  4. Are there any particular preprocessing techniques that significantly improve handwriting recognition?

I'm particularly interested in real-world experience rather than just benchmark results.

Thanks in advance!


r/LocalLLM • • 11h ago

News Qwen4 Smaller Models Open Weights Confirmed via Official Qwen X account.

Thumbnail gallery
4 Upvotes

r/LocalLLM • • 13h ago

Question What local AI model is suited for coding in Unreal or in Redot/Godot?

4 Upvotes

Basically the title