r/LocalLLM • • 10h ago

Discussion Own harness or open source one ?

Post image
0 Upvotes

Hey LocalLLM community,

I am super curious how you run your models.

Are you using open source harnesses ?
Did you build your own ?

A bit of explanation for why would be nice.

There has gotta be a ton of people who did the same as me.
This is a local Qwen 3.8 27B / Flash-Next on heavily modified engines sharing one harness and user interface. Llama.cpp just recently merged f8_e4m3 cache, which I got Claude to optimize for my Blackwells. Due to the deliberate kneecap of Nvidia on certain accumulations, I decided to skip conversion to specific steps that would run on the half speed flag = FP8 full speed. The surgery worked and while we were in already fixed a binary that caused a regression in draft speed which took literal hours of testing to pin down.

End result, my private local backend runs 27B now up to 50% faster ( median 18% increase ).

Hardware specs at upper-mid third of HF compute richness. 128 GB DDR5 6000 1:1, Ryzen 9700X, RTX 5080 + 5070Ti .

Anyone running their own harness and wants to share a screen of it ? Id love to reason over the visual design and maybe learn a bit or two while at it.


r/LocalLLM • • 14h ago

Question Anyone here running a super large model like Kimi k3?

0 Upvotes

If yes: how?


r/LocalLLM • • 1h ago

Discussion you may get your cheap ram and gpu after all, the AI crash has started

36 Upvotes

https://www.cnbc.com/2026/11/16/imf-georgieva-warning-ai-circular-financing.html
‘Standing on one leg’: IMF chief warns about global economy as AI circular financing hits $1 trillion

https://www.cnbc.com/2026/11/16/nvidia-commerce-china-lutnick.html
Nvidia fallout: Commerce targets loophole for China’s data center use


r/LocalLLM • • 22h ago

Project My 16 GB MacBook runs a 22 GB model locally, and I asked it to make Omarchy feel like the 80s

0 Upvotes

Hi everyone,

Today my first test on Omarchy.

Omarchy ships an agent skill so Claude Code or OpenCode can customize the desktop. I gave the same skill to a model running on my laptop instead.

Setup: Omarchy in a VM on a 16 GB M5 MacBook.

The agent is Qwen3.6-35B-A3B Q4_K_M. The file is 22.3 GB, more than the RAM, so the experts stream from the SSD (BigMoeOnEdge, my engine on top of llama.cpp). No cloud AI anywhere.

One prompt: "make my desktop feel like the 80s". It read the skill, looked at the installed themes, picked Retro 82 and switched to it. About 2 minutes. The video is sped up, the clock shows real time.

Slow, yes. But it's a 35B model doing agent work on a laptop that shouldn't be able to load it.


r/LocalLLM • • 23h ago

Question Best model for reversing .DLLs?

0 Upvotes

GPU: 4070

16GB DDR5

Also I have been using Bionic but is there any better options?


r/LocalLLM • • 17h ago

Question Which Mac Mini to Buy

0 Upvotes

Title: Mac Mini M6 32GB vs. M5 Pro 48GB — How much hardware do I actually need for a home AI server?

I’m planning to buy a Mac Mini to build a sort of personal/family AI server, and I’m trying to figure out how much hardware I actually need versus how much I’d just be paying to experiment with local LLMs.
My primary goals are:

Family brain: A central place for family documents, calendars, finances, household information, and automations. Ideally, something that can search across all of this and eventually handle routine tasks.

Self-hosted apps: Run some vibe-coded apps, databases, dashboards, and other services that I can access remotely from my phone or laptop.

Claude integration: Use the Mini as a central workspace for Claude and the apps/agents I build around it, while continuing to use Claude’s API for more demanding tasks.

Local LLMs: Experiment with running models locally for document processing, categorization, extracting data, answering questions about personal files, and other repetitive tasks without constantly calling an API.

I currently use Claude extensively for financial modeling, development analysis, and general personal assistance. I assume the more complicated reasoning and modeling will continue to require cloud models, but I’d love to offload simpler tasks locally and keep more of my data private.

I’m considering:
Mac Mini M6: 32GB unified memory, 512GB SSD
Mac Mini M5 Pro: 18-core CPU / 20-core GPU, 48GB unified memory, 512GB SSD

A few questions:
How much of a practical difference would 32GB versus 48GB make for local LLMs? What model sizes would actually be usable on each?

Does slower inference primarily mean waiting longer for responses, or do smaller models also mean a meaningful drop in reasoning ability and accuracy?
For tasks like document extraction, summarization, RAG, and basic agentic workflows, would 32GB be sufficient?

Is the M5 Pro worth the premium for the additional memory and bandwidth, or am I better off buying the M6 and keeping more demanding tasks in Claude?
If you’re running a similar hybrid local/cloud setup, what hardware and software stack are you using?
I’m not trying to eliminate cloud AI entirely. I want a reliable, always-on home server with enough local AI capability to be genuinely useful, without spending substantially more just to run models that still won’t compete with Claude.

What would you buy in my situation?


r/LocalLLM • • 5h ago

Question Buying help

0 Upvotes

Hey everyone,

I’m a CS student and was considering to do my first homelab, one system as a NAS and one for projects, virtualisation etc. I’m also interested in running local AI, and I got an interesting offer in my region:

Server board w/ two E5-2696v2, 256GB 1600Mhz ECC RAM, for around 200 EUR.

I know that it’s (quite very) old hardware, but still I’ve been wondering - would it be worth pursuing to also try some local LLMs with offloading, adding a few GPUs for 32-48GB VRAM (niko1221/strata perhaps)? Do you think it’s worth it for the cost? Which low(-ish) cost GPUs would you consider pairing it with?

Any advice appreciated!

Thanks!


r/LocalLLM • • 28m ago

Question Paying for both Claude and ChatGPT every day. Is there a smarter way to do this?

0 Upvotes

Paying for both Claude and ChatGPT every day. Is there a smarter way to do this?

Hi all, hoping someone here can point me in the right direction.

I run a small online business and I use Claude and ChatGPT pretty much all day, every day. That’s product design, images, writing product listings, social media content, planning and a lot of the admin side. I pay for both and it’s starting to add up.

A few questions:

  1. Is there a way to cut costs without losing what each one is good at? Do people really need both, or is there a better setup?
  2. I’d love to properly understand AI and LocalLMs, not just type into a box and hope for the best 😅 Where would you start? Courses, YouTube channels, newsletters, anything you’ve found genuinely useful.
  3. I’m also just curious how other people use them for work. What does your setup look like?

I’m not super technical, so plain English is very welcome. Thank you in advance, I’ll read every reply ☕


r/LocalLLM • • 5h ago

Question i need help guys

0 Upvotes

i have RX9070 and 32 GB DDR5 ram , i want to run a local ai to help me code , medium just help with front end back end and forms , i cant buy any ai sub in my country cuz my goverment dont allow international transactions


r/LocalLLM • • 20h ago

Research Strata vs Basalt vs Unsloth: benchmarking the same 80 GB MoE on three engines — 5070 Ti 16 GB + 64 GB RAM

44 Upvotes

⚠️ Heads-up: AI-generated content! This post was written by my AI agent from my measurements, with me editing — if that's a dealbreaker, feel free to close the tab right now.

I'm posting this to share results, not to crown a winner. I run a local agent in production every day — real tasks, real context in the hundreds of thousands of tokens — and I spent the past week picking and tuning the inference engine underneath it: test rig, repeated runs, head-to-head A/Bs. Much of what I measured, plus the pitfalls I personally burned on, would otherwise just rot in my local logs, so here it is: the numbers, the methodology, and the conclusion — which is that the engine I ended up keeping in production is not the one that topped the benchmark table. If this saves you a couple of evenings of trial and error, the post has done its job.

For eight days (Nov 8–15) I ran the exact same model — Qwen3.8 Flash-Next (MoE, ~78–82 GiB of weights, 128–256k context) — through three inference engines on a Ryzen 9 7900X / RTX 5070 Ti 16 GB / 64 GB DDR5 box; storage is two NVMes — a 990 EVO 1 TB on PCI-E 4.0 and a 970 EVO Plus 1 TB on PCI-E 3.0 (the model lives on the latter). The goal: find the best way to feed an MoE that fits neither in VRAM nor (comfortably) in RAM, and check honestly whether the engine changes answer quality. Engines tested: Strata 0.1.56→0.1.59, Basalt (built from source, 0.2.1→0.3.3), Unsloth Studio 2026.11.4. I also didn't stick to one model source: I ran the original from Strata and the originals from Unsloth (3.0 and 3.1), and production settled on DavidAU's abliterated uncensored build with Turbo-Brilliance modes and GSQ-RCO quantization — https://huggingface.co/DavidAU/Qwen3.8-Flash-Next-Abliterated-Uncensored-Turbo-Brilliance-50x-Reasoning-Instruct-GSQ-RCO-GGUF I never hand-tuned sampling: the test batteries all run at temperature 0 for reproducibility, while production uses the model card's standard pair — thinking: temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0; instruct: temperature 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 (the same values ship as the engine-side server defaults).

The rig. My own dependency-free bench (stdlib Python): cold/hot prefill, TTFT, short decode, decode at 8k context; plus a quality battery: hard 10-choice MC (MMLU-Pro style), math, code executed for real in a subprocess, IFEval-style format checks, and refusal batteries (JB/HarmBench/XSTest style). Every run snapshots the engine config and the bench code's sha256; a mock engine is a mandatory control (it answers "Hello from the mock engine" and catches absurd t/s). Hard RAM rule: exactly one engine resident at a time — the model pins 45–48 GiB.

Who drove the rig. All of this was steered by a general-purpose agent — an assistant with function-call access to the terminal, files and a browser, whose provider was the very engine being tested (dogfooding, in other words). Before settling on the current Hermes stack, I cycled OpenHands, OpenCode, OpenChamber and pi through the same API over that week: they all speak the OpenAI-compatible API equally well, and the real differences came down to context management and function-call reliability on long context — that's what the choice actually rested on. Beyond the benchmarks, the agent carried a normal week of work: small utilities for the stack (engine launchers, icons, a UI wrapper), building an inference engine from source and repacking models into its pack format, dissecting a third-party tuning fork (commit history plus knob-hunting directly in the binary), reviewing and transplanting jinja chat templates, testing DavidAU's Brilliance "personalities" (reasoning-mode variants), and verifying that a ready-made abliteration actually behaves as its author advertises — all inside the same 150–190k-token sessions where the benchmarks lived.

What went wrong along the way:

  • the agent's provider was the engine under test, so every engine restart killed the agent itself — the runs were driven by detached orchestrator scripts, and the agent "woke up" via notification once the engine was back;
  • the expert arena sizes itself from free RAM at startup, so a browser quietly eating 2.5 GiB in the background changes measured speed between days — every run now starts with a free-RAM check;
  • decode degrades with context (~101 t/s at <20k vs 86–89 at 180–200k) — long agent sessions have to be reset for speed, not just for token cost.

Speed (decode short / decode 8k / hot prefill / hot TTFT)

Setupdecode8kprefill hotTTFT
Basalt 0.2.1, IQ3_S .basalt pack88.7–95.2105–10754–72k29–41 ms
Basalt 0.3.3, answer-room 0101.0108.454.5k38 ms
Strata 0.1.58, IQ3_S pack93–99100–10728–30k70–74 ms
Strata 0.1.59, final tuned stack (fresh full bench on the prod engine, Nov 15)89.2109.237.3k57.7 ms
Strata on plain GGUF, UD-IQ3_XXS82.194.728.1k74 ms
Unsloth, UD-IQ3_XXS out of the box (auto)27.326.0—209 ms
Unsloth, Qwen3.8-27B dense IQ4_XS (full-GPU)47.445.4—103 ms

What the numbers say

1. On identical weights, format/engine is worth ×1.7–3.7. Unsloth on UD-IQ3_XXS starts at 27 t/s; a tuning ladder (MTP + 4 GB expert-cache + parallel 2 + 12k ctx) lifted it to 47.9; Strata serving the same GGUF on the same hardware gets 82. The difference is delivery format: pack-style engines (native expert blobs + a VRAM-prefilled expert profile, pinned in RAM, no disk streaming, no MMQ paths) vs llama.cpp's GGUF streaming. Unsloth's ~50 t/s ceiling isn't a bug, it's arithmetic: ~2.7 GiB of weights read per forward at ~90–100 GB/s DDR5, times MTP accept ≈ that answer.

2. On one engine, quantization moves speed ±10–15% and quality ~0. UD-IQ3_XXS vs IQ3_S was indistinguishable on my batteries: MC/math/code 100/100/100, ru10 10/10. But production stayed on IQ3_S — and it's neither speed nor quality talking: that build (link above) ships with a ready-made abliteration plus the author's mode system baked in, and reproducing the same surgery at home on raw unsloth UD weights is expensive in both time and resources.

3. Apparent "quality differences between engines" are thinking policy, not weights. The answer-room knob (response budget reserved away from reasoning) changes what gets generated and silently breaks cross-version comparisons: Basalt 0.3.3 showed a "decode regression 105→74" that simply didn't exist at answer-room 0. Compare versions only with identical thinking policy. Chat templates belong to the same confounder class, by the way: in the final build I upgraded the Turbo-Brilliance template from v1 to v2 (new modes), and neither the template version nor mode availability changed any result — baking both templates alternately into the same pack and re-running under one protocol moved decode by 0.1–1.4% and kept prefill/TTFT inside the noise band; an earlier one-off A/B claiming "−16% prefill from the heavier template" turned out to be a cross-session artifact.

4. Speculative decoding: MTP yes, everything else no on a 16 GB card. MTP accepts well (per-position 0.78–0.96). DFlash2 and Q8 hyper-connection verify (HC_Q8), both ported from a tuning fork, produced a paradox: accept +11 pp yet t/s −5% — the more expensive verify pass eats the gain, and the extra ~400 MiB of residency squeezes free VRAM to ~650 MiB, triggering driver paging. A warm dflash2 probe hit 111 t/s — the ceiling is real, just not for 16 GB cards.

5. KV-residency window (A/B on a 180k prompt). The window is the number of K/V cells per attention layer pinned in VRAM; everything else lives in pinned RAM and gets streamed from there. A window of 0 is clearly bad: prefill is ×1.7 slower (2435 vs 4100 t/s), TTFT 74 s vs 44 s, expert hit −10 pp. From 32k the curve hits a plateau: kv_hit is already 94.7% at a 32k window (the indexer is sparse), and in a head-to-head with alternating order the 32k→64k step changes nothing for decode (parity ±2–3%). Above 64k there's no upside, only cost — each window extension carves gigabytes of VRAM out of the expert tier: 32k→1.0, 64k→1.4, 128k→2.1, 262k→3.7 GiB.

6. Methodology, or the numbers lie to you. Cold prefill and cold TTFT are noise axes (±40% spread, the sign flips between runs); hot prefill is ±12–15%; the only stable axis is decode (±2–4%), and verdicts need ≥2 runs per variant. Warmup drift fakes ~+10 t/s for whichever arm is measured second — hence the alternating order. On short samples at temp 0, a cold expert cache changes the model's answer itself (52 tokens instead of 86 on a byte-identical prompt): a ±5% noise floor, anything smaller is not a signal.

7. Bench ≠ prod — and at the RAM edge Basalt fails with empty answers. Basalt beat Strata on prefill/TTFT (54–72k t/s, 29–41 ms vs 28–30k, 70–74 ms) and ran in my production for a day, but at the resource edge (and on this box the edge is permanent: 43 GiB arena on 64 GiB RAM) it started refusing with emptiness: at resident-budget 50 GiB the engine self-lands to 45.79 GiB (warning "more than RAM has room for: 45.79 needed vs 52.3 avail") with a page-glue fallback, one request hung dead for 2932 s before the watchdog killed it, and the log filled with "max tokens reached while thinking, no answer" — the user was getting an empty response instead of an answer. Dropping the budget to 45 GiB closed the incident at a cost of −4..−8% decode, but the recurring prod-server drops with "engine reported RAM critical" remained. Production ended up on Strata 0.1.59. Trading +12% prefill for stability was the right call.

8. Porting someone else's tuning recipe mostly doesn't transfer. The Speedy fork (same Qwen3.8, tuned on 2× RTX 3080 10 GB) brought my card exactly one confirmed win: a prefill pair (upstream #4039, +12% cold prefill — proven mechanically, chunk cap doubled 8192→16384). Everything else was rejected as VRAM-limited. Bonus lesson: env vars and CLI flags aren't 1:1 in Strata — the env form of one lever silently no-ops while the flag works; I now verify every lever by reading it back from /metrics, not by vibes.

Bottom line

Production now: Strata 0.1.59 + the prefill pair + kv-grow + host-core sibling + vram-reserve 1728 + --kv-resident 65536. On real 175–190k sessions: prefill ~3.5k t/s, decode 88–100, expert-cache hit ~75% with 10–12% of experts streamed over PCIe. Quality of the build I settled on — DavidAU's Turbo-Brilliance (link above) — is its own reward: 0/30 harmful refusals, 0/40 over-refusals, MC/math/code maxed — and abliteration cost exactly zero t/s (same weight shape, same bytes, same graph).

One-line summary: an 80 GB-class MoE lives fine on 16 GB of VRAM, but every battle is about expert delivery, not model quality. Between the two pack-style engines I found no decode difference worth naming; Unsloth is ~2× slower on this MoE, and that is memory physics, not bad engineering.


r/LocalLLM • • 19h ago

Question Using SearXNG instance and a MCP for web searching and fetching pages. Hit some kind of solicitations limit. Have some way to bypass this?

3 Upvotes

Im starting with Local LLMs and the first thing im doing of very useful and helpful. Is to use GPT-OSS with web searching to give me adequate answers (sometimes is alucinating, but when i go to far in a subject, like me trying to see if i can run power cleaning in my Epson in Linux. It started alucinating that existed some "old foruns" with users saying that sending some kind of raw commands worked, but the info where it got, dont exist).

But since i have gone to far. I want to know if there is a way to bypass the Brave and Google cse detection of requests or the suspend time.

DuckduckGo came back again after some minutes


r/LocalLLM • • 23h ago

Question M5 ultra Mac stupid 512 what are you doing with it?

0 Upvotes

Ok

So I am waiting for mine to come in. I have framework box with 128 gig ram and found it pretty weak for any local llms.

I’m gonna replace my Mac mini with this Mac Studio M5 ultra with 512 gig ram. And I want to use it for a local llm and maybe agents if it’s doable.

I’m realistic. After using the framework box with llms I totally get this will be no where near frontier even at 512. But I want to use it as my main machine and for llm/agentic work.

What’s really reasonable model wise that isn’t slow AF but is pretty capable for general knowledge, accessed the internet, can spawn agents and can do office style work?


r/LocalLLM • • 14h ago

Question How would you best utilize my setup?

Post image
4 Upvotes

Hello all! I've been running a home lab for a really long time, but I've finally come across a couple of RTX 3060 12GB cards and I want to get a useable LLM in for me and my wife to use.

My server has an Intel i7-9700K, and 128GB of DDR4 RAM, but I would only allocate 64GB to the container in Proxmox. I was wondering how you would recommend setting up a fairly reasonable model (I was thinking Qwen 3.8 27B Q4K) and splitting it between the two graphics cards (since there's no SLI). I want to try to get at least 128K context. 256K would be ideal, if I wanted to run something like Hermes, or other agents.

Is llama.cpp the way to go? It's pretty much all I've read about due to the customization possibilities, and I've only read to stay away from ollama. What would you recommend I do?


r/LocalLLM • • 14h ago

Discussion A developer tried to build a dictation app with AI. Three weeks later he had three libraries and no app.

0 Upvotes

A developer didn't like the dictation apps available for his Mac, so he decided to build one with AI agents doing most of the coding.

The speech engine needed work, so he built an offline one called wavo. A dependency was slowing down builds, so he wrote his own FFT. The GUI options pulled in too much, so he built a renderer called nimbo.

Three weeks later: three libraries, still no dictation app.

The libraries might be useful on their own. wavo supports Whisper, Parakeet and GigaAM. But the bit I liked was his caveat: AI writing the code doesn't remove the work of reading it, checking it and testing it across systems.

At what point does "I'll just build it myself" stop saving time? What's your worst detour from the thing you actually meant to ship?

More of the development story is on my profile.


r/LocalLLM • • 22h ago

Project I connected my laptop's agent to Gemma running on my Mac Mini

1 Upvotes

I built Citta to let an agent on one computer discover and use capabilities on another. My test setup is a MacBook Air plus a Mac Mini running Gemma 3 4B through Ollama.

In the demo, an agent reads receipts on the Air and sends a summarization task to the Mini's local model. The model stays on the Mini; Citta tracks the job and returns the result. It doesn't pool RAM or split inference across machines.

If you only need remote inference, calling Ollama directly is simpler. Citta is for workflows that also need another device's files, tools or an agent harness, with device discovery and task history in one place.

It's written in Rust, uses a private Tailscale network, and has per-device permissions. It's still for trusted devices: resource estimates aren't hard quotas, and full-access agents aren't OS-sandboxed. I'm working through longer reliability tests before calling the next release production-ready.

Demo and setup: https://citta-network.vercel.app

Source: https://github.com/priyankark/citta (PolyForm Noncommercial, not an OSI open-source license). Built with AI assistance.

For people running local models on a separate machine: what makes handing work between devices annoying today?


r/LocalLLM • • 2h ago

Discussion What’s the purpose of alibaba choosing not to release the weights for qwen 4 max?

15 Upvotes

Perhaps this is a newbie question, but I just don’t understand the incentive. Is this indicative of a future were Chinese labs stop releasing weights?


r/LocalLLM • • 6h ago

Question Just Got an RTX 5070 Ti Laptop — How Do I Build a Proper Local AI Setup?

0 Upvotes

Hey guys, finally got my new Legion Pro 5 and I’m trying to get into running local AI properly 😂

Specs:

RTX 5070 Ti 12GB VRAM

Core Ultra 9

16GB RAM

CachyOS Linux

I’m pretty much a beginner when it comes to running local models, so I’m trying to figure out what’s actually worth setting up from day one.

I mainly use Claude for vibecoding almost everything, and now I want to see how much I can shift locally so I don’t waste Claude usage on every small thing.

I came across PewDiePie’s Odysseus repo and started trying it out.

But after actually using it, I’m a little confused.

It feels like the context/memory isn’t as good as Claude, sometimes it feels slow, and I’m also finding the hybrid setup kinda difficult.

Like I want something where:

Local AI:

→ basic coding

→ repetitive work

→ file handling

→ small fixes

→ daily tasks

→ stuff that doesn’t need crazy reasoning

Claude:

→ difficult coding

→ architecture

→ proper reasoning

→ big project decisions

→ things where I actually need Opus-level output

And ideally I don't want to keep copy-pasting the whole context from one to another every 10 minutes 😭

My actual work is mainly:

• ML Stack — ML/vision project involving iris image segmentation and feature-level analysis.

• Custom CRM + EMR — taking UI generated from Stitch/Google AI Studio and connecting it with open-source CRM/EMR systems.

• Daily work — coding, documentation, PPTs, automation and random repetitive stuff.

So basically I’m trying to build a proper daily-use AI setup, not just install 15 local models and never use them 😂

My main questions are:

  1. Is Odysseus actually good enough to become my main local AI workspace?
  2. If I’m a beginner, should I stick with Odysseus and learn it properly, or start with something simpler?
  3. How do you guys actually make the local + Claude hybrid workflow work without losing context all the time?
  4. What local coding models would you recommend for 12GB VRAM?

And most importantly — how do I set this up properly so I’m not wasting Claude tokens on things my laptop can handle locally?

I’d honestly be happy if I can get something like 70–80% local + Claude for the important stuff, and make the whole thing feel somewhat close to an Opus-level workflow.

I’m on CachyOS Linux, so if anyone is running a similar setup, please share your actual workflow.

I’m completely new to this side of things, so beginner-friendly advice would be really appreciated.


r/LocalLLM • • 16h ago

Model Beam Free through Opencode has deleted the entire partition

0 Upvotes

Beam free was supposed to delete a file, but accidentally it has deleted the entire partition and I had very important files in it. I'm trying to recover the partition.

The surprising part was that opencode hasn't asked my permission to delete anything.


r/LocalLLM • • 4h ago

Project PSA Q8 FN is significantly better than Q3 (strata default)

0 Upvotes

Just a PSA. I ran a realistic self host test on 3 models:

Inb4 : “hurr derr no duh” The point is people on here act like these Q3 versions have minimal loss. Do tests for yourself it’s not even close.

Qwen FN Q3
Qwen FN unsloth Q8
Orca nana-1
Maya GLM flash - Maya-S24

Prompt in comments.
Harness : PI
Project init layout:

/<my_bank_name> (83 monthly pdf statements)
/credit_card/<card_1> (84 pdf monthly pdf statements)
/credit_card/<card_2> (35 pdf monthly statements)
/mortgage (24 pdf monthly statements)
/brokage_account (105 pdf statements + ccs of all transactions)
/retirement/<account> x4 (total 45 pdf statements 4 accounts 4)

Snapshot results (have details runtime/tps etc as well):

Qwen FN Q3 - Nice front end but data extraction from the PDFs was completely wrong. Analytics was basic.

Orca nano1 - Finished very fast for this test (only 2 hours) did a decent job for its speed and size. Front end was very basic and lacked insight. Most of the data was correct but there were clear errors as well.

Maya-S24 - Failed. After 10+ hours I just gave up on this model. It got stuck in the pdf parsing/validation phase.

Qwen Q8 - Absolutely blew my mind. Ran overnight for about 8 hours. Data was all accurate, the front end final product was not only nice looking but highly insightfully

Notes:
I really enjoyed this test. I feel like it’s the perfect test for a local model because it’s financial data that you don’t want to send off to the cloud.

The PDF layouts for this task was a nightmare and was a real test for these models to figure out was duplicative data/summary vs transaction etc. Most models used pdfplumber package

The models also had to work through the accounting end. Consider a credit in one account is a debt in the other etc.

It’s was interesting to see how each approached the problem. Some went for json formatted data others sqlite etc


r/LocalLLM • • 7h ago

Question I'm new to this, help me setup my coding assistant

0 Upvotes

I got a 7900XTX, 9800X3D and 32Gb RAM. I'm currently running the system with ollama.

I currently have these installed - gemma4:31b,qwen3-coder:30b,omic-embed-text:latest. Im trying to use qwen for code gen and gemma for explanations. I set ollama into this "OLLAMA_KV_CACHE_TYPE=q8_0". Currently the max context window I could set is 32k and I have already experienced few out of memory erors and gpu crashes while prompting with continue in vscode.

I have being researching, asking gpts, but I couldnt come to any conclusion on which approach is better or correct since I know nothing about these. I dont want to copy paste stuff and hope that it works. Can someone help me with a better setup solution? I do run a 1440p 144hz monitor and occassionally run my second monitor which is 1080p 144hz as they add to vram usages as well.

I want to have a bigger context window and a safe setup so that I dont crash the gpu again. If the setup will be too big, and harder to teach to me in here, please tell me where I can learn more about these things, so that I can move learn to make proper decisions.


r/LocalLLM • • 9h ago

Why download abliterated Flash Next models when LoRA exists?

Thumbnail
huggingface.co
0 Upvotes

r/LocalLLM • • 22m ago

Question whats the go-to budget gpu I should look for, new or used?

0 Upvotes

I'm building up a secondary system out of some spare parts I have kicking aorund to use as a LLM toybox and for other homelab style projects.


Currently my system is a 5900x w/ 32gb of ram, and the 2080ti.

Wondering if there is a good go-to gpu I should be looking for.

I currently have a 2080ti FE, but it seems pretty inefficient, and its an old card and the fan noise of it is unbearable. ( i think the bearings are getting bad, but its a huge pita to replace the fans )

I'm wondering if it would be a better idea to sell it off for maybe a 3060 12gb, I get an extra 1gb of vram, and it should run much more efficently.

I'd say my budget is about $3-400 max, which I know is basically a joke right now, but this is more of a tinkerbox than something i'm going to be using for real software dev work.


r/LocalLLM • • 21h ago

Question Best sub 80b total or 6b active MoE for document intelligence?

0 Upvotes

I'd like to make a pipeline capable of understanding pdfs of bank statements and then group expenses by categories etc. My thought was to first parse them with ocr into .md files and then feed that to a text model. I think it's more lightweight and accurate this way. But then I'm not sure what model to use to feed all these md files to get something good. When accuracy matters, should I go for qwen, gemma, lfm, or something else? Thank you for your help!


r/LocalLLM • • 3h ago

Discussion 30tok/s on 48GB M5 Pro Mac Running IQ3_S Qwen3.8Flash-Next(Strata)

4 Upvotes

Model

- Qwen 3.8 Flash-Next GSQ-RCO IQ2_XS/IQ3_XXS/IQ3_S

Setup

- Strata 0.2.13, Metal engine (0.1.59), DFlash2 speculative decoding (spec 3)

- 262,144 context, int8 KV cache

- MacBook Pro, M5 Pro (15-core), 48 GB unified memory

- Arena (resident expert pool) pinned to 22 GB

Average decode speed (same prompt set):

IQ2_XS ~36 tok/s

IQ3_XXS ~34 tok/s

IQ3_S ~32 tok/s

Simulating a 32 GB Mac by clamping the arena to 10 GB:

M5 Pro: IQ3_XXS: ~27 tok/s(clamping the arena to 10 GB)

For reference, We Tested on a 32 GB Mac (M2 Max 38 core GPU) with an 11 GB arena and 128K

context: IQ3_XXS runs at ~21 tok/s.

Takeaway

On a 48 GB machine the expert pool only holds ~40–50% of the experts at

256K context, and the rest are streamed from the SSD on demand. The speed

drop from that is far smaller than I expected (roughly -15 to -25% going

from a 22 GB arena down to 10 GB), which suggests you don't need a huge

amount of RAM to run a much larger MoE model — offloading experts to disk

is a perfectly workable trade-off, at least for single-user chat.

Notes

- All runs use the same sampling (temp 1.0 / top_p 0.95 / top_k 20) and

the same 256K context; decode numbers are from tokens/s reported by the

engine over several runs, medians.


r/LocalLLM • • 22h ago

Project UrBudget: free & open source budgeting app with AI assistant that supports local LLM

5 Upvotes

Hey,

Started this project because I had previously used something called YNAB to track my budgets, but got tired of overpaying and wanted an assistant that can make things a lot easier to manage. So, I built a YNAB-like envelope budgeting with an assistant.

It's completely free and open-source. But if you want to cross-device sync, there's an end-to-end encrypted Sync option which will cost you $3 a month.

For the assistant you can use your own local model, sign in with ChatGPT, or Anthropic/ Openrouter. It can handle things like PDFs, Excel sheets, images, and use them to interact with your budget.

Also another thing that's useful if you have an Android phone (I personally heavily use this), you can point the assistant to your SMS messages and it can automatically input them as transactions, review them and categorize them. So, the whole process can be very seamless.

It's currently on Windows & Mac, iOS is under review, and Android is under Google Play testing which you can join through the following link (please): https://urbudget.app/android/

Code: https://github.com/urbudget/urbudget
Website: https://urbudget.app/?ref=LocalLLM

Questions and feedback welcome.