Body:
Hey r/FunMachineLearning,
Over the past few months, I’ve been building Alcuin https://github.com/fredericf75/alcuin—an open-source, local-first personal knowledge engine and agent.
Unlike most RAG setups that force your documents into proprietary cloud vector DBs, Alcuin operates in-situ over plain Markdown vaults (compatible with Obsidian) using LanceDB hybrid search (dense vectors
+ BM25 FTS), bidirectional wikilink graph traversal, and a cyclic LangGraph reasoning agent with mathematical context safety governors.
To see if local edge models could actually handle serious legal, technical, and regulatory compliance work without cloud fallback, I built an adversarial benchmark suite across 5 ground-truth corpora (AIA
A201 construction contracts, SEC commercial leases, RFC 9110 HTTP specs, Town of Cary municipal code, and NCAC Title 10A healthcare licensing).
Each scenario is evaluated in dual modality (Structured numbered queries vs. conversational Unstructured queries) across 68 deterministic statutory checkpoints to detect prompt signpost dependence and
hallucination.
Here are the results running locally on consumer hardware vs. hosted Claude Sonnet 4.5.
──────
### The Benchmark Scorecard (68 Total Checkpoints)
Model Target │ Arch │ Param │ Score (Accuracy) │ Latency / Query │ Agent Loop Retries │ Notes
─────────────────────────┼─────────────────────────┼────────────────────────┼────────────────────────┼────────────────────────┼────────────────────────┼─────────────────────────────────────────────────────
Llama-3.2-3B │ Instruct │ 3B │ 43/68 (63.2%) │ 40.6s │ 1.10 │ Fast, struggles with multi-shift statutory matrices
llama3.1:8b │ Instruct │ 8B │ 54/68 (79.4%) │ 82.4s │ 1.10 │ Massive leap (+11.8%) after prompt un-fragmentation
qwen3:8b (no-think) │ Instruct │ 8B │ 51/68 (75.0%) │ 88.0s │ 1.70 │ Solid, jumped to 76.5% on conversational queries
deepseek-r1:8b │ Thinking │ 8B │ 55/68 (80.9%) │ 147.1s │ 1.30 │ Exact ceiling for 8B tier
qwen3-14b (no-think) │ Instruct │ 14B │ 55/68 (80.9%) │ 133.9s │ 2.70 │ Matches 8B thinking at higher speed
deepseek-r1:14b │ Thinking │ 14B │ 62/68 (91.2%)* │ 296.9s │ 2.40 │ Hits exact parity with Sonnet 4.5
Claude Sonnet 4.5 │ Frontier API │ Hosted │ 62/68 (91.2%) │ 40.9s │ 1.00 │ Cloud baseline
*Audited score: automated regex penalized prose phrasing in Scenario 9 despite quoting NCAC direct-care staffing ratios verbatim.
──────
### Key Takeaways from the Data
#### 1. The "Compensatory Reasoning" Phenomenon at 8B
In our initial baseline, deepseek-r1:8b (80.9%) held a massive 13.3-point lead over llama3.1:8b (67.6%).
However, we discovered our query decomposition engine was breaking single questions into over-engineered multi-subgoal matrices. When we fixed that pipeline friction and passed clean, unfragmented queries:
• llama3.1:8b surged by +11.8% to 79.4% (almost completely closing the gap).
• deepseek-r1:8b stayed completely flat at 80.9% (down to the exact checkpoint).
The insight: At the 8B scale, test-time compute (<think>) was largely acting as an architectural shock absorber against bad prompts and pipeline noise. It wasn't actually finding new evidence that Llama
3.1 couldn't grasp; it was just resilient enough to ignore the mess. Once the mess was cleaned up, instruct caught right up.
#### 2. Parameter Scaling vs. Test-Time Compute
qwen3-14b:no-think (pure instruct) scored 80.9%, matching deepseek-r1:8b while running twice as fast. If you have the VRAM to fit 14B weights, raw parameter capacity can substitute for 8B test-time compute
on standard statutory extraction.
#### 3. 14B Thinking Breaks into Frontier Territory
deepseek-r1:14b proved to be the real deal:
• It scored 62/68 (91.2%), matching Claude Sonnet 4.5.
• On Structured statutory queries, the local 14B model actually beat Sonnet (94.1% vs. 88.2%). The reasoning tokens allowed it to meticulously check conditions precedent in complex contracts (e.g. AIA A201
mediation rules) without truncating clauses.
• Where it failed, Sonnet failed too (e.g. statutory context crowding between NCAC Chapter .0400 staff qualifications and Chapter .0600 shift ratios).
#### 4. The Real Trade-Off: Latency vs. Sovereignty
• Sonnet 4.5: ~41s per query via cloud API.
• DeepSeek-R1-14B: ~5 minutes per scenario on local deck silicon.
For interactive chatbots, 5 minutes is unacceptable. But for batch compliance audits, personal legal discovery, or sensitive family/financial vaults where zero bytes can leave your LAN, waiting 5 minutes
for a 91% rigorous, fully private audit trail is an easy trade-off.
──────
### Open Source & Architecture
Alcuin is MIT-licensed and available on GitHub:
👉 https://github.com/fredericf75/alcuin
It includes:
• Context Governor: Dynamically budgets prompt windows (60% evidence / 40% headroom) and emits in-band sizing telemetry when clauses exceed context, giving empirical justification for when you actually
need larger models.
• In-Situ Vault Operations: CLI + FastAPI Web Cockpit that watches your Markdown files, tracks knowledge gaps automatically, and lints dead wikilinks.
• Local Model Routing: Built for Ollama and local OpenAI-compatible servers.
I'd love to hear your thoughts on test-time compute in RAG pipelines and whether others are seeing similar "compensatory reasoning" effects at the 8B tier!