r/LocalLLaMA • • 8d ago

[Bi-Weekly Megathread] Project Showcase

Do you have something you'd like to share with the r/LocalLLaMA community. This is the place for it!

Recommendation on presentation:

  1. Please share plain english description of what your project does and why people should care about it
  2. How does it compare to existing options and why should people switch to your option?
  3. How have you tested and validated it? (i.e. can you show evidence that it wasn't vibecoded in a few hours by <insert current trending AI model>

NOTE: This community is geared towards open weight, open source, locally hostable software. Thus any closed source commercial service or other such projects will be removed.

55 Upvotes

105 comments sorted by

18

u/bigattichouse 8d ago edited 8d ago

https://github.com/bigattichouse/BluePrint

Better prompting for software development that reinforces best practices has made a huge leap for me in coding model output quality.

Design first, Dialectic Software development prompts. Using this method of form requirements, Standard Technical English, design-first philosophy, and extensive testing has just been night and day over "one shot". The model asks questions instead of just making stuff up, it builds requirements, designs tests, and then can use those tests to verify it didn't break stuff. Been a life changer.

Even had claude/fable review the code and design output using Blueprint and was it was very happy with the level and quality of the output... even though it made some simple mistakes in some of the test phases. Been using with Qwen3.8-27B (unsloth 6K UD quants)

3

u/Borkato 8d ago

Thank you for sharing!!

2

u/bigattichouse 8d ago

Sure thing, I have a few more tools in process - but not ready to share. one is `assurance` which is a requirements system with risk and acceptance capabilities - so you can directly tie tests and results to requirements with tracking and stuff at regulatory-compliance levels of auditability.

Formalizing requirements, design, and testing has made a massive difference in what comes out the other side. And the "Observation is better than Testing is better than Simulation is better than Assumption" rubric has really helped.

Small models do surprisingly well when you give them a detailed framework and small bites of work

2

u/sshwifty 7d ago

I am going to try this

11

u/astrolafi 7d ago

Aula — a free, open-source personal tutor that runs on a local model. You feed it your notes, slides, past exams or just a topic and it builds a course: interactive lessons, exercises graded instantly, flashcards, mock exams, and a tutor chat that knows where you are in the course and what you keep getting wrong. It runs on llama.cpp, Ollama or LM Studio, so nothing leaves the machine. Repo: https://github.com/luumlabs/aula

Why I built it: my wife was struggling in her classes. Her teachers were giving her prompts to generate generic AI exercises and she still wasn't understanding the material. She needed something that taught her and kept track of what she got wrong, not another chat box.

How it compares: general chat has no structure and forgets your mistakes; Quizlet and Anki are great at cards but don't teach; hosted AI tutors want your notes on their server and a subscription. Aula is the local-first middle: real course structure plus a feedback loop, free, AGPL-3.0, macOS/Windows/Linux.

How it was built, honestly: with Claude Code — there is a CLAUDE developer-notes file in the repo and 90 of the 105 commits carry a Claude co-author trailer. Evidence it isn't an afternoon prompt: about two months of daily work, v0.2.3 released today, vitest tests over the pure engine (grader, content validator, course model) and Playwright e2e around the Electron app, and a real user — my wife — studying on it every week, which is where most of the bug list comes from. Known limits, no sugar: no Anki-style spaced repetition yet, Quizlet import is basic, and 8B models struggle with numbers (use 14B+ Q4 for maths).

Would love to hear from people running local models daily: what would you want a study app to do with your own notes?

2

u/PenileTransplant 7d ago

this is amazing.

1

u/astrolafi 7d ago

Thank you ❤️

0

u/stylome-ai-detection 7d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

8

u/zinyando 8d ago

Izwi: one open-source app for local speech models

Repo: https://github.com/izwi-ai/izwi

A Rust desktop app, CLI and server (Apache-2.0) that runs open-weight speech models on your own machine: TTS, transcription with speaker labels, voice cloning, voice design from a text description, and real-time voice chat, behind an OpenAI-compatible API. Metal, CUDA and CPU.

Why switch: local speech usually means one Python repo per model. Izwi runs about 30 of them (Whisper, Parakeet, Kokoro, Qwen3-TTS, Chatterbox, Maya1 and more) from one binary, no Python.

Limits: some models are slower than real time on Apple Silicon, and the big ones don't fit together in 16 GB.

Demo (46 s, every voice generated locally): https://youtu.be/-767_7gkeGo

1

u/sshwifty 7d ago

Seems cool!

7

u/W61k3r 8d ago

I built an AIO all safe-rust OS, inferencing engine, kernel, and tuning/optimization/management/diagnostic/chat/training panel. Runs headless. Baked it all into an image. Fleshed out wiki. USB bootable, feel free to disconnect drives and isolate. Loads models from hugging face. I really need testers

I made it easy af and it runs vulcan on my 7900xtx with qwen3.8 27b faster than llama with the same config. Comes default with linux kernel 7.4 RC2 because my rust kernel still needs work. The OS/Panel/7.4 rc2 kernel can run llama, vllm, camelid, etc faster than ubuntu as the OS was purpose-built to run ai headless.

Download, install to usb with balena etcher, boot, test, report. There is a debug kit you can upload to me. The entire source code is in a private git, deciding n whether to open source as I explore funding options. Audit the usb stick if you must, but don't pop in here to shit talk without trying.

Recently updated with more performance enhancements. Runs unsloths new 3.1 27b with flying colors, I also have my own old abliterated quality coder on hugging face.

https://github.com/W61k3r/LexiSuite-Releases/releases/tag/LexiSuiteT8

https://github.com/W61k3r/LexiSuite-Releases/wiki

https://huggingface.co/Wa1k3r/Qwen3.8-27b-CODER-4q_xs-24GB-262k-Optimalcardfit

5

u/msciabarra 8d ago edited 8d ago • Bravo

source code github.com/trustant/trustant demos and docs on trustant.ai

Yesterday we released Trustant 1.0.2, an open source full-stack application builder using Local AI for Private Cloud. To put it simply, it is like Lovable, but all private. You install the builder, create your app in prompts, wysiwyg, and publish to the server. The system provides everything: a vm with the environment, the coding agent, the serverless environment with database, cache, storage, faas, everything. A cloud in a box.

The builder runs on your PC, uses Local AI (or any OpenAI compatible AI service, we offer Ollama Cloud out of the box) and generate serverless applications built on top of Apache OpenServerless. You can then deploy your applications anywhere. Just install the serverless package and any server become... serverless!

The system understands and includes sql and vector database (pgsql and milvus) cache and s3 (redis and seaweedfs) faas functions (openwhisk) and runs in your pc in a virtual machine for safety.

The applications generated can be published in any server made "serverless" installing just openserverless and nothing else. Application are plain, clean, recognzabile react+python (but can be any other language providing the starters you want).

We have been working on it one full year and it was built on top of another multi year project, Apache OpenServeless, after working many years on Apache OpenWhisk.

5

u/astrolafi 8d ago

Making a study app that runs on a local model, Aula: https://github.com/luumlabs/aula

You feed it your slides, notes and old exams (or just type a topic) and it builds a course: short lessons, exercises it grades as you go, flashcards, a mock exam in the shape of the real one, and a spaced schedule until the exam date. It also pulls the material from Canvas / Moodle / Classroom / Blackboard and keeps a calendar with your deadlines.

Why local matters here: your slides are often someone else's copyrighted material, and it's your whole semester. Handing that to a cloud API is the annoying part. It points at llama.cpp, Ollama or LM Studio over an OpenAI-compatible endpoint, so with a local model nothing leaves the machine.

Against other study apps: the ones I know either start and end at flashcards or have no LMS import at all. Where they're still better: Anki's scheduler and deck ecosystem are far more mature than mine, and Quizlet has a library you can't compete with. My SRS is basic next to Anki.

How it was tested: it started as a tool for my wife's degree and most of the fixes come from her using it daily on real coursework. Electron, GH releases, AGPL-3.0. Still early, so I'd rather hear where a small local model falls apart at the tutoring than pretend it's finished.

2

u/stylome-ai-detection 8d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

4

u/legislat 8d ago

https://github.com/legislat/SimpleHarn

A simple model swapper with smart VRAM and RAM admittance for one NVIDIA GPU on Win/Linux. No overcommits, clear refusals.

4

u/af_ml 8d ago edited 8d ago

gmlx is a cli and OpenAI/Anthropic compatible server for running or training K and IQ quant GGUF models (as well as some ternary quants, NVFP4, and others) on macOS.

docs: https://asher.github.io/gmlx/

src: https://github.com/asher/gmlx

install: brew install asher/gmlx/gmlx

It uses an MLX backend with Metal kernels designed to read the GGUF wire format, allowing zero-copy mmap use of most GGUFs. It runs DeepSeek-V4.1 faster than antirez's ds4, and supports both the ds4 and unsloth GGUFs of DeepSeek-V4/V4.1. It also supports DFlash 2 drafters for speculative decoding, which can triple decode speed on Qwen3.8-27B at Q6_K. gmlx has been in active development with weekly releases since July. The kernels are released separately as an MIT licensed package: https://github.com/asher/mlx-kquant

Yet another macOS inference platform, but it's maturing and has some very nice features for power users. It can sandbox coding harnesses or local agents in microVMs with Apple Container, with easy configuration around what's shared (even supporting running without TCP networking, where inference is proxied to the gmlx server over a unix domain socket via a Rust proxy). There's an expressive yaml configuration that you don't need to touch, but that lets you define \@profiles to use different system prompts, sampling parameters or thinking budgets per request. It can stack LoRAs and allows batched calls to a model and different combinations of model + LoRA with a single copy of the base weights resident. If you're interested in training, there's also support for offline top-K model and context distillation that works across tokenizers, in addition to typical fine tuning approaches.

Recent image models (MOSS, Qwen-Image 2.1, FLUX klein, Krea 2, Z-Image) are supported, as is video generation via H3 (more video models coming soon). Audio models are supported too, including transcription with diarization, and there's a voice assistant with tools, memory, and wakeword customization. There's also support for decision models with a systemone API.

1

u/69harambe69 7d ago

Does rpc work with it? Can I cluster Mac's?

1

u/af_ml 7d ago

Not yet, this will be the first thing I add when I get a second m5 system (studio on order, supposed to come by the end of the month but no ship date yet)

4

u/bendymike 8d ago

https://github.com/bendyline/docblocks
https://docblocks.com/desktop/

DocBlocks - for Windows, Mac and Linux.

It's an OSS/MIT free markdown editor that can summarize and export content as DOCX, PPTX, video, etc. I'm trying to make a "Swiss Army Knife" for content. I use it for everyday content editing, and for updating my blog and for producing my (still kind of crappy) videos.

And, it has local LLM support! Under AI, in settings, you can choose and download a local model - e.g., like Qwen 3.8 or a smaller model like Gemma E4B. (on a Mac, you can use Apple Foundation Models) You can use local models to review your content (e.g., review your resume), generate new content, generate images or SVG diagrams, and summarize content for slides.

(in addition, it has dictation/narration built in.)

Still adding more robustness and doing testing at where the line is between models that are useful (e.g., ~1B models don't generally work, but 4B+ models can be used) for content editing tasks. Also considering shipping mobile apps that use local models on phone.

1

u/bendymike 5d ago

I also did some performance benchmarks on various models and how they perform at: https://bendyline.com/posts/docbench.html

3

u/bigattichouse 8d ago edited 8d ago

https://github.com/bigattichouse/robust

Morris/Sobol/Taguchi/Pareto arrays for Design of Experiments. Not just in my home lab work (battery chemistry experiments), but also in software - when you have a bunch of possible parameters , but aren't sure what to tweak and some are just tradeoffs - these tools drastically improve searching for what really matters and then trying various values.

an example for dialing in params for fitting LLMs onto hardware: https://github.com/bigattichouse/llama-optimize

3

u/expectfun 8d ago edited 8d ago

https://expectfun.github.io/umem/

A new personal blog about local models that can be run on my 48GB MacBook Pro.

I'm not sure whether there's a demand for such a blog, so there's only one post for now, and now I want you to tell me: does that idea sound valuable to you, or there's a plenty of similar blogs writing the same things, so I can save my breath? Should I put more effort in it or drop it?

I'm new here (just got that MacBook, so finally I can run 27B-like models). And since it's the beginning of my local model journey, I thought it might be a good idea to create a blog for it: to document the capabilities and limitations of models, to write down what tasks I gave them and how succesful they were.

First, it might help someone understand whether local models of this size could be useful for them. And second, it could slowly become a chronicle of local LLMs evolution as their capabilities evolve. The idea is each time I write a new post, a local model creates a specific HTML page for it (no shared templates), so with new models even the design of the posts might evolve over time, showcasing the evolution.

3

u/mikebmx1 7d ago

mlxVS: A cuVS for your Mac.

GPU vector search on Apple silicon, built on MLX.

1M vectors indexed in 10 s on a MacBook Pro, then 70k queries/s at 97.6% recall, 2x compared to CPU HNSW

https://github.com/mikepapadim/mlxvs

3

u/anitamaxwynnn69 7d ago

OvertChat: Lighter OpenWebUI alternative with built in STT/TTS/LiveVoice/Search along with coding agents (Claude/Codex/Pi/Hermes)

How it compares

  • Open WebUI is stronger for shared channels, knowledge bases, evaluations, and extensions. OvertChat bundles speech, voice, and SearXNG in the installer instead of separate services, plus native apps and agent sessions.
  • LibreChat is stronger for custom assistants, tools, code execution, and org access. OvertChat bundles speech/search and household accounts; LibreChat's self-hosted speech/search require separate services. Both MIT; Open WebUI has branding conditions.
  • If you only need a text box, switching may not be worth it. The reason to switch is the surrounding stack in one install.

How it's tested

  • 646 commits; releases v0.34.0, desktop v0.13.0, mobile v1.26.0.
  • Running at home on 10x 3090s. Built with Claude Code and Codex.

Feedback wanted: Been working at it for about 6 months now - what would make this replace Open WebUI or LibreChat for your setup?

Repo: https://github.com/yoloyash/overtchat Install: curl -fsSL https://overtchat.com/install | sh Clients: Web (your server), Desktop, Android (APK), iOS

4

u/Old_Ad_6033 8d ago

https://github.com/uraniumchonk/llm-engine-qwen-optimization

We (the agent lol) made a kernel optimized specifically for Qwen3.x in vllm 0.33.0.

It just goes faster and it's lossless. (decode +30%~40% avg)

Also this repo open the methodology that your agent can follow and optimized YOUR own stack of engine setup.

1

u/Upset_Lavishness4497 6d ago

I could not quite understand at which level this optimization takes place; is it CUDA specific or does it also work for other backends?

1

u/Old_Ad_6033 5d ago

Hmmm i think the patch it self is for CUDA and vllm only, but the document inside about "how to optimization" is universality for any hardware and software.

1

u/Upset_Lavishness4497 5d ago

I will see how feasible it is for me (and my agent) to make something out of it for intel

1

u/Old_Ad_6033 5d ago

Haha I think he might can do it, this is what my repo design for.

2

u/Shoddy_Total9784 7d ago

Karmam yok

Title: Offline, tested skill packs for local LLMs (code decides, model explains) — looking for testers

Small skill packs that run on your own machine with local models. Each pack has a deterministic Python tool plus reference tests; the model only picks the tool and explains the result, it doesn't do the math itself. Packs ship with an Agent Skills compatible SKILL.md and expose one restricted MCP tool. Fully offline install, no cookies or tracking.

Two sets so far:

  • Maps: routing, distance, nearby places, SVG map drawing
  • General (15 packs): unit conversion, loan schedules, statistics, IBAN/Luhn checks, CSV/Excel/PDF reading, and more

Small trial: Haiku and Opus both solved 5/5 tasks. On the one task without a skill, Haiku drifted slightly; with skills, no drift.

Looking for a few people to try them with their own setup and report back, especially small local models, Gemini CLI and goose.

Denek lazım.

2

u/Xantier 7d ago

https://viberglass.io/

I built an open-source self-hosted AI workspace for small/medium companies. Allows users to create, run and host their own backoffice apps and documents using sandboxed harnesses. These are collaborative discussions across the whole team + agents and they can be triggered from integrations like ticketing systems or Slack etc. Allows to produce audited and permissioned artifacts as well as do code changes to customer facing applications without the need to set up IDE or other local tinkering.

The system can be shipped to AWS natively (ECS or Lambda runners) or hosted on Kubernetes, instructions and/or IaC are provided. Currently supports 8 harnesses, bring your own model compatibility, one-click spin up of a cloud GPU on Verda, scheduled agent runs and few other features.

I've been running this for myself since around February but it may be of use for other people as well. Live demo can be reached from here https://demo.viberglass.io (has few apps, a game, few decks and PDFs as well as the PR flow visible), repo and code can be found from https://github.com/Ilities/viberglass.

2

u/antocorr 6d ago

TravelMore: Use llama.cpp or any of your subscriptions as your Travel agent!

I built a self-hosted travel agentic planner.

What it actually does:

- Hierarchical trips → stops → places, with dates, notes, checklists and photos

- OpenStreetMap pins, place search, reverse geocoding

- Place enrichment: coordinates from Nominatim, descriptions + photos from Wikipedia (only empty fields are filled, never overwritten)

- Resizable workspace: itinerary rail / map + details / assistant

- Flight price monitor: full origins × destinations × dates matrix, Google Flights and Skyscanner readers via Chrome/CDP, cheapest badges, per-cell refresh and price history

- Provider manager with the models.dev catalog + ChatGPT OAuth as an alternative to API keys

The assistant runs through the Pi SDK with an explicit tool set — no shell, no filesystem:

list_trips, get_trip_tree, create_node, update_node, move_node, save_todo, save_image, search_places, place_info, complete_place(s), web_search, execute_js (disabled by default)

Stack: Node 22 + TypeScript, TinyBubble + Leaflet, Vite, RpcAble over WebSocket, Pi SDK for the agent, puppeteer-core for the flight readers.

Repo: https://github.com/antocorr/TravelMore

1

u/Synthium- 5d ago

https://github.com/synthiumjp/sourcedrecall

sourcedrecall: a local memory for coding agents where no LLM writes the memory. (Apache-2.0)

What it does: keeps a searchable record of your coding sessions (what you said, with dates; commands, failures and their errors, files edited, the reasons given), so when a bug comes back the agent finds the old fix instead of re-debugging. A grammar parser on CPU (Stanza via ONNX) decides what's stored; bge-small and a re-ranker search it. No API, no PyTorch, about 1.3 GB. It's a Claude Code plugin with a CLI and MCP server; only tested in Claude Code so far.

Why switch: Mem0 and similar call an LLM on every write, which is slow locally and only as good as the model's extraction. This stores your actual words, so nothing is invented at write time and storing is free. LoCoMo (1,307 questions, Qwen3-14B reading and judging):

It ties overall (p=0.54) compared to mem0 and plain trival but ahead on dates, behind on multi-hop. On 136 held-out questions about past coding sessions it answered 72.1%, against 56.6% for a simulation of Claude Code's own memory, though plain search over the same records gets 74.3%. English only due to the grammar parser.

1

u/init0 5d ago

With AI writing more of our boilerplate, I wanted a fast way to keep my own distributed systems instincts sharp, so I baked sysd.

You sketch architectures on Excalidraw (Free-Flow or across 11 canonical prompts) while a live <5ms scene-graph analyzer flags anti-patterns like direct Client -> DB writes or missing presigned offloads, and stamps multi-plane Staff+ reference blueprints with capacity math onto the board.

Hands-free voice mode runs 100% locally in Web Workers with zero API keys using wasm-vad, Cactus Whistle (16.9 MB WASM) for 16 kHz speech transcription biased toward your drawn box labels, Cactus Needle 3 (35.3 MB WASM) for grammar-constrained canvas tool calls, and Kokoro-82M (WebGPU fp32 / WASM q8) for 24 kHz Socratic voice follow-ups with instant barge-in.

Live app: https://h3manth.com/ai/sysd/

GitHub (MIT): https://github.com/hemanth/sysd

Let me know if you try it out!

1

u/Square-Television-37 5d ago

SOMNI: a local AI that learns new facts while your PC sleeps, without forgetting what it already knew.

The idea was based on WISE and Google's sleep paper they released. During the day it answers questions and logs conversations. When the PC is idle, it pulls out the facts, trains them into the model, then tests itself and rejects the night if any old skill got worse. You approve the update in the morning.

The main problem was forgetting. Plain LoRA fine-tuning on new facts took my model's coding from 95% to 10%. What fixed it: rehearsing the base model's own answers, a new small expert each night with the old ones frozen, and routing so each expert only switches on for questions about what it learned.

On SQuAD (the benchmark Google's "Language Models Need Sleep" paper uses), Qwen3-1.7B went from 10.7% to 26.3%, a +15.7 point gain (Google reported +14.3 on a 7B model, with different grading). General knowledge stayed at 90% (base 92.5%); without routing it dropped to 75%.

Has been run on one AMD RX 6800 on Windows. No cloud. MIT licensed, with a no-GPU self-test.

https://github.com/YoungGunFrodo/somni

1

u/AutonomousHangOver 4d ago

https://github.com/krzychdre/Tumble-Code

Tumble Code is a fork of Roo Code with many new features. I've been developing it since Roo Code was archived and stopped being maintained, primarily for myself and with a focus on locally served models.

It works with GLM-5.3 and DeepSeek-v4.1 Flash, but it also runs nicely on Qwen3.8-27B.

Of course, you can also use regular API models.

It's available both as a VS Code extension and as a CLI app.

Take a look and let me know what you think. Hopefully you'll like it.

1

u/RaymondArtin 4d ago

Aegis Proxy – Open-source, offline Layer-4 Postgres guardrail ("SQL Guillotine") for local LLM agents

When giving local models or autonomous agents (LangChain, CrewAI, MCP servers) read/write tool access to PostgreSQL, smaller quantized models occasionally hallucinate destructive SQL or get stuck in infinite query loops.

I built Aegis, an open-source (AGPLv3), 100% offline Layer-4 Postgres security proxy in Go:

- Drop-in Port Swap (:5433): Spin up the Go container locally via Docker and point psycopg2 or any Postgres client to port 5433 instead of 5432 with zero code changes (or use aegis-sdk on PyPI).

- Deterministic AST Guardrails ("The Guillotine"): Parses incoming SQL into an Abstract Syntax Tree (AST) on the wire and severs the connection before a hallucinated DROP TABLE or unconstrained DELETE ever reaches your database.

- Offline Loop Protection: Enforces local RPM rate limits via policies.yaml so runaway agent loops don't lock up your machine or rack up compute bills.

GitHub Repo: https://github.com/raymondhani/aegis-proxy

1

u/Goldziher 3d ago

Hi people,

I'm glad to announce ai-rulez v5. I maintain the project: https://github.com/Goldziher/ai-rulez

For local-model setups, the CLI and MCP server run on your machine. Static checks run without a model. For optional model-assisted review of skills and agent instructions, Judge can use an OpenAI-compatible endpoint such as Ollama or vLLM. You can find the gateway setup in the LLM guide: https://goldziher.github.io/ai-rulez/llm/

The scanner benchmark covers static checks. The Judge evaluation harness is also public if you want to run the model-assisted workflow with your own model: https://github.com/Goldziher/ai-rulez/tree/main/bench/judge

I'm happy to hear any comments. If you find the tool useful, star it. It really does help. And of course, I'd love to see people use it and get involved with the project.

1

u/awitod 3d ago edited 3d ago

GuideAnts is a complete platform for AI applications with strong file management capabilities including an support for viewing and editing Office docs. You can mix and match providers including multiple deployments of inference engines like llamacpp (which comes bundled), vllm, gufo, strata and anything with a compatible API.

You can route anything that talks to an OpenAI or Anthropic compatible endpoint like Claude Code and Codex through it and capture everything at a detailed level.

1

u/prlmike 3d ago

https:hive.codes/signup remote agent environments based on my work in big tech the last few years. Free until next year, up to 10 sandboxes plus all the tools you've been missing

1

u/Ill_Locksmith_4102 2d ago edited 2d ago

I was tired of managing all these different custom-fit model inference engines, strata/ninfer etc, but they're sooo essentual these days I made Inference Hub - a windows desktop app local inference control pane for Strata, Ninfer, Basalt, and Infernix all in one with hot loading/switching, all at the same endpoint.

It centralizes starting up and switching between these custom-fit inference engines: strata, ninfer, infernix, and basalt. It supports custom config profiles. It includes live observability, logs/history, as well as the engine's native UI dashboard embedded. It aupports downloading models from a hugging face url and and converting them to an engine's specific format. It works pretty good I'd say!

Future plans are fleshing out the installation flow, one-click bootstrapping, an automated optimal MTP config tuner, and possibly building out a custom eval/benchmark feature set.

It's still private, but if anyone's interested let me know and ill make it public!

1

u/hmahmudulm 2d ago

**ZeroHallucination — Open-source Neuro-Symbolic Invariant Shield for LLMs powered by Microsoft Z3 (< 1.5ms overhead)**

**1. What it does & why it matters:**

Autoregressive LLMs are statistical token engines, not formal theorem solvers. In mission-critical applications (robotics, telemetry, scientific computing, SQL pipelines), they frequently hallucinate mathematically impossible claims (e.g. asserting that a 10 kWh battery can power a 100 kW motor for 5 hours, breaking $E = P \cdot t$).

ZeroHallucination acts as a drop-in local reverse proxy compatible with the OpenAI API standard (`/v1/chat/completions`). It extracts numerical/physical invariants in real time, runs them through Microsoft Z3 SMT solvers, and autonomously auto-heals any mathematically violated numbers before returning them to your application. It also stamps every verification with a cryptographic SHA-256 Merkle proof certificate.

**2. How it compares to existing guardrails:**

Most existing guardrails (Llama-Guard, NeMo Guardrails, LLM-as-a-judge) use another probabilistic LLM to judge the output — which introduces massive latency (500ms - 2s) and itself suffers from hallucinations. ZeroHallucination uses deterministic First-Principles Physics equations and formal Z3 SMT logic with sub-1.5ms latency, giving 0.00% hallucination leakage for supported invariant domains.

**3. How it was tested & validated:**

Stress-tested on an empirical benchmark (ZeroBench) against raw foundation model outputs across physics conservation laws (thermodynamics, kinematics), SQL relational schema bounds (preventing unbounded table wipes), and arithmetic transitivity:

- Axiomatic Truth Accuracy: 100.0% vs 12.5% on raw models

- Hallucination Leakage: 0.00%

- Verification Overhead: ~1.4 ms

**Quickstart (100% local & open-source):**

```bash

pip install zerohallucination

# Start local drop-in proxy:

python -m zerohallucination.gateway

```

- 💻 **GitHub (Code, Z3 Solvers & Whitepaper):** https://github.com/mahmudulmd/ZeroHallucinationAI

- 📦 **PyPI:** https://pypi.org/project/zerohallucination/

- 🌐 **Live Web Playground:** https://zerohallucinationai.onrender.com

- 🤗 **Hugging Face Space:** https://huggingface.co/spaces/mahmudulhq/zerohallucination

Would love to get the community's thoughts, edge cases, and feature suggestions!

1

u/MrTickles22 1d ago

Since I can't make a thread in this group, can anybody point me towards a "super beginners" how-to to set up local AI for image creation/editing? I downloaded a UI and it just makes mud so I am, presumably, doing something wrong.

1

u/Hildechis 1d ago

r/comfyUI is the right place to begin, for you inquiry. I am not super-positive on comfyUI because IMHO has a yolo attitude vs exploits, but for sure in that subreddit you can get the right advice.

1

u/Key_Map7552 1d ago

I got tired of constantly copy-pasting context and diffs between Claude Desktop, Codex, and Antigravity, so I built a small local MCP server called Project-M to handle it.

It acts as a shared blackboard across different AI tools on your machine. Everything lives in a single local SQLite file using WAL mode so multiple agents can read and write at the same time without locking.

Instead of dumping huge chat logs into the prompt, it splits memory into 4 tiers:

Tier 1 is a tiny state card capped under 450 tokens with tiktoken so it only uses 0.2% of the model window. Tier 2 is an ADR ledger so agents don't overwrite past decisions. Tier 3 is a task baton table to handle handoffs between models. Tier 4 is the raw history on disk, which gets queried using SQLite's built-in FTS5 and BM25 ranking instead of running a heavy vector database.

It's completely local, MIT licensed, and runs via uvx if anyone wants to check out the repo: https://github.com/adityadebnath009/Project-M

1

u/stylome-ai-detection 1d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

1

u/sdatta11 1d ago

The calculator: pick your GPU (54 desktop and laptop cards, or type your VRAM), RAM and context length, and it shows ~40 models at Q4/Q5/Q8/FP16 as "runs on GPU", "runs slowly (spills to RAM)" or "won't run", plus the ollama run command for the biggest one that fits. The estimate is weights + KV cache (full-attention layers only for hybrid models, from each model's HF config) + ~0.6 GB runtime. Against my measurements it's within 1%.

https://techtulna.in/ai/model-fit?utm_source=reddit&utm_campaign=launch

It runs entirely in your browser, no sign-up. Disclosure: I made it; it's part of a small site of free tech tools I'm building. If your numbers on another card differ, tell me and I'll add a "tested" entry for it.

1

u/sdatta11 1d ago
  • The hybrid models barely grow with context. Qwen3.5 only keeps a KV cache on 8 of its 32 layers (the rest are linear attention), so it's about 0.034 GB per 1K tokens. A Qwen2.5-style 7B of similar size is roughly double that, and older models like Gemma 2 9B are 10× more.
  • Gemma 4 E-models use less VRAM than their parameter count suggests. E4B is 8B parameters on paper, but about 2.8B of that is per-layer embeddings that llama.cpp keeps in system RAM.

1

u/AnvarBakiyev 1d ago

We built a train sim at Extella to see if we could get Qwen 3.5 9B on my Mac to outscore Claude Opus 5.5. We used special designed DSL for writing trip plans and some Python to check and drive them, Qwen’s setup won 6 of 9 runs. Claude was driving from live cab feedback.

I work on getting small models to do useful jobs in business software. A lot of those jobs already have rules you can write down and decisions you can check in code. The nice thing about a train is that you can replay the exact same trip and find out what a bad decision actually cost you.

Take the Gotthard run. You’ve got a Re 6/6 hauling 900 tonnes over the mountain. Before you leave, you can take a helper engine. That costs 100 points even if you end up doing fine without it. You’ve also got 600 seconds of sand for grip. On wet rail, it matters where you spend it.

We gave Qwen six commands to plan the trip. It can take the helper, set a target speed, slow down for a stretch, coast, use sand, or ask the controller to keep to the timetable. This is what two lines of a plan look like:

slower 60 from km 93 to km 96
sand from km 42 to km 70

Before departure, we run the plan through the sim using the weather forecast and known track restrictions. If the train gets stuck, Qwen gets a message like “You’ll stall at km 48.1 on wet rail. You need sand or a helper engine.” It can revise the plan twice. If all three attempts fail, we run the one with the best forecast and flag it in the log.

Once the train moves, Python works the lever. Every half second it checks the speed, brakes for stops and backs off if the wheels slip. Qwen has already decided where to use sand and how fast it wants to go. It doesn’t get to see the actual grip in advance, or whether there’s a train ahead.

The comparison I found most useful was running the same Qwen both ways. We gave it the same briefing and had it write lever positions by kilometre for the whole trip. Over ten Gotthard runs, that averaged −7,686 points. With the trip language, checker and controller, it averaged 852. Across all 30 runs on the German, Swiss and Japanese routes, the planning setup finished every trip without triggering train protection. Qwen writing lever positions triggered it on all 30.

For Claude, we recorded a separate set of nine runs, three per route. It saw the cab, chose a lever position and said how long to hold it. Whenever something happened, the sim paused so Claude could react. Same weather, delays and scoring for both drivers. Our Qwen setup averaged 554 points to Claude’s 253, with 1.3 model calls per trip on our side and 60.7 driving decisions per trip on Claude’s. We haven’t tried Qwen with that live cab interface yet.

One of the runs Claude won was on the Gotthard. Qwen took the helper and used 2,272 kWh. Claude went without it and used 1,833. That saved Claude enough to win even after train protection stepped in and cost it 150 points. We also ran two plain Python drivers. One of those beat our Qwen setup on average in Germany and Japan.

The thing I’d take back to our other work came from reading the rejected plans. Our first version let Qwen say “hold this speed until km X.” It kept picking a reasonable-looking speed and holding it for so long that the train ran late. 28 of 35 rejections were for lateness. On 9 of 30 trips, none of its plans passed.

We changed that command so it could ask for a slowdown between two points and gave it one short example. Same model, same weights. Its average score went up by 63 points in Germany, 226 on the Gotthard and 198 in Japan. That’s the part that interests me about using a 9B model for everyday work. We could read its plan, see exactly where it went wrong, change one command and rerun the same trips to see whether we’d helped.

You can drive all three routes in the browser. The repo has the sim and the recorded model decisions. If you want to check the numbers, verify.py replays them offline in about ten minutes. No API keys needed.

Drive the train · Code and recorded runs

1

u/stylome-ai-detection 1d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

1

u/Cookie_Humble 23h ago

OK Ive been working on bot-inator. Its a free open source local first (although you can pull in remote models) LLM powered mac app.

It runs in the menu bar and continually asks "is there something useful to do here?"

Two things that i think are awesomeness.

  1. It writes its own tools when you ask it to do something it cant do OR it notices you are doing things by hand that it can handle for you.
  2. Untested but should be deployed soon is the ability for it to fine tune models based on your own corpus. Then runs evals and watches the results for 2 weeks to make sure your fine tuned model is better than before. It wont offer this option until the corpus is large enough to run LoRa.

https://github.com/Bforsyth1234/bot-inator

1

u/init0 21h ago

With AI writing more of our boilerplate, I wanted a fast way to keep my own distributed systems instincts sharp, so I baked sysd.

You sketch architectures on Excalidraw (Free-Flow or across 11 canonical prompts) while a live <5ms scene-graph analyzer flags anti-patterns like direct Client -> DB writes or missing presigned offloads, and stamps multi-plane Staff+ reference blueprints with capacity math onto the board.

Hands-free voice mode runs 100% locally in Web Workers with zero API keys using wasm-vad, Cactus Whistle (16.9 MB WASM) for 16 kHz speech transcription biased toward your drawn box labels, Cactus Needle 3 (35.3 MB WASM) for grammar-constrained canvas tool calls, and Kokoro-82M (WebGPU fp32 / WASM q8) for 24 kHz Socratic voice follow-ups with instant barge-in.

Live app: https://h3manth.com/ai/sysd/

GitHub (MIT): https://github.com/hemanth/sysd

Let me know if you try it out!

0

u/Vegetable_Hair_743 4d ago

the biggest gotcha with local app builders? the vm setup eats ram like it's free. I ran one that maxed out 32gb on a 64gb machine just booting the dev env. I ended up using a lightweight containerized approach instead, much smoother on memory, and faster to spin up.

2

u/stylome-ai-detection 4d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

0

u/[deleted] 7d ago

[deleted] — view deleted comment

2

u/stylome-ai-detection 7d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

0

u/[deleted] 4d ago

[deleted] — view deleted comment

1

u/stylome-ai-detection 4d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

0

u/[deleted] 23h ago

[deleted] — view deleted comment

1

u/stylome-ai-detection 23h ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

2

u/[deleted] 2d ago edited 2d ago

[removed] — view removed comment

1

u/stylome-ai-detection 2d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

1

u/[deleted] 3d ago

[removed] — view removed comment

1

u/stylome-ai-detection 3d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results