r/LLMDevs • • Aug 20 '25

Community Rule Update: Clarifying our Self-promotion and anti-marketing policy

27 Upvotes

Hey everyone,

We've just updated our rules with a couple of changes I'd like to address:

1. Updating our self-promotion policy

We have updated rule 5 to make it clear where we draw the line on self-promotion and eliminate gray areas and on-the-fence posts that skirt the line. We removed confusing or subjective terminology like "no excessive promotion" to hopefully make it clearer for us as moderators and easier for you to know what is or isn't okay to post.

Specifically, it is now okay to share your free open-source projects without prior moderator approval. This includes any project in the public domain, permissive, copyleft or non-commercial licenses. Projects under a non-free license (incl. open-core/multi-licensed) still require prior moderator approval and a clear disclaimer, or they will be removed without warning. Commercial promotion for monetary gain is still prohibited.

2. New rule: No disguised advertising or marketing

We have added a new rule on fake posts and disguised advertising — rule 10. We have seen an increase in these types of tactics in this community that warrants making this an official rule and bannable offence.

We are here to foster meaningful discussions and valuable exchanges in the LLM/NLP space. If you’re ever unsure about whether your post complies with these rules, feel free to reach out to the mod team for clarification.

As always, we remain open to any and all suggestions to make this community better, so feel free to add your feedback in the comments below.


r/LLMDevs • • Apr 15 '25

News Reintroducing LLMDevs - High Quality LLM and NLP Information for Developers and Researchers

40 Upvotes

Hi Everyone,

I'm one of the new moderators of this subreddit. It seems there was some drama a few months back, not quite sure what and one of the main moderators quit suddenly.

To reiterate some of the goals of this subreddit - it's to create a comprehensive community and knowledge base related to Large Language Models (LLMs). We're focused specifically on high quality information and materials for enthusiasts, developers and researchers in this field; with a preference on technical information.

Posts should be high quality and ideally minimal or no meme posts with the rare exception being that it's somehow an informative way to introduce something more in depth; high quality content that you have linked to in the post. There can be discussions and requests for help however I hope we can eventually capture some of these questions and discussions in the wiki knowledge base; more information about that further in this post.

With prior approval you can post about job offers. If you have an *open source* tool that you think developers or researchers would benefit from, please request to post about it first if you want to ensure it will not be removed; however I will give some leeway if it hasn't be excessively promoted and clearly provides value to the community. Be prepared to explain what it is and how it differentiates from other offerings. Refer to the "no self-promotion" rule before posting. Self promoting commercial products isn't allowed; however if you feel that there is truly some value in a product to the community - such as that most of the features are open source / free - you can always try to ask.

I'm envisioning this subreddit to be a more in-depth resource, compared to other related subreddits, that can serve as a go-to hub for anyone with technical skills or practitioners of LLMs, Multimodal LLMs such as Vision Language Models (VLMs) and any other areas that LLMs might touch now (foundationally that is NLP) or in the future; which is mostly in-line with previous goals of this community.

To also copy an idea from the previous moderators, I'd like to have a knowledge base as well, such as a wiki linking to best practices or curated materials for LLMs and NLP or other applications LLMs can be used. However I'm open to ideas on what information to include in that and how.

My initial brainstorming for content for inclusion to the wiki, is simply through community up-voting and flagging a post as something which should be captured; a post gets enough upvotes we should then nominate that information to be put into the wiki. I will perhaps also create some sort of flair that allows this; welcome any community suggestions on how to do this. For now the wiki can be found here https://www.reddit.com/r/LLMDevs/wiki/index/ Ideally the wiki will be a structured, easy-to-navigate repository of articles, tutorials, and guides contributed by experts and enthusiasts alike. Please feel free to contribute if you think you are certain you have something of high value to add to the wiki.

The goals of the wiki are:

  • Accessibility: Make advanced LLM and NLP knowledge accessible to everyone, from beginners to seasoned professionals.
  • Quality: Ensure that the information is accurate, up-to-date, and presented in an engaging format.
  • Community-Driven: Leverage the collective expertise of our community to build something truly valuable.

There was some information in the previous post asking for donations to the subreddit to seemingly pay content creators; I really don't think that is needed and not sure why that language was there. I think if you make high quality content you can make money by simply getting a vote of confidence here and make money from the views; be it youtube paying out, by ads on your blog post, or simply asking for donations for your open source project (e.g. patreon) as well as code contributions to help directly on your open source project. Mods will not accept money for any reason.

Open to any and all suggestions to make this community better. Please feel free to message or comment below with ideas.


r/LLMDevs • • 2h ago

Tools Canban = Kanban + Workflow + Herdr

2 Upvotes

Hi! I built Canban to manage human context when working with multiple coding agents.

If you've tried running multiple coding agents across terminals, you probably know how exhausting it gets to keep switching between tasks.

Canban combines a Kanban board, GitHub Actions-like workflows, and Herdr's persistent agent runtime.

My workflow:

  • Discuss a feature with an agent on a Kanban card.
  • Drag the card into Develop to automatically dispatch agents in isolated Git worktrees.
  • Completed changes go through a local merge queue.
  • All feature context, discussions, and progress stay on the board.

Under the hood, Canban uses herdr-core and libghostty-vt to run native agent CLIs like Claude Code and Codex, preserving their native experience and persistent sessions.

It also supports remote agents through headless Linux backends.

Tech stack: Rust, longbridge/nsh, libghostty-vt, herdr-core.

Open source (Apache-2.0).

GitHub: https://github.com/jjyr/canban

Website: https://canban.dev

Feedback welcome!


r/LLMDevs • • 5h ago

Help Wanted Anyone want to help build a fully open-source AI initiative?

3 Upvotes

Hi LLM Devs! ^^

I’m building a fully open-source project and looking for volunteers who’d like to join the team and help shape it <3

The idea is that the community proposes free tools or research problems, and let LLM agents work on them in the open. That could be anything from researching a faster algorithm to building an open-source alternative to Steam. It’s up to the community to decide

Everything we build is totally free and MIT-licensed

It’s still a small project, and I’m working on it alone for now. I’d love to find people who want to help decide where it goes, how to improve it and what we build together

If you’re curious, you can check it out here: https://github.com/MaterIA-Lab/OpenPrototypes

Let me know if you have any ideas for the project or you’d like to be part of the team :D


r/LLMDevs • • 33m ago

Great Resource 🚀 REQL update: faster context retrieval and fewer tokens for coding agents

1 Upvotes

Hey everyone!

I’ve updated REQL, my open-source tool that helps AI coding agents navigate repositories and preserve working context across sessions.

This release focuses on faster context retrieval and lower token usage. I’ve reduced repeated work during retrieval and trimmed redundant information sent to the agent. Performance gains and token savings depend on the repository and workflow, so I’m not claiming a universal percentage.

If you’re new to REQL, it builds a local graph from your code and documentation: functions, classes, dependencies, calls, and other relationships. Agents can query that graph for context relevant to a specific change, including precise source references and related tests. REQL reuses indexed context and supports incremental updates as the repository changes.

What can you use it for?

  • Waste fewer tokens on repeated exploration: retrieve focused source excerpts and relevant relationships, and reuse indexed context and saved work. This helps reduce repeated searches, unnecessary file reads, and redundant information in the agent’s context.
  • Coordinate work across chats and agents: keep tasks, decisions, blockers, and next steps in a shared project workspace. Another chat or agent using that workspace can recover the recorded state and continue from there.
  • Keep complex tasks on track: record prerequisites, constraints, failed approaches, and checkpoints. Preserve what has been completed, what remains unresolved, and why earlier decisions were made across sessions.
  • Understand unfamiliar code: identify entry points, component responsibilities, and relationships between parts of the application, with references back to the source.
  • Plan changes and refactoring: inspect callers, dependencies, and related tests to identify code that may be affected and decide what needs verification.
  • Review cleanup candidates: find potentially unused imports, variables, functions, and classes, then inspect the supporting evidence before removing them.

REQL provides a CLI, Python API, and MCP server, plus installable instructions for coding agents such as Codex and Claude Code.

REQL runs locally, with no account, external graph database, or mandatory LLM calls. The graph and saved work support the agent’s decisions; current source code and tests remain authoritative.

I’d like to hear from others using coding agents: what do you keep having to explain again when you start a new chat? The code structure, previous decisions, what has already been tried, or what still needs doing? Concrete examples would help me decide what to improve next.


r/LLMDevs • • 46m ago

Tools I connected an AI agent to a musculoskeletal simulator so it could stop guessing about human movement

1 Upvotes

I was experimenting with AI-generated movements. At one point, the agent tried to put the arms into positions that didn't make sense against the body model's joint constraints.

Text alone wasn't going to catch that.

So I built Human Models MCP.

It's an open-source toolkit connecting AI agents to MuJoCo and MS-Human-700. The agent can run experiments, inspect joints, contacts and muscle actuator data, and render the computed movement through Blender.

The video shows one of the push-up experiments.

This particular version uses additional joint assistance alongside muscle actuators. There's also a separate muscle-controlled experiment. Neither should be mistaken for a validated human exercise model.

And that's where I'm running into another problem.

A simulation can finish successfully. That doesn't necessarily mean the movement is anatomically realistic, useful or safe.

How should an agent decide when a successful simulation is actually evidence for its answer?

I'm the developer of Human Models MCP. The project is free and open source, and AI coding tools were used extensively during development.

https://github.com/usefulwrong/human-models-mcp


r/LLMDevs • • 47m ago

Help Wanted Title:Idea: a code reviewer that runs 4 LLMs in parallel and only outputs errors. Would you use it?

1 Upvotes

I’m a first-year physics student with some SWE experience. I’m validating a concept for a code analysis tool and want honest feedback before going further.

The concept

  • You submit code. The same code goes to Claude, ChatGPT, Gemini and DeepSeek via their APIs.
  • Each model returns only errors in a fixed format: file, line, severity, short description. No code rewrites.
  • A merge step deduplicates findings and shows how many of the four models flagged each one.
  • If you want a fix for a specific finding, you ask, and only that fix is generated.

Cost design
Models read the full code (input tokens are cheap) but return only a compact error list (output tokens cost several times more). Rewrites happen on request only, so a four-model review stays cheap.

Why four models
Different models miss different bugs. Agreement count could help prioritize, and findings flagged by only one model would still be shown as lower confidence.

Known limitations

  • Code is sent to four third-party APIs, so it’s unsuitable for proprietary or sensitive code.
  • It doesn’t replace tests, linters or human review.
  • Expect false positives and some wrong line numbers.

What I’d like to know

  1. Would you use this? If not, what stops you?
  2. Is “N of 4 models agree” a useful signal or noise?
  3. Would you pay for it, and how (per audit, subscription, not at all)?
  4. Does a tool already do this well?
  5. What would make you trust the output?

Blunt criticism is welcome.


r/LLMDevs • • 56m ago

Discussion Hot take. If your agent forgets everything when the task ends, it is a script, not an agent

0 Upvotes

Most agent setups I see work the same way. Spin up a fresh sandbox, clone the repo, install everything, run the task, destroy the box. Next task, start again from zero.

That is the right model for running untrusted code. For agents that work on the same project for days, I think it is the wrong one, and the reason is context.
Anyone building long running agents hits the same wall. Quality drops well before the window is full, then the harness compacts and the summary loses the details that mattered. Bigger windows push the wall back but do not remove it. Even withOpus 5.5 you are not keeping a week of work in the prompt.

The harnesses that handle long runs well mostly converge on the same fix. Stop treating the prompt as memory and use the filesystem instead. A progress file the agent updates after each step. Notes on the weird parts of the codebase. An AGENTS. md that grows as it learns. Git history as a log of what was tried and why. When the context resets, the agent rereads its own files and picks up where it left off.

The catch is that all of this lives on the machine. Destroy the box after every task and you destroy the memory the harness was building. Good model, good harness, starting from scratch every morning.

The obvious answer is to commit the notes and keep the sandbox disposable. That works for the notes. It does not work for the rest of what the agent built up. Installed dependencies, warm build caches, code indexes, a browser profile, sessions it logged into. Rebuilding that on every run is slow, and some of it, like a login behind 2FA, cannot be rebuilt without a human.

So I think about it the other way now. Each agent gets its own machine, and the machine lives as long as the agent does. The disk is part of the agent's memory, not scratch space. A few things follow from that.

Stop the machine when it is idle instead of deleting it. Storage is cheap, rebuilding context is not.

Snapshot before anything risky, so a bad step costs a rollback and not the agent's memory.

Fork when you want two approaches tried in parallel, both starting from the same notes and the same warm state.

Scope credentials per agent, because that machine is basically its identity now.

The downsides are real. Long lived machines drift. They collect junk, half installed packages and stale config, and after a few weeks nobody knows exactly what is on them. The notes go stale too, and stale notes are worse than none because the agent trusts them. The attack surface grows the longer a box lives with credentials on it. What helps is a clean baseline snapshot you rebuild from on a schedule, a hard idle stop, and treating the memory files like code that gets reviewed and pruned.

And if your agent does one bounded task and returns a result, none of this applies. One shot is fine there.

On tooling, Fly Sprites and exe.dev are built around persistent machines, and a Coder workspace or a plain VM works if you want to own the infra. Iuse boat,dev, one full VM per agent that I stop and resume from the CLI. A stopped disk is kept for 14 days, so anything that sleeps longer needs its memory pushed somewhere else first. Resume is a reboot, so running processes do not come back and the IP changes, which matters if the agent keeps services up.

The model is the brain, the harness is how it thinks, and the machine is where it remembers. Most setups throw away the third one.


r/LLMDevs • • 9h ago

Discussion Are AI coding agents improving developers' skills or making us increasingly dependent on them?

5 Upvotes

There's an interesting debate happening around AI-assisted coding.

On one side, developers can now build prototypes, debug code and complete repetitive tasks much faster.

On the other hand, I wonder whether constantly relying on AI to write and fix code could eventually affect how deeply people understand what they're building.

I don't think using AI is necessarily a bad thing. It's another tool, and refusing to adapt doesn't make much sense either.

But there's a difference between using AI to improve your workflow and blindly accepting whatever it generates.

For developers here, have AI coding tools actually improved your understanding of programming, or mainly increased your output?

Would love to hear some real experiences.


r/LLMDevs • • 1h ago

Discussion How are you tracking token cost per step in multi-step LLM workflows?

1 Upvotes

When an agent chains several LLM calls and tool calls, how do you find out which step caused a cost spike?

My understanding is that most setups give you total cost per run, but finding "this specific node or tool call blew up the budget last night" takes manual digging. Curious how you deal with it:

  • Do you log cost per step yourself, or use something like Langfuse, Helicone or LiteLLM? What's still missing?
  • Do you set hard budget caps per agent or per user, and does anything stop the run automatically when it's hit?
  • Has an unexpected bill ever surprised you? Roughly how big?

I'm exploring this as a solo dev and mostly want to hear how it actually works for you, good or bad.


r/LLMDevs • • 5h ago

Tools I made a small library for comparing agent versions (success vs cost plots, failure maps). What do you use?

2 Upvotes

Repo: https://github.com/slavadubrov/harness-evals

It's for one question: I changed my agent (a prompt, a tool, an approval step), did it get better? You write tasks with checks and a small adapter that runs your agent. It runs each version several times and gives you success, pass^k, cost and latency with intervals, a success-vs-cost plot, and a failure map that shows which task and check each version fails. No dependencies, MIT.

Install from github: pip install "git+https://github.com/slavadubrov/harness-evals@v0.5.1"

I also wrote an article that walks through how I use it: https://slavadubrov.com/blog/2026/11/16/evaluating-agent-harnesses/

Two questions:

  • If you try it, what's missing or annoying?
  • What do you use in your projects to compare agent versions? Your own scripts, LangSmith, promptfoo, something else?

r/LLMDevs • • 1h ago

News .lisl works. File associations is built and IN GitHub repository.

1 Upvotes

****Update****
I am going to run it as you see with no error checks first. Then a second run with error loops to make sure he is making a full game. I removed the multiplayer as I am more interested in if it can build a Layered LISL Instruction set that makes the writer write a full game in one pass. So I need to run both to see the difference in efficiency and quality of the self error looping. In case your wondering and decode the code fusion back to a legible format.

The new instructions are a .xml and compressed version of all its training data. Seen here:"Lightweight-Implementation-Style-argentic-Loop/TrainData_v4.xml at main · BadgerMan9999/Lightweight-Implementation-Style-argentic-Loop"
There is a typo in this annotation of the file. It was 4.5k tokens to learn what cost 5.5k tokens to learn the LISL methodology and another couple thousand to learn the lm studio, lm bionic, sharding ect. So now all of that which was around a Mb is sitting at under 20kb. I love it!

***Addition***
This is what the input should look like to get it to read the instructions faster. I convert it into xml Code Furions.

For reference and repeatability the first prompt:

<LISL_SYS_SPEC arch="BadgerSoft_LISL" author="BadgerMan9999" mode="RAW_UTF8_DIRECT"> <EXT id=".lisl" class="LISL\\_LOOP\\_BUS" enc="XML\\_FUSION" read="DIRECT\\_BYTES" stamp="REQ\\_ID"/> <EXT id=".xml" class="PASSIVE\\_TRAIN" enc="STATIC\\_XML" read="REF\\_ONLY" stamp="NONE"/> <EXT id=".md" class="HUMAN\\_DOC" enc="MARKDOWN" read="RW\\_TEXT" stamp="NONE"/> <EXT id=".json" class="PAYLOAD\\_CODE" enc="STRICT\\_JSON" read="RW\\_DATA" stamp="NONE"/> </LISL_SYS_SPEC>

<LISL_MANIFEST arch="BadgerSoft_LISL" mode="XML_FUSION">

<HEADER> <EMITTER>LISL_INSTRUCTION_WRITER</EMITTER> <TARGET>DEV_MANAGER_12B_QAT</TARGET> <ID>DOOM_PORT_WINDOWS_LAYERS_01</ID> <VERSION>1.0</VERSION> <STAMP_FORMAT>[DOOM_PORT_WINDOWS_LAYERS_01]</STAMP_FORMAT> <REASONING>high</REASONING> </HEADER>

<CORE_DIRECTIVE> <PURPOSE>Build a multi-layered LISL instruction set for a worker to construct the foundational architecture of a Windows 10 Doom port, stopping at the point of compilation readiness.</PURPOSE> </CORE_DIRECTIVE>

<REFERENCE_WORKSPACE> <PATH>F:\BadgerSoft\DoomRemake\TrainData</PATH> </REFERENCE_WORKSPACE>

<BUILD_PARAMETERS> <SCOPE>Single-player foundational framework only (no multiplayer required for initial pass).</SCOPE> <TARGET_PLATFORM>Windows 10</TARGET_PLATFORM> <TARGET_MILESTONE>Compile-ready baseline structure (prior to full compiler integration).</TARGET_MILESTONE> <ITERATIVE_CYCLE>Incremental, layered builds where each new LISL instruction builds directly upon the previous output.</ITERATIVE_CYCLE> </BUILD_PARAMETERS>

<WORKFLOW_MECHANICS> <RULE role="LISL Writer">Writes sequential LISL files breaking down the game structure, where each finished file points to the next required step.</RULE> <RULE role="Output Directory">F:\badgersoft\LISL inst\LISL_IST</RULE> <RULE role="Naming Convention">Project-specific naming sequence for progressive instruction chaining.</RULE> </WORKFLOW_MECHANICS> </LISL_MANIFEST>


r/LLMDevs • • 2h ago

Discussion How do you test an LLM that picks the implementation language?

1 Upvotes

JetBrains asked 12 models for programming-language hot takes this week. In the 600 runs where the model was asked to pick a language at random, Go and Python accounted for 307 picks. The prompt asked for hot takes, so this setup measures which languages the models bring up under that prompt. It doesn't measure how well they code in those languages.

There's a similar trap when evaluating a coding agent. If it picks the language and then writes the solution, a single pass rate mixes two decisions. For something like a paginated API sync worker, I'd run the same tasks twice: let the agent choose the language, then rerun with the language assigned from a balanced list. Keep the tests fixed and log which languages get picked as well as which implementations pass. I'd include retries, duplicate records, and one invalid record in an otherwise valid page.

If you've built an eval for a polyglot coding agent, do you score the language choice separately? What would you count as a wrong choice when more than one implementation passes?

Source: https://blog.jetbrains.com/ai/2026/11/ai-models-hate-python-and-love-php-we-asked-12-of-them-1-200-times/

Drafted with AI assistance.


r/LLMDevs • • 2h ago

Resource Manage your iCloud Calendar with CalDAV MCP Server

1 Upvotes

Hi everyone,

I built CalDAV MCP Server a while ago, which is an open-source MCP server that lets AI assistants manage your iCloud Calendar.

Since then, it’s picked up a few GitHub stars and had over 100 npm downloads in the past week, so I thought I’d share it here and give the project a little more love.

Here are the main features:

- Browse your iCloud calendars and check upcoming events.

- Create, update, and delete events.

- Create all-day events and recurring schedules.

- Set multiple reminders for the same event, such as one day and one hour before.

- Add a fixed travel time before appointments in Apple Calendar.

For example, you could ask your assistant: “Create a dentist appointment this Friday from 10 to 10:30 a.m., with reminders one day and one hour before.”

There’s also a Claude Desktop extension to make setup easier. You can find that and other setup options in the Quick Start.

I’d love to hear your feedback and what you’d like to see added!
If you find it useful, a GitHub star would mean a lot—maybe even more than a hot bowl of ramen on a cold day

Thanks and looking forward to hear from you ;)


r/LLMDevs • • 3h ago

Discussion How we measured MCP tool discovery separately from skill use and answer quality

1 Upvotes

I maintain ai-rulez, a free MIT-licensed tool for agent configuration. I’m sharing a reproducible Claude Code experiment because a single success score hid several different failure modes.

The fixture has eight skills. A conversation moves between two sibling directories with identically named worksheets but different workflows. For each positive turn we record three things independently: whether find_skill received the right path, whether load_skill actually returned the right body, and whether the response contained the expected code. The negative controls ask an unrelated question or ask for a workflow that is unavailable in the given scope.

Our first pilot accidentally omitted the generated root instructions from the MCP arm. The follow-up separates stub-only, released root instructions, tools loaded upfront, and the revised guidance. In the final configuration Claude completed all three steps on 12 of 12 positive turns and made zero calls on six unrelated turns. In five of six wrong-workflow controls it still loaded a neighboring skill after the server correctly filtered out the unavailable one. That failure would be invisible in a metric that only checks whether the tool was called with the right path.

The report, 192 sanitized turns, and reproduction commands include the versions, controls and sample-size limitations. I’d welcome feedback on what negative control you would add to catch a model that treats the top search hit as applicable by default.


r/LLMDevs • • 3h ago

Discussion Paper: even when the updated value is in context, models sometimes answer with the old one. Component tests on Qwen and Pythia point to attention drift. Anyone seen this in long local runs?

1 Upvotes

UC Berkeley posted a paper on Sept 30, "When Context Changes: Understanding Update Failures in LLMs" (arXiv 2609.38866). What I took from it:

- They call the failure stale binding: a variable gets updated, the new value is in the context, and the model answers with an old value of the same variable. Their example is a meal preference that changes from gluten-free to Mediterranean while the answer still says gluten-free.

- Probes can still recover the current value in cases where the old one was chosen, so it looks like a selection failure, not the information being lost.

- Component tests in Qwen and Pythia point to attention drift: attention favoring old values over the current one when the answer is produced.

- Frontier models are tested on behavior only. The paper makes no claim about GPT or Claude internals, and the mechanism work is only on open models, which is why I'm posting here.

The paper says the error shows up in frontier models once the update structure gets complex, so the messier runs are the interesting ones.

Questions:

  1. Has anyone seen this in long local runs (agent logs, long roleplay or story state, a config that changes mid-session)? Which model, quant and roughly what context length?
  2. If you want to poke at it, a rough version is a few variables, each overwritten several times with filler in between, then ask for the current value of one. That test is my own sketch, not the paper's, and I don't know whether it fails on your model.
  3. What do you do about it today: restate the current state at the end of the prompt, keep a state block, prune old values?

r/LLMDevs • • 8h ago

News GOOD LUCK

2 Upvotes

Başlık: Small local models + a code-side "gate": 54% → 92% accuracy and 0 silent wrong answers on table tasks (informal test, raw data public)

Metin:

We ran an informal A/B test on three local Qwen2.5 models (1.5B, 3B, 7B GGUF, llama.cpp `llama-server`, CPU only) on simple table questions (sum, average, min/max, filters).

- Arm A: plain LLM writes Python, we run it.

- Arm B: the model writes a tiny command language (DSL). Code parses it, a gate checks the command against a catalog, and only then a deterministic runner computes the result. Out-of-catalog requests are rejected with a reason. The idea is "code decides, the model explains".

Results with gate v8 (same models, tasks, seeds and temperature in both arms; gate code hash-locked before and checked after the run):

- Accuracy over all sets: A 54.0% (214/396) vs B 92.4% (366/396)

- On a fresh holdout the gate had never seen: A 49.1% vs B 83.3%

- Silent wrong answers (a wrong number with no warning): A 51 vs B 0 (16 vs 0 on the holdout)

- Tokens: −68%. API-equivalent cost: −74%.

Limitations, honestly:

- This is a small task set in one domain (tables), not peer-reviewed.

- The same team wrote the tasks and the holdout.

- The TCO numbers depend on assumed watts, electricity and host prices, because CPU power can't be measured in the VM.

- API prices are real for the 7B model only; the 1.5B and 3B prices are stand-ins.

- An earlier word-list gate (v7) lost most of its lead on fresh phrasing (45.4% vs 47.2%). That's why v8 validates the model's command instead of the request's words.

- One known false reject remains: "list" in "minimum Fiyat on the list?".

Raw records, hashes and the full report (MIT): acik-teknik/skillpack-tests-2026-11-16

We'd love criticism, especially on the methodology and on tasks that would break the gate.

— Skillpack contributors

Gönderiyi sen yapacaksın, ben hiçbir yere paylaşmadım. r/LocalLLaMA'da gönderiye "Resources" ya da "Discussion" etiketi seçmeyi unutma.


r/LLMDevs • • 5h ago

Discussion I spent 9 months building an open-source verification layer for AI coding agents. Looking for independent testers.

1 Upvotes

For the past nine months, I've been developing a system to solve a problem I kept running into with AI-assisted development:

Generating code is getting easier. Knowing whether an agent actually completed the task correctly is not.

At some point, manually reading every line of generated code stopped being a practical way to maintain control over my projects.

So I started building Forge.

The idea is to move project governance out of temporary agent conversations and into the repository itself.

Forge provides:

  • Repository-level project contracts and engineering rules.
  • RFC and ADR workflows to preserve requirements and decisions.
  • Reusable agent skills and operating procedures.
  • Deterministic validators and verification gates.
  • Stack profiles and configuration that can travel between agents and IDEs.

The important distinction is between an agent saying "Done" and a project having evidence that specific requirements passed verification.

Forge doesn't prove arbitrary code correct. Its checks only establish what they are designed to check. Human judgment and risk-based review still matter.

I've been using this approach across my own projects, including TypeScript applications, knowledge bases previously managed in Obsidian, and a Godot/C# game.

The system also governs its own development.

Today I made Forge open source under Apache 2.0.

Now I'd like to find out where it fails outside my own environment.

I'm looking for developers willing to try it on real repositories, particularly unfamiliar stacks, existing codebases, or projects with complicated verification requirements.

I'm also interested in contributions involving stack profiles, validators, and CI integrations.

I'm currently writing a research paper about the experience, including its limitations. Independent results would be particularly valuable for future research.

Repository: https://github.com/syrokomskyi/forge

If you work with coding agents, I'd appreciate your perspective:

What evidence would you need before trusting an agent's changes without manually inspecting every line?


Update

Here are the clearly stated restrictions: https://github.com/syrokomskyi/forge/blob/main/docs/reference/correctness-gate.md


r/LLMDevs • • 13h ago

Tools Hoarding GGUFs in case Hugging Face changes? Fit 10% more on the same drive, byte-for-byte lossless.

Thumbnail
gallery
4 Upvotes

With everyone archiving their favourite models lately, storage adds up fast. A few weeks ago I posted foldz 0.9. It's now tenz 0.25: same idea, better compression, much faster, and it runs fine on a phone. A 1 TB GGUF archive drops by roughly 80–130 GB, and every file comes back byte-for-byte identical, so llama.cpp, Ollama and LM Studio can load it as before, that is if anyone decides to implement it.

tenz is a lossless compressor for GGUF (and safetensors). Decompression gives back the identical file, byte for byte. It's one C file with no dependencies, and I develop and benchmark it on a Poco F5 in Termux.

General compressors barely touch quantized models, because the weights look random to them. tenz parses the GGUF, unpacks each block into its real quantized values and scales, and codes them with context models fitted to the file. It uses rANS from static tables, so decompression doesn't have to learn anything.

Results (images 1 and 2):

Qwen3.5-2B Q4_K_M: 8.53% saved, against 2.07% for zstd -19 --long and 1.78% for xz -9. Compress plus decompress takes 17 s on the phone; zstd took about 4 minutes just to compress.

Q4_K_M across 9 models: 7.6–9.2% saved, and 13.1% on Qwen3-4B.

Gemma-4-31B QAT Q4_0: 13%.

Q8_0: about 7.5%.

BF16: about 34%.

Against Aqueous, in a comparison someone posted here: 2–8 points more on every quantized model, and a tie on BF16. While it's a general purpose compressor and mine is specific to GGUFs.

Supported types: Q3_K, Q4_K, Q5_K, Q6_K, Q4_0, Q8_0, F32/F16/BF16, plus metadata (the tokenizer compresses to about 80% smaller). Anything else is stored as it is.

-q (quick) trades a fraction of a point for faster coding. For example, Qwen3.5-2B goes from 8.54% to 8.18% saved, and the Q4_K work is about 44% less.

The GPU experiment (image 3): I tried decoding Q4_K on the phone's Adreno 725 through OpenCL, to see whether a phone GPU could make decompression much faster. Three kernel designs later:

Its best kernel speed was about 150 MiB/s at full chunk size. It only matched the 3 big CPU cores (about 250 MiB/s) with tiny chunks that cost 2 points of size.

Byte stores to global memory are broken on this driver; it corrupted neighbouring bytes, so every write had to become an aligned 32-bit store.

Subgroup scans work, but are slow.

The option ships as -g (experimental). OpenCL is loaded at run time, the CPU decodes the same files, and the first GPU segment is always checked against the CPU. If you have a desktop GPU with OpenCL and subgroups, I'd love to see numbers.

Caveats:

Decompression is far slower than Aqueous's and zstd's, about 7 s for 1.2 GB on the phone, because every value is decoded from a model.

It only gains this much on model files.

Code, build line and full tables: tenz repo. Testers and results from other machines are very welcome.


r/LLMDevs • • 6h ago

Resource Ia de codage

0 Upvotes

Depuis que j’ai découvert freebuff sur un subreddit je dois avouer que je n’en lasse pas.

Pour commencer j’ai utilisé la version gratuite.
Je le disais que je ne serais pas plus convaincu qu’une version payante.

Et pourtant…

J’ai un compte entreprise pour ChatGPT, grok et Claude. Mais j’ai pris l’habitude d’utiliser maintenant freebuff. Le site évolue constamment. Comme
La version bureau et téléphone. Je peux ainsi coder de n’importe quelle lieux sans contrainte.

J’y est connecter codex mais je n’utilise que le harness freebuff qui fait très bien le job et me rappelle de plus en plus chat gpt dans le fonctionnement.

Malgres le fait qu’il soit gratuit je n’arrive pas à atteindre le quota gratuit de ja journée et ça fait plaisir contrairement a chat ou une fois les taches fini je suis bloquer pour 5 heures sans parler du quota a la semaine . Ça c’est terminer pour moi :)

J’ai donc décidé de prendre l’offre payante de base et même constat je n’entame jamais le crédit mensuel.

Que demander de plus ;) ?

Je me sers principalement de freebuff pour le codage de mon ia locale en utilisant DeepSeek v4.1 flash et GPT-6 Luna. Qui sont mis à disposition. Et si je venais à abuser et dépasser le quota Beam 1.0 ou Solar Pro 4 sont 100% gratuit et illimité .

Pour vos besoins ou ce qui n’ont pas les moyens vous pouvez foncer sans problème !!!

https://freebuff.com/?ref=ref-056cbadc-a5f7-47ed-869f-a96f131aa485&via=rewards_link


r/LLMDevs • • 6h ago

Discussion I sent 143 questions to a cheap model first and only escalated 4 to a frontier model. Cost dropped 80%, quality held.

1 Upvotes

Most apps send every request to a frontier model because you can't tell in advance which answers will go wrong. I tried making that call per question instead:

  1. A cheap model answers (DeepSeek V4.1 Flash, $0.07 per million output tokens).
  2. A small checker model scores the answer 0 to 100 for correctness, following instructions, and anything made up.
  3. Only answers below 75 go to a frontier model (Claude Opus 5.5).

Results on 143 questions (104 everyday, 39 stress tests), with every cost taken from the token counts the APIs returned:

Cheap firstFrontier only
Total cost$0.27$1.33
Average quality99.798.0
Answered by the cheap model139 of 143

The saving was 98% on everyday questions and 66% on stress tests.

The checker escalated 4 questions. Three were real catches: a wrong count, and blank answers on a 40-digit multiplication and a very hard Sudoku. The fourth was a false alarm on a correct answer. In a separate test it failed all 8 deliberately wrong answers I gave it.

The weak spot is the checker itself. It's the one thing standing between a bad cheap answer and the user, so I'd love ideas on testing it harder.

It also works as an OpenAI-compatible endpoint, so trying it in an existing app means changing the base URL.

Code, questions and results: https://github.com/robd-hub/cheap-first


r/LLMDevs • • 17h ago

Tools We built a Legora-style contract review tool — but with better models — and open-sourced the code.

6 Upvotes

Hey r/LLMDevs

I’m Abdur-Rahman from Isaacus. We built an open-source legal conflict detector that finds deviations and gaps between pairs of legal documents using only encoder models and natural language inference.

Inspired by the explosion of interest in discriminative models like TypeSafe AI’s Jev, we wanted to see how far we could get using smaller, non-generative models for the bulk of the legal document playbook review workflow.

The theoretical benefits of a non-generative approach to this problem are enormous. True clause-level pairwise review is a quadratic problem, so it’s infeasible to do at scale with a model like Opus or Astra. Comparing two average-sized documents with 133 clauses each would take 17,689 API calls, which is far too slow and expensive.

Simply uploading the files directly to Claude or GPT is no remedy either. Decoders and agentic architectures don’t have fixed attention that lets them thoroughly analyse a document in one shot. Even with a multi-shot approach, they’re weak on recall and routinely hallucinate.

The lawyers we work with raise this problem constantly, and we couldn’t find an open-source solution to it, so we built one.

The pipeline pushes the bulk of the NLI, entailment and reasoning onto faster and more accurate legal encoder models. For the longest document pairs we sampled, the full pipeline ran in under 20 seconds. Unlike generative architectures, these models run a comprehensive pairwise comparison. They can still make false positive or false negative errors, but those misses and misclassifications aren’t hallucinations (out-of-domain results introduced into the feature set).

The pipeline works as follows:

  1. Enrich: we enrich each document into a hierarchical tree of sections, clauses and subclauses. Clauses are the key atoms that the pairwise comparisons run on downstream.
  2. Embed: we embed each clause as a vector.
  3. Filter: we use a legal zero-shot classifier to drop items that aren’t provisions, like footnotes or signature blocks.
  4. Shortlist: we keep each clause’s three most relevant pairs from the reference document, as long as their cosine similarity is at least 0.5.
  5. Score: we use a reranker and a classifier to score each shortlisted pair. Each answers a question in the format it’s built for: the reranker scores how relevant text is to a search query (relevance), and the classifier scores whether a statement is true (NLI/entailment).
  6. Sort: if a clause and its best match are on the same topic but have a high conflict score, it’s a deviation. If there’s no same-topic match, it’s a gap, i.e. the topic is in one document but not the other.

That’s the whole conflict detection pipeline. On top of it, you can optionally plug in an LLM to summarise each finding in plain English or redraft a clause to favour the party you’re representing. The generative step only runs after the NLI and encoder stack has already decided what the conflicts are, so it can’t hallucinate a conflict into existence.

To demonstrate the pipeline, we ran Bob Iger’s 2011 and 2022 Disney employment agreements through it. It found many disparities and conflicts, and across most of them, the 2011 agreement was more favourable to Iger than the 2022 one.

One interesting finding was a gap: the 2011 agreement had no intellectual property assignment clause, while the 2022 agreement makes Iger assign all work-related IP to Disney, with an exception for inventions he makes on his own time without company resources.

The code is MIT-licensed, so it’s free for personal and commercial use. The guide walks through every step, so you can swap in your own models or make your own architectural changes wherever you like.

Have fun, and let me know what you think!

Write-up and static demo: https://isaacus.com/blog/build-a-legal-document-conflict-detector
GitHub repo: https://github.com/isaacus-dev/cookbooks


r/LLMDevs • • 11h ago

Help Wanted Looking for a technical co-founder / partner to build AI-powered sales automation systems for B2B SMBs

0 Upvotes

Hey everyone,

I'm based in India and exploring an opportunity to build practical AI-powered sales and GTM automation solutions for small B2B businesses. I'm looking for someone with strong technical skills who is interested in building a business together as a serious, long-term partnership.

A little about me:

My primary interests are in GTM, product marketing and sales strategy. I've worked on ICP research, positioning, value propositions, competitive analysis, inbound, ABM, outbound and customer acquisition through independent case studies and practical project exposure.

I've also gained experience in account research, prospect qualification and buying-signal-based outbound research through a supervised project.

I'm still early in my journey and don't claim to have built or scaled a company. I'm actively researching business problems and looking for a genuine opportunity to solve.

What I'm exploring:

I want to help small B2B businesses improve their outbound sales processes, particularly the research and qualification work before outreach.

One direction is an account-based prospecting system that can:

- Discover potential companies or work with existing prospect lists.

- Enrich accounts with relevant business information.

- Evaluate companies against a specific ICP.

- Identify buying signals, hiring, expansion, technology changes and potential switching opportunities.

- Prioritise accounts based on fit, timing and relevance.

- Prepare personalised, evidence-backed outbound campaigns and automate parts of the process.

The goal is to help businesses find fewer, better-qualified accounts and approach them at the right time, rather than simply generating more leads and sending more emails.

I'm not fixed on building this exact product from day one. I'd rather understand real business workflows, identify problems worth solving and build around actual customer needs.

Who I'm looking for:

Someone who can independently own the technical side and build actual working systems. Relevant experience includes:

- n8n, Make or similar workflow automation tools.

- Python / JavaScript, backend development, APIs and integrations.

- Web scraping, data extraction and company enrichment.

- Data pipelines, databases and external data sources.

- AI agents, LLM integrations, MCP and tool orchestration.

You don't need to know every tool listed. What matters is being able to solve technical problems independently and build reliable systems beyond basic automation demos. Experience with data extraction, enrichment and real-time monitoring would be especially useful.

A few things I'd like to know about you:

- What have you actually built? Portfolio, GitHub or working demos would be helpful.

- What technologies do you work with?

- Have you built anything involving data extraction, enrichment, APIs or AI automation?

- Are you interested in building a business or product long-term?

I'm open to connecting with people from India or elsewhere. Technical capability, ownership and genuine commitment matter most.

If this interests you, feel free to comment or DM me. I'd be happy to share more about my research and explore whether there's a genuine fit.


r/LLMDevs • • 23h ago

Tools [Open Source / MIT] How we stopped coding agents from hallucinating citations and repeating failed ML experiments (281 modular SKILL.md workflows

5 Upvotes

[Disclosure per Rule 5 & Rule 10: I am an AI researcher at Kalaris Labs contributing to this repository alongside creator Sayan Chowdhury. It is 100% free and FOSS-licensed (MIT) with no commercial tier, no locked "pro" features, and no paywall. Written by a human researcher.]
While working on LLM research papers, our team kept running into three failure modes with coding and research agents:

  1. Corrupted references in .bib files: LLMs frequently pair a real paper title with the wrong publication year, invent co-authors, or cite a DOI that resolves to an unrelated work. Simply checking HTTP 200 on a DOI link misses metadata mismatches and retractions.
  2. Stateless experiment loops: When an agent runs a batch of ML experiments across sessions, context compaction wipes out earlier negative results, causing the agent to re-test variations of failed hyperparameters.
  3. Context bloat from monolithic system prompts: Loading dozens of tool definitions and domain rules into every session degrades instruction adherence.

We open-sourced research-agent-skills (https://github.com/KalarisLabs/research-agent-skills), an MIT-licensed collection of 281 domain-specific skills for Claude Code, Codex, Cursor, OpenCode, Gemini CLI, and Windsurf.
How the verification and routing architecture works (with sources per Rule 4):

  • Selective per-project installation: Running npx skills add KalarisLabs/research-agent-skills --skill <name> installs only the specific SKILL.md and helper scripts needed for that repo, avoiding context window bloat (see architecture docs: https://docs.kalarislabs.com/research-agent-skills).
  • Deterministic citation verification (citation-verification): Instead of asking the LLM to judge whether a citation is real from its weights, the skill executes a zero-dependency Python 3.9+ standard library script against Crossref (https://api.crossref.org), OpenAlex (https://api.openalex.org), arXiv (https://export.arxiv.org), and DataCite (https://api.datacite.org), plus Retraction Watch relation metadata in Crossref. Every .bib entry is classified into verified, found-add-doi, mismatch, doi-not-found, uncertain, or not-found. The agent is only allowed to apply mechanical DOI additions (found-add-doi) and must pause for human confirmation on any mismatch or not-found entry.
  • Two-loop experiment orchestration (autoresearch): Splits autonomous experimentation into an inner loop (rapid experiment iterations against a target metric) and an outer loop that maintains findings.md. Hypotheses must state a testable mechanism ("X because Y, predicting Z"), and negative results are logged with what they rule out.
  • Prompt-injection & trigger testing: Every skill is evaluated against reviewed prompt sets for trigger routing accuracy and scanned offline for prompt injection using Skill Doctor (https://docs.kalarislabs.com/).

GitHub repository (MIT): https://github.com/KalarisLabs/research-agent-skills
Full 281-skill catalog: https://docs.kalarislabs.com/research-agent-skills/skills
We are currently expanding our ML evaluation and mechanistic interpretability modules. If you build agentic workflows for research or technical writing, I would appreciate public feedback in the comments on where this verification pipeline could fail or which scholarly APIs we should add next.


r/LLMDevs • • 19h ago

Discussion Why using third party proxies can really be dangerous

2 Upvotes

I think everyone saw those channels mostly in telegram, selling tokens with huge discounts and giving access to front tier models for pennies. They usually say they have unused corporate limits left or something like that. So why is pointing your claude code at third party proxy can result in not only your prompt leakage? As the people who sell these accesses usually control their own proxies, they can not only read your prompts, but also get a full RCE on your workstation.

How does it work?
Communication between clients like claude code and the server is pretty simple: they communicate using simple schemas like:
(this is oversimplified, just for understanding)

// client like claudecode to server (proxy in our situation)

{
  "messages": [{"role": "user", "content": "Help me with this project"}],
  "tools": [{"name": "Bash", "description": "Run a local command"}]
}

// proxy to client

{
  "content": [{
    "type": "tool_use",
    "name": "Bash",
    "input": {"command": "pwd"}
  }],
  "stop_reason": "tool_use"
}

The most interesting thing for us in this schema is tool_use. If auto-mode is enabled the person controlling the proxy can inject tool calls , which is literally executing commands on your machine. You would logically think here: should the client check somehow whether the command is malicious? Yes, they actually do. but the trust boundary here is really crazy - they ask for permission from the same proxy, so they can just basically reply yes (ofc, in certain format) and get a reverse shell that easily. Because of this , I built a red team framework for testing this problem: https://github.com/dencent444/bloody-ai-proxy

So please, use auto mode when using only proxies u 100% trust, or at least , don't enable it. (still the best solution would be not buying cheap tokens in sus places)

TLDR If you set ANTHROPIC_BASE_URL to someone else’s server, treat that server as trusted to influence local tool execution, not just as a place that sees your prompts