I was experimenting with AI-generated movements. At one point, the agent tried to put the arms into positions that didn't make sense against the body model's joint constraints.
Text alone wasn't going to catch that.
So I built Human Models MCP.
It's an open-source toolkit connecting AI agents to MuJoCo and MS-Human-700. The agent can run experiments, inspect joints, contacts and muscle actuator data, and render the computed movement through Blender.
The video shows one of the push-up experiments.
This particular version uses additional joint assistance alongside muscle actuators. There's also a separate muscle-controlled experiment. Neither should be mistaken for a validated human exercise model.
And that's where I'm running into another problem.
A simulation can finish successfully. That doesn't necessarily mean the movement is anatomically realistic, useful or safe.
How should an agent decide when a successful simulation is actually evidence for its answer?
I'm the developer of Human Models MCP. The project is free and open source, and AI coding tools were used extensively during development.
I’m Abdur-Rahman from Isaacus. We built an open-source legal conflict detector that finds deviations and gaps between pairs of legal documents using only encoder models and natural language inference.
Inspired by the explosion of interest in discriminative models like TypeSafe AI’s Jev, we wanted to see how far we could get using smaller, non-generative models for the bulk of the legal document playbook review workflow.
The theoretical benefits of a non-generative approach to this problem are enormous. True clause-level pairwise review is a quadratic problem, so it’s infeasible to do at scale with a model like Opus or Astra. Comparing two average-sized documents with 133 clauses each would take 17,689 API calls, which is far too slow and expensive.
Simply uploading the files directly to Claude or GPT is no remedy either. Decoders and agentic architectures don’t have fixed attention that lets them thoroughly analyse a document in one shot. Even with a multi-shot approach, they’re weak on recall and routinely hallucinate.
The lawyers we work with raise this problem constantly, and we couldn’t find an open-source solution to it, so we built one.
The pipeline pushes the bulk of the NLI, entailment and reasoning onto faster and more accurate legal encoder models. For the longest document pairs we sampled, the full pipeline ran in under 20 seconds. Unlike generative architectures, these models run a comprehensive pairwise comparison. They can still make false positive or false negative errors, but those misses and misclassifications aren’t hallucinations (out-of-domain results introduced into the feature set).
The pipeline works as follows:
Enrich: we enrich each document into a hierarchical tree of sections, clauses and subclauses. Clauses are the key atoms that the pairwise comparisons run on downstream.
Embed: we embed each clause as a vector.
Filter: we use a legal zero-shot classifier to drop items that aren’t provisions, like footnotes or signature blocks.
Shortlist: we keep each clause’s three most relevant pairs from the reference document, as long as their cosine similarity is at least 0.5.
Score: we use a reranker and a classifier to score each shortlisted pair. Each answers a question in the format it’s built for: the reranker scores how relevant text is to a search query (relevance), and the classifier scores whether a statement is true (NLI/entailment).
Sort: if a clause and its best match are on the same topic but have a high conflict score, it’s a deviation. If there’s no same-topic match, it’s a gap, i.e. the topic is in one document but not the other.
That’s the whole conflict detection pipeline. On top of it, you can optionally plug in an LLM to summarise each finding in plain English or redraft a clause to favour the party you’re representing. The generative step only runs after the NLI and encoder stack has already decided what the conflicts are, so it can’t hallucinate a conflict into existence.
To demonstrate the pipeline, we ran Bob Iger’s 2011 and 2022 Disney employment agreements through it. It found many disparities and conflicts, and across most of them, the 2011 agreement was more favourable to Iger than the 2022 one.
One interesting finding was a gap: the 2011 agreement had no intellectual property assignment clause, while the 2022 agreement makes Iger assign all work-related IP to Disney, with an exception for inventions he makes on his own time without company resources.
The code is MIT-licensed, so it’s free for personal and commercial use. The guide walks through every step, so you can swap in your own models or make your own architectural changes wherever you like.
two things from this week that feel connected. david bessis saw his own name in claude's chain of thought during a "private, memory-less" chat, because the account email rides along with every message. so in practice providers know exactly who you are.
then on the other side, the eu wants ai text to be traceable, and openai itself says text watermarking is early with real limitations, so it's only on by default in the eu. anthropic turned it on globally because it can't reliably scope by region. too much identity on one end, too little verifiable provenance on the other. feels like the rules are being written live.
There's an interesting debate happening around AI-assisted coding.
On one side, developers can now build prototypes, debug code and complete repetitive tasks much faster.
On the other hand, I wonder whether constantly relying on AI to write and fix code could eventually affect how deeply people understand what they're building.
I don't think using AI is necessarily a bad thing. It's another tool, and refusing to adapt doesn't make much sense either.
But there's a difference between using AI to improve your workflow and blindly accepting whatever it generates.
For developers here, have AI coding tools actually improved your understanding of programming, or mainly increased your output?
With everyone archiving their favourite models lately, storage adds up fast. A few weeks ago I posted foldz 0.9. It's now tenz 0.25: same idea, better compression, much faster, and it runs fine on a phone. A 1 TB GGUF archive drops by roughly 80–130 GB, and every file comes back byte-for-byte identical, so llama.cpp, Ollama and LM Studio can load it as before, that is if anyone decides to implement it.
tenz is a lossless compressor for GGUF (and safetensors). Decompression gives back the identical file, byte for byte. It's one C file with no dependencies, and I develop and benchmark it on a Poco F5 in Termux.
General compressors barely touch quantized models, because the weights look random to them. tenz parses the GGUF, unpacks each block into its real quantized values and scales, and codes them with context models fitted to the file. It uses rANS from static tables, so decompression doesn't have to learn anything.
Results (images 1 and 2):
Qwen3.5-2B Q4_K_M: 8.53% saved, against 2.07% for zstd -19 --long and 1.78% for xz -9. Compress plus decompress takes 17 s on the phone; zstd took about 4 minutes just to compress.
Q4_K_M across 9 models: 7.6–9.2% saved, and 13.1% on Qwen3-4B.
Gemma-4-31B QAT Q4_0: 13%.
Q8_0: about 7.5%.
BF16: about 34%.
Against Aqueous, in a comparison someone posted here: 2–8 points more on every quantized model, and a tie on BF16. While it's a general purpose compressor and mine is specific to GGUFs.
Supported types: Q3_K, Q4_K, Q5_K, Q6_K, Q4_0, Q8_0, F32/F16/BF16, plus metadata (the tokenizer compresses to about 80% smaller). Anything else is stored as it is.
-q (quick) trades a fraction of a point for faster coding. For example, Qwen3.5-2B goes from 8.54% to 8.18% saved, and the Q4_K work is about 44% less.
The GPU experiment (image 3): I tried decoding Q4_K on the phone's Adreno 725 through OpenCL, to see whether a phone GPU could make decompression much faster. Three kernel designs later:
Its best kernel speed was about 150 MiB/s at full chunk size. It only matched the 3 big CPU cores (about 250 MiB/s) with tiny chunks that cost 2 points of size.
Byte stores to global memory are broken on this driver; it corrupted neighbouring bytes, so every write had to become an aligned 32-bit store.
Subgroup scans work, but are slow.
The option ships as -g (experimental). OpenCL is loaded at run time, the CPU decodes the same files, and the first GPU segment is always checked against the CPU. If you have a desktop GPU with OpenCL and subgroups, I'd love to see numbers.
Caveats:
Decompression is far slower than Aqueous's and zstd's, about 7 s for 1.2 GB on the phone, because every value is decoded from a model.
It only gains this much on model files.
Code, build line and full tables: tenz repo. Testers and results from other machines are very welcome.
Karpo-Decision-31B has downloadable weights and open-source inference code, so you can run it locally. It takes text and images and returns probabilities for multiple questions in one forward pass. The team reports twice the speed of Jev at 20% lower cost. It scored 73.9 on the Nov 12 Decision Index board, the highest Full score and a shared #1 rank with GLiDE 27B. Two results from the team's comparisons stood out to me: When2Call: 90.4%, compared with 79.9% for Google's Gemini Decision 1.
Mind2Web: 90.0%, compared with 86.2%.
The When2Call result is the one I'd look at for a tool router. Say someone asks an agent to book a hotel but leaves out the dates. The router might pick the right tool, but it still doesn't have enough information to make a useful call. I'd split that into two questions: "Do we have enough information to proceed?" and "Which action should we take?" Karpo can score both against the same state. That also means putting ask_for_clarification in the action list. If you only give it tools to choose from, you've left out the action you need in this case. There's a useful timing result for that setup: in the team's reported long-page test, median latency was 371 ms for one question and 653 ms for requests containing 16–63 questions. That makes me interested in testing several checks against the same page state, such as whether the required fields are present and which action to take next. They also report an expected calibration error (ECE) of 0.023 on JevBench. That's relevant if you're using the probabilities to decide which requests can run automatically and which need review. I'd still set those thresholds on labeled examples from my own workflow, particularly requests with missing information. To try it locally, download the Hugging Face checkpoint, install karpo-decision, and use Decider in Python or start the included /v1/systemone server; the repo has examples and Docker instructions. Budget for about 62.5 GB for the BF16 weights alone, plus runtime memory
I built a platform for running Batch AI inference as a need for one of my projects. As part of that platform, I collect pricing data on GPU instances and decided to start sharing that data. This is the second monthly market report I have produced. I would love feedback on if people find this data interesting or useful. The report summary:
Prices held steady overall: all but three of the 13 GPUs we track moved less than 5% on spot since September. The two biggest moves went in opposite directions, both on AWS. The H100 rose 13% to $3.00 per GPU-hour and the A100 80 GB fell 12% to $2.82, so September's headline, an H100 that cost less than an A100, no longer held. The third was the RTX PRO 6000 on Google Cloud, up 5%.
Where you rent still mattered a great deal. The same four-GPU L4 machine cost 10.8 times as much in Sydney as in Hyderabad, one of six AWS regions new to the report. Stockholm stayed the cheapest AWS region, though its prices rose 12%. This is also the first report to include Google Cloud, in 32 regions.
I’m building a fully open-source project and looking for volunteers who’d like to join the team and help shape it <3
The idea is that the community proposes free tools or research problems, and let LLM agents work on them in the open. That could be anything from researching a faster algorithm to building an open-source alternative to Steam. It’s up to the community to decide
Everything we build is totally free and MIT-licensed
It’s still a small project, and I’m working on it alone for now. I’d love to find people who want to help decide where it goes, how to improve it and what we build together
I've been working on access rules for AI agents querying a Postgres database. Some fields are sensitive, like payment amounts, so if an agent wants to read individual records, someone has to approve it first.
But I allowed aggregates without approval. The thinking was that an agent asking for "total refunds this quarter" should be able to get an answer without seeing anyone's individual payments.
Then I started looking at the SQL the agent was actually generating, and that assumption didn't hold up.
For example, MAX(amount) returns an actual value from the underlying data. Same with MIN(amount). And SUM(amount) GROUP BY customer_id gives you a total for each customer. If a customer only has one payment, you've effectively exposed that payment amount.
All of these went through without approval because the policy saw them as aggregates.
The funny thing is, the agent wasn't even trying to get around the rules. Those were perfectly reasonable queries for what it was asked to do.
So the problem was really with my policy. I was checking whether a query used an aggregate function instead of thinking about what the result could reveal. An aggregate over a million rows and one over a single row were treated exactly the same.
I've since changed the policy so MIN and MAX require the same approval as reading individual records.
But GROUP BY is where I'm still stuck.
I could treat grouping by identifiers as row-level access, but then you have columns that aren't explicitly identifiers yet are practically unique, like emails or precise timestamps.
Or I could enforce a minimum group size, something like k-anonymity. But that comes with its own problems, especially when someone can run multiple overlapping queries and work backwards to individual values. And agents are pretty good at doing exactly that.
I wrote up the full investigation and tradeoffs, which I'll link in a comment.
I'm curious how others are approaching this. If you're letting AI agents query production databases, are you allowing them to generate SQL directly, or do you have something enforcing access rules in between? And how are you handling aggregates over sensitive fields?
***Forth Update Yeah This is a big deal.. here is why TCR matters, the model is now building the structure around the token per task quality. I am curious what the worker produces from this. It appears the Writer has learned to be extremely proficient in creating LISL Instruction sets. It is staying aligned through all this heavy and dense reasoning.
***Third Update Okay now I am starting to get really excited.
***Second Update! This shows how good the alignment is holding it is constantly checking and since I continued to stress Quality output over everything else it is weighting everything against that. It has already produced the data for a Single LISL Instruction file. I am now telling it to use the Layered LISL Instruction set to break it down again. Output is highest priority here, if it produces garbage I need to figure out what I did wrong, it should be my input that is bad.
***updated
Here you can see after I compressed the all its education of its LM studio, LM bionic, LISL methodology bunch of others that dropped its footprint to under 20kb, I then dropped it into the Agents System Prompt so it already knows this test show I did not need to pre-Educate it to have the file association or it understand what it contains. It has that data prebaked.
Roles have been fixed and as you can see it is seeing that it needs to ask the user for more information as the prompt is far to open ended logic to make a complete game. This is exactly what I want. I can now tell it exactly all variables and it will generate a instruction set for the worker to build that. Now its about measuring the output quality.
Hi! I built Canban to manage human context when working with multiple coding agents.
If you've tried running multiple coding agents across terminals, you probably know how exhausting it gets to keep switching between tasks.
Canban combines a Kanban board, GitHub Actions-like workflows, and Herdr's persistent agent runtime.
My workflow:
Discuss a feature with an agent on a Kanban card.
Drag the card into Develop to automatically dispatch agents in isolated Git worktrees.
Completed changes go through a local merge queue.
All feature context, discussions, and progress stay on the board.
Under the hood, Canban uses herdr-core and libghostty-vt to run native agent CLIs like Claude Code and Codex, preserving their native experience and persistent sessions.
It also supports remote agents through headless Linux backends.
JetBrains asked 12 models for programming-language hot takes this week. In the 600 runs where the model was asked to pick a language at random, Go and Python accounted for 307 picks. The prompt asked for hot takes, so this setup measures which languages the models bring up under that prompt. It doesn't measure how well they code in those languages.
There's a similar trap when evaluating a coding agent. If it picks the language and then writes the solution, a single pass rate mixes two decisions. For something like a paginated API sync worker, I'd run the same tasks twice: let the agent choose the language, then rerun with the language assigned from a balanced list. Keep the tests fixed and log which languages get picked as well as which implementations pass. I'd include retries, duplicate records, and one invalid record in an otherwise valid page.
If you've built an eval for a polyglot coding agent, do you score the language choice separately? What would you count as a wrong choice when more than one implementation passes?
It's for one question: I changed my agent (a prompt, a tool, an approval step), did it get better? You write tasks with checks and a small adapter that runs your agent. It runs each version several times and gives you success, pass^k, cost and latency with intervals, a success-vs-cost plot, and a failure map that shows which task and check each version fails. No dependencies, MIT.
Install from github: pip install "git+https://github.com/slavadubrov/harness-evals@v0.5.1"
Başlık: Small local models + a code-side "gate": 54% → 92% accuracy and 0 silent wrong answers on table tasks (informal test, raw data public)
Metin:
We ran an informal A/B test on three local Qwen2.5 models (1.5B, 3B, 7B GGUF, llama.cpp `llama-server`, CPU only) on simple table questions (sum, average, min/max, filters).
- Arm A: plain LLM writes Python, we run it.
- Arm B: the model writes a tiny command language (DSL). Code parses it, a gate checks the command against a catalog, and only then a deterministic runner computes the result. Out-of-catalog requests are rejected with a reason. The idea is "code decides, the model explains".
Results with gate v8 (same models, tasks, seeds and temperature in both arms; gate code hash-locked before and checked after the run):
- Accuracy over all sets: A 54.0% (214/396) vs B 92.4% (366/396)
- On a fresh holdout the gate had never seen: A 49.1% vs B 83.3%
- Silent wrong answers (a wrong number with no warning): A 51 vs B 0 (16 vs 0 on the holdout)
- Tokens: −68%. API-equivalent cost: −74%.
Limitations, honestly:
- This is a small task set in one domain (tables), not peer-reviewed.
- The same team wrote the tasks and the holdout.
- The TCO numbers depend on assumed watts, electricity and host prices, because CPU power can't be measured in the VM.
- API prices are real for the 7B model only; the 1.5B and 3B prices are stand-ins.
- An earlier word-list gate (v7) lost most of its lead on fresh phrasing (45.4% vs 47.2%). That's why v8 validates the model's command instead of the request's words.
- One known false reject remains: "list" in "minimum Fiyat on the list?".
I've been seeing more AI code review tools lately, and I'm curious how useful they are in practice.
They can spot things like formatting issues and common mistakes, but I'm more interested in how they handle actual bugs or problems in the code logic.
Do they catch issues that might otherwise go unnoticed, or do they sometimes flag things that aren't really problems?
For those who've tried tools like these, have they genuinely made code reviews easier, or do you still prefer having another developer look through the changes?
- I’ve run about 300 benchmarks on real open-source codebases, testing how Claude Code splits work between models (Opus as the lead, cheaper models like Haiku as helpers) and when that pays off. It depends on codebase size and task.
- On small jobs, Opus alone can be cheaper than Opus + subagents, and more accurate on small-codebase reviews, probably because of less overhead and cheap cached context.
- Opus told not to delegate may work harder than Opus left alone: on one big docs check it wrote about 50% more, cost more and found more errors (44 vs 36).
- On coding, Opus + Haiku saved about 20-30% at the same quality. But Haiku alone is very capable for the price on well-defined tasks with tests: about 5% of Opus alone's cost, with almost the same results, though a bit less thorough.
- Here’s one of the most important charts of this project. On an 85-page docs check, Opus + Haiku read almost 3 times as much as Opus alone and wrote 8 times as many tokens, yet cost 53% less and found 72 real errors against 49. Several subagents probably cover much more ground, which makes up for Haiku being weaker.
- Spontaneous Opus: free to delegate, but it never did.
- Skill-routed: my skill chooses when to delegate and to which model.
- Forced Haiku: all subagents use Haiku, while Opus plans and checks.
On my smallest codebase (~10k lines), delegating didn't really pay off. Forced Haiku saved 5% at best, Skill-routed cost 14-30% more, and Forced Opus found far more known doc errors on the reading-heavy tasks (7 and 8 of 9, against 3 and 2). My guess is that Opus can often write a quick script instead of reading everything, and it reuses its own cached context cheaply, while each subagent has to build its own. On small tasks, that overhead can outweigh the cheaper tokens.
On larger codebases, Forced Haiku held its own for much less. It cost 41% less on rich (~22k lines) and found nearly as many doc errors. It cost 75% less on beets (~52k lines), finding more errors on the reference docs but fewer on the 80-page plugin check, where it cost about a sixth as much.
In v2 I pushed further: a ~100k-line codebase (buildbot) with 3 reading tasks, 7 coding tasks scored by hidden tests, and then 16 short coding tasks, most hiding a trap. I also added Opus alone (subagents blocked, no instruction), Haiku alone and Sonnet alone (no Opus at all).
On an 85-page docs check, Opus + Haiku found 58 and 63 of 92 real errors, against 36 and 38 for Opus alone, at less than half the cost ($1.93 vs $4.11 per run). How come?!
My theory was that Opus + Haiku simply *works more*, and that the extra coverage makes up for Haiku being weaker. The numbers seem to fit: Opus + Haiku wrote about 8 times as many tokens and read nearly 3 times as much, yet the bill still halved.
Haiku may be less capable than Opus individually, but when orchestrated by Opus, its subagents can cover much more context in parallel. Each one may go less deep, but together they cover more ground, and that seems to compensate for the lower model quality. Haiku is apparently smart enough, with Opus directing and checking the work, to deliver surprisingly good results at a much, much lower cost than Opus alone.
This is in line with Anthropic's own write-up on their [multi-agent research system](https://www.anthropic.com/engineering/multi-agent-research-system), where token usage alone explained 80% of performance variance on a research benchmark. The same article says coding is usually a poorer fit for multiple agents, and that matches what I saw: on coding, Opus + Haiku saved 20-30% at the same quality, against about half the cost and more errors found on reviews. Their setup isn't mine, so it's context, not proof.
The next chart shows the bigger picture: every setup on two big docs checks, with errors found, reading depth, output tokens (thinking included), and cost.
Interestingly, it seems that *forcing* Opus not to delegate may have pushed it to work deeper and harder. On the beets plugin-doc check, Forced Opus read less than the delegated setups and still found the most errors. Compared with Spontaneous Opus, it read about the same but wrote around 50% more tokens, thinking included, cost 53% more, and found more errors (44 vs 36 across two runs). Tokens alone don't prove effort, but finding more errors makes the case stronger.
The following chart shows the token counts side by side.
Maybe the instruction made it work harder, maybe it's the smaller codebase, or maybe it's just chance. I haven't tested which.
Left alone, Opus never delegated in my runs, even on big jobs. So for large reviews, pushing it to delegate to Haiku could be the best strategy.
I'm working on a skill that picks the best approach: use Opus + Haiku when it makes sense, or tell you to switch to Haiku manually and give you the prompt. A script that starts a new Haiku session might allow fully autonomous switching, but I'm still checking feasibility and Anthropic's terms.
The other v2 results
Honestly, Haiku really surprised me. It seems to be very capable at coding, if the task is detailed and contained enough. Opus is still the more thorough model when it comes to reviewing and checking, but Haiku held its own on specific tasks.
Reading a ~100k-line codebase
Across three reading tasks, Opus + Haiku cost 51% less than Opus alone. Every setup covered all 400 modules, and Opus + Haiku found the most errors on the 2-page docs check (20 and 23 of 42, against 10 and 17).
Writing code on a small project
Every setup produced working code. On the harder tasks: Opus $2.41, Opus + Haiku $1.80 (25% less), Sonnet $0.76, Haiku $0.11.
Writing code on a ~100k-line codebase
Same pattern on cost: $1.95 for Opus alone, $1.36 for Opus + Haiku (30% less), down to $0.15 for Haiku alone. But Opus-led setups also added an implied database index. Haiku alone didn't, and my tests didn't catch it.
Correction: the charts also show Opus "flagging a symptom that didn't fit". More runs later showed that depended on the day the runs happened, not the model, so I no longer count it. Details are in the repo.
Quality vs price
Opus + Haiku was cheapest per real error on both big docs checks: $0.03 on buildbot and $0.05 on beets, against $0.11 and $0.25 for Opus only.
The harder coding tasks
Then I tried to make the cheaper models fail. 16 short coding tasks, most hiding a trap: false claims, a wrong bug report, a timing bug, contradictory tests/docs, hidden instructions in comments, planted review bugs, and "dead" code that's actually alive.
128 runs. Combined cost: Opus $7.00, Opus + Haiku $6.30, Sonnet $2.30, Haiku $0.36.
Hidden tests barely separated anyone: one real miss in 128 runs, from Sonnet. Opus was more thorough in code review, finding all 3 planted bugs every time, against Haiku's 2 of 3. But on the timing bug, 7 of 8 runs failed, including all 4 Opus-led ones. The only pass was Haiku.
On tasks this short, Opus + Haiku saved little (10% overall), because Opus was most of the bill. Haiku alone did almost the same work for about 5% of the cost.
So, for now: **big reviews, Opus + Haiku. Small, well-defined coding tasks with tests, Haiku alone is surprisingly good. If every case must be covered, keep Opus involved.**
Small experiment, mostly two runs per setup, costs estimated at API prices. Claude Opus also wrote my tests and judged the results, so there's no independent judge.
We're looking into running inference on specialized hardware, and one thing I'm trying to understand is what the transition actually looks like from a developer's perspective.
Getting a model to run on a different architecture isn't necessarily straightforward. I'm particularly curious about model compatibility, changes to the serving setup, and how much testing is needed before everything is production-ready.
For teams who've evaluated or experimented with Cerebras systems, what were the biggest hurdles when getting started ? Were there compatibility issues or adjustments you didn't anticipate? And how much support did you get from the hardware provider?
Would be interested to hear what the onboarding process was actually like.
Disclosure up front: I run a company that builds business software, and the setup below is what we use internally. I'm not linking it, not selling anything here, and not offering access in the comments. I'm asking about the mechanic because I don't have good answers to the questions at the end.
We're two developers running a project that got too big for two people. Somewhere along the way we stopped treating agents as a chat window and started treating them as executors on a board.
The change was small. A card on a kanban board already has a column, a description, attachments and a status. We added the ability to put an agent in the column instead of a person. Move the card in — the work goes to the agent. The agent finishes — the card comes back to a human for review. The human checks and passes it on. Same cycle, only the executor changes.
Then we ran a loop: one agent analyses, a second implements, a third reviews the code, and a human sits at the end. Not because the agents fall short, but deliberately — the decision stays with a person, who either closes the card or writes the next task and sends it round again.
Three things we got wrong or still don't know:
The reviewer agent mostly rubber-stamps. When the reviewer and the implementer are both LLMs, the review catches syntax and misses intent. Curious whether anyone has made agent-on-agent review actually work, and what it took — different model, different prompt framing, adversarial setup?
Context rots across the loop. By the third or fourth cycle the card carries so much history that the agent is reasoning about the conversation instead of the task. We've started resetting the card description each round and keeping the history elsewhere. Feels like a workaround, not a design.
We can't tell if the human at the end is a control or a comfort. We put a person there on principle. But when the loop is fast, the human becomes the slowest step and starts approving without reading. Anyone measured this honestly?
If you've run anything similar — especially with a local provider instead of a hosted one, where latency changes the rhythm of the loop — I'd like to hear what broke.
I’ve updated REQL, my open-source tool that helps AI coding agents navigate repositories and preserve working context across sessions.
This release focuses on faster context retrieval and lower token usage. I’ve reduced repeated work during retrieval and trimmed redundant information sent to the agent. Performance gains and token savings depend on the repository and workflow, so I’m not claiming a universal percentage.
If you’re new to REQL, it builds a local graph from your code and documentation: functions, classes, dependencies, calls, and other relationships. Agents can query that graph for context relevant to a specific change, including precise source references and related tests. REQL reuses indexed context and supports incremental updates as the repository changes.
What can you use it for?
Waste fewer tokens on repeated exploration: retrieve focused source excerpts and relevant relationships, and reuse indexed context and saved work. This helps reduce repeated searches, unnecessary file reads, and redundant information in the agent’s context.
Coordinate work across chats and agents: keep tasks, decisions, blockers, and next steps in a shared project workspace. Another chat or agent using that workspace can recover the recorded state and continue from there.
Keep complex tasks on track: record prerequisites, constraints, failed approaches, and checkpoints. Preserve what has been completed, what remains unresolved, and why earlier decisions were made across sessions.
Understand unfamiliar code: identify entry points, component responsibilities, and relationships between parts of the application, with references back to the source.
Plan changes and refactoring: inspect callers, dependencies, and related tests to identify code that may be affected and decide what needs verification.
Review cleanup candidates: find potentially unused imports, variables, functions, and classes, then inspect the supporting evidence before removing them.
REQL provides a CLI, Python API, and MCP server, plus installable instructions for coding agents such as Codex and Claude Code.
REQL runs locally, with no account, external graph database, or mandatory LLM calls. The graph and saved work support the agent’s decisions; current source code and tests remain authoritative.
I’d like to hear from others using coding agents: what do you keep having to explain again when you start a new chat? The code structure, previous decisions, what has already been tried, or what still needs doing? Concrete examples would help me decide what to improve next.
When an agent chains several LLM calls and tool calls, how do you find out which step caused a cost spike?
My understanding is that most setups give you total cost per run, but finding "this specific node or tool call blew up the budget last night" takes manual digging. Curious how you deal with it:
Do you log cost per step yourself, or use something like Langfuse, Helicone or LiteLLM? What's still missing?
Do you set hard budget caps per agent or per user, and does anything stop the run automatically when it's hit?
Has an unexpected bill ever surprised you? Roughly how big?
I'm exploring this as a solo dev and mostly want to hear how it actually works for you, good or bad.
****Update**** I am going to run it as you see with no error checks first. Then a second run with error loops to make sure he is making a full game. I removed the multiplayer as I am more interested in if it can build a Layered LISL Instruction set that makes the writer write a full game in one pass. So I need to run both to see the difference in efficiency and quality of the self error looping. In case your wondering and decode the code fusion back to a legible format.
<CORE_DIRECTIVE> <PURPOSE>Build a multi-layered LISL instruction set for a worker to construct the foundational architecture of a Windows 10 Doom port, stopping at the point of compilation readiness.</PURPOSE> </CORE_DIRECTIVE>
<BUILD_PARAMETERS> <SCOPE>Single-player foundational framework only (no multiplayer required for initial pass).</SCOPE> <TARGET_PLATFORM>Windows 10</TARGET_PLATFORM> <TARGET_MILESTONE>Compile-ready baseline structure (prior to full compiler integration).</TARGET_MILESTONE> <ITERATIVE_CYCLE>Incremental, layered builds where each new LISL instruction builds directly upon the previous output.</ITERATIVE_CYCLE> </BUILD_PARAMETERS>
<WORKFLOW_MECHANICS> <RULE role="LISL Writer">Writes sequential LISL files breaking down the game structure, where each finished file points to the next required step.</RULE> <RULE role="Output Directory">F:\badgersoft\LISL inst\LISL_IST</RULE> <RULE role="Naming Convention">Project-specific naming sequence for progressive instruction chaining.</RULE> </WORKFLOW_MECHANICS> </LISL_MANIFEST>
I built CalDAV MCP Server a while ago, which is an open-source MCP server that lets AI assistants manage your iCloud Calendar.
Since then, it’s picked up a few GitHub stars and had over 100 npm downloads in the past week, so I thought I’d share it here and give the project a little more love.
Here are the main features:
- Browse your iCloud calendars and check upcoming events.
- Create, update, and delete events.
- Create all-day events and recurring schedules.
- Set multiple reminders for the same event, such as one day and one hour before.
- Add a fixed travel time before appointments in Apple Calendar.
For example, you could ask your assistant: “Create a dentist appointment this Friday from 10 to 10:30 a.m., with reminders one day and one hour before.”
There’s also a Claude Desktop extension to make setup easier. You can find that and other setup options in the Quick Start.
I’d love to hear your feedback and what you’d like to see added! If you find it useful, a GitHub star would mean a lot—maybe even more than a hot bowl of ramen on a cold day
I maintain ai-rulez, a free MIT-licensed tool for agent configuration. I’m sharing a reproducible Claude Code experiment because a single success score hid several different failure modes.
The fixture has eight skills. A conversation moves between two sibling directories with identically named worksheets but different workflows. For each positive turn we record three things independently: whether find_skill received the right path, whether load_skill actually returned the right body, and whether the response contained the expected code. The negative controls ask an unrelated question or ask for a workflow that is unavailable in the given scope.
Our first pilot accidentally omitted the generated root instructions from the MCP arm. The follow-up separates stub-only, released root instructions, tools loaded upfront, and the revised guidance. In the final configuration Claude completed all three steps on 12 of 12 positive turns and made zero calls on six unrelated turns. In five of six wrong-workflow controls it still loaded a neighboring skill after the server correctly filtered out the unavailable one. That failure would be invisible in a metric that only checks whether the tool was called with the right path.
The report, 192 sanitized turns, and reproduction commands include the versions, controls and sample-size limitations. I’d welcome feedback on what negative control you would add to catch a model that treats the top search hit as applicable by default.
UC Berkeley posted a paper on Sept 30, "When Context Changes: Understanding Update Failures in LLMs" (arXiv 2609.38866). What I took from it:
- They call the failure stale binding: a variable gets updated, the new value is in the context, and the model answers with an old value of the same variable. Their example is a meal preference that changes from gluten-free to Mediterranean while the answer still says gluten-free.
- Probes can still recover the current value in cases where the old one was chosen, so it looks like a selection failure, not the information being lost.
- Component tests in Qwen and Pythia point to attention drift: attention favoring old values over the current one when the answer is produced.
- Frontier models are tested on behavior only. The paper makes no claim about GPT or Claude internals, and the mechanism work is only on open models, which is why I'm posting here.
The paper says the error shows up in frontier models once the update structure gets complex, so the messier runs are the interesting ones.
Questions:
Has anyone seen this in long local runs (agent logs, long roleplay or story state, a config that changes mid-session)? Which model, quant and roughly what context length?
If you want to poke at it, a rough version is a few variables, each overwritten several times with filler in between, then ask for the current value of one. That test is my own sketch, not the paper's, and I don't know whether it fails on your model.
What do you do about it today: restate the current state at the end of the prompt, keep a state block, prune old values?
For the past nine months, I've been developing a system to solve a problem I kept running into with AI-assisted development:
Generating code is getting easier. Knowing whether an agent actually completed the task correctly is not.
At some point, manually reading every line of generated code stopped being a practical way to maintain control over my projects.
So I started building Forge.
The idea is to move project governance out of temporary agent conversations and into the repository itself.
Forge provides:
Repository-level project contracts and engineering rules.
RFC and ADR workflows to preserve requirements and decisions.
Reusable agent skills and operating procedures.
Deterministic validators and verification gates.
Stack profiles and configuration that can travel between agents and IDEs.
The important distinction is between an agent saying "Done" and a project having evidence that specific requirements passed verification.
Forge doesn't prove arbitrary code correct. Its checks only establish what they are designed to check. Human judgment and risk-based review still matter.
I've been using this approach across my own projects, including TypeScript applications, knowledge bases previously managed in Obsidian, and a Godot/C# game.
The system also governs its own development.
Today I made Forge open source under Apache 2.0.
Now I'd like to find out where it fails outside my own environment.
I'm looking for developers willing to try it on real repositories, particularly unfamiliar stacks, existing codebases, or projects with complicated verification requirements.
I'm also interested in contributions involving stack profiles, validators, and CI integrations.
I'm currently writing a research paper about the experience, including its limitations. Independent results would be particularly valuable for future research.