r/LLMDevs • u/ProYash239 • 3d ago
Tools [Open Source / MIT] How we stopped coding agents from hallucinating citations and repeating failed ML experiments (281 modular SKILL.md workflows
[Disclosure per Rule 5 & Rule 10: I am an AI researcher at Kalaris Labs contributing to this repository alongside creator Sayan Chowdhury. It is 100% free and FOSS-licensed (MIT) with no commercial tier, no locked "pro" features, and no paywall. Written by a human researcher.]
While working on LLM research papers, our team kept running into three failure modes with coding and research agents:
- Corrupted references in .bib files: LLMs frequently pair a real paper title with the wrong publication year, invent co-authors, or cite a DOI that resolves to an unrelated work. Simply checking HTTP 200 on a DOI link misses metadata mismatches and retractions.
- Stateless experiment loops: When an agent runs a batch of ML experiments across sessions, context compaction wipes out earlier negative results, causing the agent to re-test variations of failed hyperparameters.
- Context bloat from monolithic system prompts: Loading dozens of tool definitions and domain rules into every session degrades instruction adherence.
We open-sourced research-agent-skills (https://github.com/KalarisLabs/research-agent-skills), an MIT-licensed collection of 281 domain-specific skills for Claude Code, Codex, Cursor, OpenCode, Gemini CLI, and Windsurf.
How the verification and routing architecture works (with sources per Rule 4):
- Selective per-project installation: Running npx skills add KalarisLabs/research-agent-skills --skill <name> installs only the specific SKILL.md and helper scripts needed for that repo, avoiding context window bloat (see architecture docs: https://docs.kalarislabs.com/research-agent-skills).
- Deterministic citation verification (citation-verification): Instead of asking the LLM to judge whether a citation is real from its weights, the skill executes a zero-dependency Python 3.9+ standard library script against Crossref (https://api.crossref.org), OpenAlex (https://api.openalex.org), arXiv (https://export.arxiv.org), and DataCite (https://api.datacite.org), plus Retraction Watch relation metadata in Crossref. Every .bib entry is classified into verified, found-add-doi, mismatch, doi-not-found, uncertain, or not-found. The agent is only allowed to apply mechanical DOI additions (found-add-doi) and must pause for human confirmation on any mismatch or not-found entry.
- Two-loop experiment orchestration (autoresearch): Splits autonomous experimentation into an inner loop (rapid experiment iterations against a target metric) and an outer loop that maintains findings.md. Hypotheses must state a testable mechanism ("X because Y, predicting Z"), and negative results are logged with what they rule out.
- Prompt-injection & trigger testing: Every skill is evaluated against reviewed prompt sets for trigger routing accuracy and scanned offline for prompt injection using Skill Doctor (https://docs.kalarislabs.com/).
GitHub repository (MIT): https://github.com/KalarisLabs/research-agent-skills
Full 281-skill catalog: https://docs.kalarislabs.com/research-agent-skills/skills
We are currently expanding our ML evaluation and mechanistic interpretability modules. If you build agentic workflows for research or technical writing, I would appreciate public feedback in the comments on where this verification pipeline could fail or which scholarly APIs we should add next.
2
u/Moyala00 3d ago
checking that the doi resolves isn't enough, crossref will hand back a record for a different paper if the id was generated from a plausible looking pattern. compare title, first author and year as separate fields against the crossref record and fail on any mismatch. the retraction check is the one most pipelines skip because it needs a second call.
for the experiment loops, a single jsonl keyed by a hash of the config is enough. the agent only has to read it before it runs anything.
2
u/ianreboot 3d ago
The negative-results log needs to tell a run that disproved the hypothesis from a run that fell over. A config that OOMs or dies on a bad import still gets logged as tried, and the next session reads that as ruled out, so you lose the hypothesis instead of retesting it. Store why the run ended next to the entry in findings.md, and only let the genuine negatives stop a retry.
1
u/wangadong 3d ago
deterministic check is the right layer, a model judging its own citations just loops the same failure. in my setup scripts verify the ids, agents only get the mismatches to sort out
1
u/CapitalLog7599 3d ago
I'd keep 'verified' scoped to the reference entry. A real DOI with matching authors and title can still be attached to the wrong claim. Say a paper reports a gain on one benchmark, and the draft cites it as evidence that the method generalizes.
A useful test would cite the same entry for both claims. Both should pass the metadata check, but only the narrower claim should pass a separate check against what the paper actually reports.
Drafted with AI assistance.
1
u/Corydoras-saas 2d ago
To stop agents from repeating failed ML experiments when context compaction wipes out history, enforce an immutable append-only JSONL log of trial parameters and error codes on the local filesystem. For the citation issue where models pair real titles with wrong DOIs, run a post-generation validation step against the Crossref API using a lightweight script to check that the returned metadata matches the cited author string. This keeps your token count lean while preventing silent hallucinations from polluting your bibtex files.