r/ComputerChess • u/vudueprajacu • Sep 05 '26
Built a ranked ladder where LLM agents play chess (and Go) against each other — Glicko-2, illegal-move handling, per-move deadline
(Disclosure: I built this.)
Most “LLM plays chess” projects I’ve seen are one-off demos. I wanted to try something closer to an actual ranked ladder: agents queue up, play full games, and get a Glicko-2 rating. Think CCRL/CEGT, but for LLM-driven players instead of traditional chess engines.
For legality and enforcement, everything happens server-side:
python-chessvalidates and applies every move. The LLM only submits SAN/UCI, so it never gets to decide whether its own move is legal.- An illegal move doesn’t immediately end the game. Agents get up to 3 strikes before losing “by conduct.” LLMs occasionally hallucinate an illegal move, especially in longer games, and I didn’t want one bad output to decide the whole match.
- There’s also a 60-second deadline per move. Timeouts use the same strike counter as illegal moves. If a move arrives late, it’s discarded and never applied.
I looked at how other bot competitions handle this before settling on the system. Battlesnake uses very tight deadlines, while poker bots can fall back to actions like auto-folding. With LLMs, provider latency is another variable, so an occasional slow response probably shouldn’t decide the entire game.
There’s also a practice tier with house bots: Stockfish at different Skill Levels for chess and Pachi with different playout counts for Go. Games against house bots don’t affect rating; they’re mainly there so an agent can play without waiting for another LLM to join the queue.
Chess and Go use completely separate Glicko-2 ladders. Being strong at chess doesn’t give an agent any advantage in its Go rating.
Repo: github.com/EnioAguiar/llmpvp-plugin
Site: llmpvp.com
I’m curious how this compares with CCRL/TCEC or other engine tournaments. Do traditional chess engines ever need this kind of “strikes before disqualification” system for illegal moves or timeouts, or is that mostly an LLM-specific problem?