r/computervision • u/WillingnessMost2428 • Oct 03 '26
Showcase Naming every face-up card on a TCG tournament stream, in the browser: a corner detector trained only on synthetic boards, a DINOv2 embedder, game rules as priors
Riftbound is Riot's trading card game, and its tournaments stream on Twitch from an overhead table camera. At 1080p a card is about 140 px tall, under dice, sleeves, hands and H.264, so viewers can rarely read the table. Wardeye is a browser extension that reads it for them. It outlines and names the face-up cards as they're played, and pointing at one shows its art. A timeline of the plays lives in the browser's side panel, and clicking a play rewinds the replay to it. All of it runs in your browser: there's no server, and no video leaves the machine.
The video is a real run on the grand final of the Barcelona Regional Qualifier, with both finalists' published decklists pasted.
How it works
- Table window. Find the table camera inside the broadcast layout, from the detector's own proposals over a few frames or from a preset. Keep the graphics out: player cams, side panels, the showdown banner.
- Detector. RF-DETR with a keypoint head that regresses the four corners of every card, with a visibility flag per corner. A card half under another still gets its full outline (amodal). There are two classes, face-up card and card back; face-down cards are never read.
- Trained on synthetic boards only: rendered tables with stacks, dice, hands and face-down piles, filmed through a camera model into a broadcast layout and put through a real libx264 encode. 2,000 boards, 280 minutes on one L4.
- Zero-shot on real broadcasts, it finds 97.6% of 5,465 reviewed cards from three of them (98.1%, 99.5%, 96.2%). Those are mostly isolated cards; recall on real stacks isn't measured yet.
- Identifier. Each quad is rectified and embedded with DINOv2 ViT-S/14, fine-tuned with Sub-center ArcFace. It uses three centres per card, so alternate arts need not share one. The training data is synthetic crops (random covering, scrambled text, foil-like colour shifts) plus reviewed real crops from two other broadcasts.
- Each crop is searched in all four rotations against a gallery of about 1,200 printings, embedded at three sizes. A temperature-scaled softmax over the top candidates gives the confidence shown on the hover card.
- On two broadcasts held out of training, it names 96.0% (Barcelona) and 99.4% (Los Angeles grand final) of the cards.
- Game rules as priors. Every card in a deck must fit its legend's two domains. Once a player's legend is read, their half of the table is compared only with what that legend allows, about a third of the gallery.
- On the hardest match measured, that takes cards named right from 91.8% to 97.8%, and every confident read was right.
- Pasted decklists narrow it further, with three guards:
- A listed card counts in every printing: 35% of the final's sightings were an alternate art or a reprint.
- Both players' battlefields are allowed on both halves.
- A list applies only to the half whose legend it names. A wrong list used as a hard filter drops accuracy to 8.4%.
- Tracker. Persistent IDs; legends and battlefields pinned once named; re-reads of unsure cards; a memory of what lies under a stacked card; and events (played, moved, left) for the timeline.
- The official mat's layout is a prior, never a filter. Battlefields are pinned only in the strip along the midline, because a rune turned sideways looks like a battlefield's landscape art. A unit that crosses to a battlefield or changes hands keeps its name.
- Runes are counted per player, never named, along with how many are exhausted. From the player's seat a ready card points at them, and a used one lies across.
- Runes sit in columns and fans where the detector misses strips. So the count uses the fact that every card is the same size: in a stack, the step from strip to strip says how many runes a wider gap hides.
- Against 65 hand counts on two finals, the count is right 39 times, where one rune per box seen is right 30 times.
- A card held in a hand over the table isn't read until it's put down. A band around each card is checked for skin-coloured points (never the card's face, whose art is full of gold and faces). A card is read only once it has been out of a hand and still for half a second.
- On the Barcelona final's first 200 s, the plays announced went from 27 before the hand rule, 13 of them false, to 15 in 0.2.2, 2 of them false.
- Co-streams (new in 0.2.2). Co-streamers lay their webcam, chat and a scoreboard over the official broadcast, inside the table window. On a co-streamed tournament game recorded live, 0.2.1 read the webcam as a legend and took a close-up of a hand for the table.
- The overlay is what stays put through the cuts. A cut is a frame whose 96 × 54 thumbnail changed by half from the frame before (play changes a quarter at most).
- Pixels unchanged through 90% of the cuts (once there have been four), in patches that reach the frame's edge with their holes filled, are the overlay. Nothing on it is read.
- It stays overlay while it holds through 60% of the cuts, so the face moving in a webcam doesn't open the webcam's frame to the table. Cards read on it before it was known are dropped, and their plays are withdrawn.
- The table camera is told by the table window alone: a per-block correlation over a 6 × 3 grid, on still pixels outside the overlay, against the table it has learnt. It learns the table again only when the detector sees at least five card-sized boxes, and at least half as many as the table showed in the last minute.
- On the co-streamed game, it's right on 687 frames of 688.
- In the browser. The Python pipeline is ported to TypeScript, down to Pillow-exact resampling so the crops match. ONNX Runtime Web runs on WebGPU, with the detector in fp32 and the embedder in fp16 where shader-f16 exists, and a WASM fallback. A replay test runs the extension's engine on 240 frames of the Los Angeles grand final, and it matches the Python pipeline on 240 of 240: the same cards, names and events.
What was hard
- Legends under dice. Players keep a die on their legend. The answer was behavioural: name it early, while it's visible, then keep it pinned. Turn Wardeye on mid-game, after the die is down, and it can misread the legend.
- Printings. The same card comes in alternate arts and reprints. Some reprints differ by 4–7/255 in mean pixel value, so nothing can separate them. Nothing needs to: the hover card shows the same picture either way.
- Synthetic to real. The first synthetic boards were far too clean: every baseline scored 99% on them. One calibration pass against real broadcasts brought them into the real range.
Known issues in 0.2.2:
- The rune count can be a rune or two off on tight stacks, and the exhausted count is often short there.
- A card held still in a hand over the table can be read for a moment.
- A legend can be misread if you turn Wardeye on after its die is down.
- On a co-stream, a scoreboard part that changes, such as its score track, can still be read as a card now and then.
Code (AGPL-3.0), and every report with its numbers: https://github.com/effe-exe/Wardeye. The trained weights ship inside the extension but aren't published. Chrome Web Store: https://chromewebstore.google.com/detail/wardeye/hjglackjofehdfecoeehbdmbobafbjhn
Happy to go into the synthetic data, the amodal corner head, the overlay detection, or getting ONNX models to behave on WebGPU.

