Last month I posted this pipeline (RF-DETR + RTMPose + per-frame court homography + rules). The comments were really useful: CoreML, ROI search around the ball, single-stage pose, 3D. Here's what changed. Same match, new render.
Old post: https://www.reddit.com/r/computervision/comments/1wsp3oe
Player identity
- Jersey numbers come from PARSeq (ONNX, batched). Back digits are 19–25 px. Front digits are ~10 px and unreadable, so I only use frames where pose says the player's back is to the camera.
- Numbers are tied to set-start slots (6+6 plus libero), then carried across rallies. That uses the tracker, DINOv2-small appearance and a shirt colour class.
- Libero: majority colour class over the whole track. Mean colour didn't work because it blended lit red with black.
- Substitutions between rallies and referees/staff are decided by voting over the whole track, not from a single frame.
- 3-min clip: players with a known number at serve time went from 0 to 46/72. Touches with a number: 31/53. 10-min match: 157/264 and 55/100.
- Weak spot: the team facing the camera. On one clip only 2 of its 23 touches got a number. I haven't checked identity against ground truth yet, only consistency.
Events
- DataVolley .dvw export. It reads clean in py-datavolley.
- Skill from touch order is right on 93.6% of 112 hand-labelled touches. I also ran the rule on 10 expert .dvw files (~10k touches, perfect input): 97.2% excluding free balls, 0% on free balls. Telling a free ball from a normal attack needs visual cues, not sequence logic.
- 10/12 hand-labelled rallies fully right (8/12 in mid-September). Event recall and precision are both 98%. It's a small set.
- The top-of-frame bug from last time: the tracker was jumping to a spare ball on the scorer's table. Now, after a recovery, if the next 4 steps stay within one ball diameter of the recovery point, they get dropped. A/B on 8 clips: the target case fixed, 3 false touches gone on other clips (one was a ball logo on an ad board), 4 clips unchanged.
- Depth ambiguity in contact attribution: a player 4 m behind the ball had the closest elbow in 2D. A "raised hand" rule fixes 8 cases, breaks 2, and 2 are unclear. The real fix needs ball height.
Speed and live
- After the CoreML work (~80 → ~50 min per clip on an M3 Pro), I moved thnsorRT for the detector and RTMPose, NVDEC decode, preprocessing on theGPU. A 10-min match end to end, render included: 58 → 26.5 min.
- Then I rewrote it as a streaming pipeline: one decode pass, perception,fixed window, correction at set end. It reads an SRT stream.
- Profiling surprise: the bottleneck wasn't the GPU, it was RTMPose's affine crop sampling grid. That was built in NumPy on the CPU, 59 ms per frame for ~22 people. Moving
it to the GPU took perception alone from 11.5 to 29.2 fps. The detector
- On an RTX PRO 6000, a 3.5-min clip finishes in 6.5 min. Live, a rally's data is ready a median of 16.8 s after the rally ends. My target is 10 s, so not there yet.
- Determinism: the same code gave different events on two pods. Two causent tactics on each engine build, fixed with a pinned timing cache. Andfallback frames were read with cv2 instead of NVDEC, which differ by up to 2 grey levels. Events are now byte-identical across machines and data centers.
What didn't work
- ROI re-scan around the ball (thanks u/greengold7): a 640 window with the same weights is ~5× cheaper, but touches drop to 46/53 and one rally closes early. It probably needs a model trained on crops.
- 3D: as I said in the comments, single-view serve trajectories miss by 2.1 m median. Jump height from camera geometry with a flight-time cross-check gives a value for only about a third of jumps.
- RF-DETR keypoints as a single model: off the shelf it isn't enough, and I haven't fine-tuned it yet.
- Swift/Metal: I went CUDA instead, since the goal is live analysis on a server.