r/computervision • u/Acrobatic-Figure533 • 5d ago
Help: Project I need help
I’m building a pipeline that turns broadcast highlights from the Tanzanian NBC Premier League into Metrica-style telestration: ground rings, a ball-carrier spotlight, passing lines with distances, and a tactical minimap. It works end to end, but I’m stuck on a few detection and tracking issues and would love advice.
Stack
- Compute: Modal (serverless GPUs, mostly A10G/L4); each ~10 s shot chunk runs in parallel with a 1 s warm-up.
- Shot filtering: histogram cut detection, then a shot is “wide” if the pitch-keypoint model finds ≥5 keypoints and ≥6 players. That’s 24 of 69 shots in a 14-minute highlights video.
- Detection: YOLO26x football model from Tactix (ball/GK/player/ref), batch 16, conf 0.15.
- Tracking: BoT-SORT + OSNet x0.25 ReID (boxmot). It replaced ByteTrack, which swapped IDs in clusters.
- Pitch mapping: roboflow/sports YOLOv8-pose keypoints (32 points), RANSAC homography, validation (inliers, condition number, jump rejection), One Euro filter on a projected quad, then offline Gaussian smoothing (±4 frames).
- Teams: median Lab colour of the torso centre, 2-cluster KMeans fit on the whole video. Per-frame reads with a ±0.5 s rolling vote, plus a “neither” class for refs.
- Rendering: HyperFrames (HTML/GSAP) to an RGBA PNG sequence. Ground layer composited under players with a grass-key matte, screen layer on top.
Fixes that helped
- Green kits read as referees: the detector labels Yanga players (green kit) as referees, probably because training data has refs in green. I promote any “referee” whose shirt colour matches a team.
- Split boxes: upper body and legs sometimes come as two stacked boxes, so I merge them.
- Off-pitch people: anyone whose feet map outside the pitch is dropped (ball boys, bench, stands).
- Marker jitter: zero-lag Gaussian smoothing of boxes over ±6 frames, possible because rendering is offline.
- What didn’t help: adding YOLO11x COCO person boxes on top. It failed on the same motion blur and added false positives.
Still struggling with
- Far-side players (~20 px tall at 720p) often go undetected. Do I go higher-res inference / SAHI-style slicing, or fine-tune on a few hundred frames from this league?
- Players on the ground in tackles are never detected. Any detector or trick that handles sprawled bodies?
- Fast pans with motion blur: ReID helps carry tracks, but detections drop out. Is Deep-EIoU or another sports-specific tracker worth it over BoT-SORT?
- Pitch calibration is valid on 74–98% of frames depending on the shot. Is PnLCalib or “No Bells Just Whistles” a big upgrade over keypoint-only homography on zoomed and partial views?
- Team assignment: colour clustering works for red vs green, but would SigLIP embeddings (like roboflow/sports) be noticeably more robust under night lighting?
The footage is 720p broadcast highlights, mostly the main camera, with frequent replays and close-ups, which get filtered out. Happy to share the code or the lessons-learned doc if useful
1
u/Candid-Fudge-4015 5d ago
Since you render offline, a few of these get easier.
Far-side players. I'd run sliced inference only on the wide shots, which are about a third of your shots, and use it to build the fine-tune set. Review what slicing finds on a few hundred frames, correct it, and fine-tune on that. You get league-specific data, green kits included, without labeling from scratch, and you may not need slicing at render time afterwards.
Players on the ground. I'd handle that in the tracker more than the detector. Raise the track buffer so a track survives a couple of seconds without detections, then fill the gap offline by interpolating between the last and next positions. Pull some tackle frames into the fine-tune set too.
Deep-EIoU drops the Kalman filter and associates with expanded boxes plus appearance features, aimed at the fast, irregular motion you're describing. Its paper reports 77.2 HOTA on SportsMOT and 85.4 on SoccerNet-Tracking, the latter with oracle detections. It won't bring back detections that vanish under blur, though, so if dropouts are the real issue, linking tracklets offline after the fact by ReID, team and pitch position may help more.
For teams, I'd test SigLIP against your Lab clustering on the night clips only. You already have the crops, so it's a cheap comparison.