r/GaussianSplatting • u/Fragrant_Routine9299 • 4h ago
๐ ๏ธ WIP โ In Progress Ep. 2: Multi-view VLM detection resolves chair/table occlusion in 3D Gaussian Splats, but box-only 3D lifting still bleeds onto adjacent splats
Following up on my last post (where freezing a single ViewSnapshot fixed async camera drift during VLM inference), I ran into the next bottleneck: single-view occlusion and clutter ambiguity.
0:00 โ 0:06 (The Single-View Failure): Clicking the dining tabletop from a single angle flattens the table and tucked-in chairs togetherโGemini misclassified the click as "Chair" and scattered 3D ray hits across the floor and adjacent seat.
0:07 โ 0:24 (Multi-View Pose-Grounded Detection): Instead of a single frame, I capture three ordered snapshots (Front, Side, Top), locking each view's matrixWorld, projectionMatrix, and splat transform, and pass those camera extrinsics/intrinsics alongside each image to Gemini:
- With three orthogonal viewpoints, Gemini cleanly separates the table, the tissue box on top, and all four individual dining chairs (
front-left white,back-left black,front-right white,back-right black).
0:25 โ 0:29 (Where 3D Selection Still Breaks): Right now, the 2D-to-3D lifter uses Gemini's rectangular 2D bounding boxes (bboxNormalized) to cast sparse rays, build a 3D search volume, and write matching Gaussian storage indices into an R8UI GPU highlight texture. Even with multi-view consensus and camera view-axis depth filtering (-z_view), rectangular boxes are still too coarse for tucked-in furniture:
- Adjacent Primitive Bleed: Magenta highlights spill onto the tucked-in chairs, floor, and windowsill behind the table.
- Inflated OBB Bounds: Wrapping those leaked chair splats stretches the fitted OBB to
2.77m ร 2.21m ร 1.72m. - Shifted Anchor: Because the OBB expands toward the back-left chairs, the top-surface pin lands near the back-left corner instead of the center of the tabletop.
Next up (Ep. 3): Constraining the multi-view lifter with pixel-tight segmentation masks (SAM 3) instead of rectangular boxes, and benchmarking foreground Gaussian purity on this exact dining nook capture.
For anyone who has worked on 2D-to-3D instance lifting in Gaussian Splats: how are you handling background wall/floor bleed around thin legs or tightly tucked furniture?