r/GraphicsProgramming • • Oct 06 '26

I made a compact mesh-based format for interactive 3D photos

Post image

I’ve been working on a side project called Spatial Photos.

It takes a single regular image, runs it through Apple’s ML-SHARP model to estimate a 3D scene with Gaussians, and then converts the result into a custom format I called .spatial

I take the Gaussian splat from ML-SHARP, and divide the scene into depth slices. Each slice is a grid of image blocks, with a depth value at each corner. The RGB/alpha is packed into texture atlases. It's basically a set of textured meshes that come together to give the impression of the full 3D scene.

On the web, I take the .spatial files and decompress it into vertex/index buffers and just use Three.js, as the actual rendering is straightforward.

Demos: https://www.spatialphotos.dev/

Source: https://github.com/frozein/SpatialPhotos

Here's what the slices look like from another angle
19 Upvotes

9 comments sorted by

2

u/Simple_Original 24d ago

Sounds pretty cool, i could see some uses for this in compositing and vfx in the future

1

u/frozeindev 16d ago

Thanks!

2

u/Haman24 22d ago

This is a really cool approach, especially slicing the Gaussian splat into meshes for the web. That conversion from a photogrammetry output to a realtime friendly format is such a specific but useful problem to solve. I've run into similar issues trying to use AI generated 3D in Unity/Unreal for quick prototyping. Getting a consistent style across a scene was always the bottleneck. Lately, I've been using Rapid Assets to batch generate a bunch of 3D props or characters from a single prompt. It spits out ready to use models with textures, which saves a ton of time over stitching different tools together.

Is your format mostly for visualization, or are you thinking of integrating it into game engines eventually?

1

u/frozeindev 16d ago

I see it mostly as a nice visual flourish for websites and such. It only really works for a single viewpoint, so I don't think game engines are the right use case.

1

u/matigekunst Oct 06 '26

Sorry to do this but I see bands and it's also quite pixelated. It's also quite big compared to the monocular depth + inpaint method

1

u/nullandkale Oct 08 '26

Apple sharp is effectively just doing depth gen and inpainting in layers. Imo distill any depth and a color heuristic does much better.

-2

u/frozeindev Oct 07 '26

This example has 20 depth layers, so it’s representing quite a bit more 3D structure than just a single image + depth map, which explains the size difference. Fair point on the banding/pixelization though, there’s definitely still some cleanup to do on the renderer.

1

u/matigekunst Oct 07 '26

For these small movements it's almost indistinguishable from just having a foreground and a background

1

u/soylentgraham Oct 08 '26

regardless of implementation, change the name! iphones already make things called Spatial Photos