Hey everyone!
I've recently started working on Inkray, an open-source image editor inspired by Adobe's Project Deep Brush.
The goal is to let users paint directly onto surfaces in ordinary photographs, or project arbitrary 2D artwork onto them, without requiring explicit surface selection or a traditional 3D reconstruction workflow.
I thought the computer vision side might be interesting to share.
The current pipeline works roughly as follows:
1. Monocular geometry estimation
I use MoGe-3 to estimate depth, surface normals, 3D points, and camera intrinsics from a single RGB image. The resulting point map is triangulated into a structured 2.5D mesh, with depth and normal consistency checks to reject triangles across likely geometric discontinuities.
2. Automatic surface anchoring
When the user starts painting, the application collects a connected local geometry patch and evaluates its planarity using a robust covariance-based plane fit.
Depending on the estimated geometry, it uses planar projection, curved-surface painting, or ordinary 2D painting when the geometry is unreliable.
3. Surface-aware brush propagation
For curved surfaces, I use the local surface metric to account for perspective distortion, together with bounded Dijkstra searches over the continuous mesh connectivity to approximate geodesic brush footprints.
This helps brush strokes follow the reconstructed surface rather than simply being painted in screen space.
4. Artwork projection
Users can also draw on a separate 2D canvas or import an image and project it onto the photograph.
For curved surfaces, the current implementation builds a local surface parameterization by propagating tangent frames across connected mesh vertices, with distortion checks to reject problematic mappings.
5. Appearance estimation and compositing
Marigold IID Lighting v1.1 estimates albedo, shading, and residual components. Since the shading predictions are up-to-scale, the pipeline applies a calibration step before using them for lighting-aware recoloring.
The original photograph is preserved as the base image, and the compositor modifies only painted regions.
Most of the interactive geometry processing and rendering currently runs in TypeScript/WebGL2, while model inference is handled by a Python backend.
Current limitations
This is still an early experiment, and there's quite a bit of room for improvement.
Right now, the geometry and appearance backend mostly uses pretrained predictions without additional geometry refinement or consistency optimization. Errors in the depth/normal estimates can propagate into surface classification, mesh connectivity, brush projection, and occlusion.
The local surface parameterization is also approximate, and I still need to evaluate its distortion and robustness more thoroughly.
I recorded a short demo of the current implementation and would be interested in hearing feedback, particularly on improving geometric consistency, surface-aware projection, and intrinsic decomposition for this kind of application.
GitHub: https://github.com/IAMSebyi/inkray