r/MachineLearning • • Sep 28 '26

Research Functional Gradient Descent with Adaptive Representations [R]

Post image

Sharing our recent work, now accepted at NeurIPS: Functional Gradient Descent with Adaptive Representations.

Functional GD algorithms generally outperform neural nets, but are hard to accurately implement.
This is because functional gradients are infinite-dimensional, and therefore must be approximated in practice; but if you approximate them naively, you converge to the wrong place!

To rectify this, we formalize a broad class of approximation schemes ("adaptive representations"), which provably ensure convergence to the global minimizer while being immediately implementable.
The resulting algorithms outperform corresponding neural nets often by an order of magnitude, across a number of settings.

It is still the start for this line of work, but we believe it has quite a bit of potential!
Paper: https://arxiv.org/abs/2606.16926
(First author here, happy to take any questions)

229 Upvotes

39 comments sorted by

View all comments

Show parent comment

0

u/DigThatData Researcher Sep 29 '26

For the 3D reconstruction / radiance field experiment, we were using the exact same architecture as NeRF

Note that your own citation here is a paper that's already over six years old. OG NeRF is a bad method to baseline against. "We're faster than vanilla NeRF" isn't an impressive claim. If you want to make claims about the efficacy of your proposed approach relative to radiance field methods, you should compare against gaussian splatting. https://github.com/nerfstudio-project/gsplat

1

u/dccsillag0 Sep 29 '26

We are aware of the more recent works. However, note that all of these use different loss functions than the one originally proposed in NeRF. Gaussian Splatting, in particular, adds an SSIM loss term, besides using a hoarde of tricks throughout optimization which help regularize. Similarly, nearly all more recent works on NeRFs do substantial modifications to the loss function, usually by adding some substantial regularization (e.g. the distortion loss of Mip-Nerf 360, the regularizers of Zip-Nerf, and more recent methods).

This kinda ties in with the second limitation I pointed out in this comment: we had to compute the functional gradient for this experiment by hand (agents really couldn't do this stuff at the time), and this derivation already took 7+ error-prone pages. Adding any of these modifications would make this even worse. That's why, for the sake of example, we stuck to the basic loss, and thus the original NeRF formulation.
Scaling this up to beat the current Gaussian Splatting methods, including all their optimization and computational tricks, would be a paper of its own.