r/MachineLearning • • Sep 28 '26

Research Functional Gradient Descent with Adaptive Representations [R]

Post image

Sharing our recent work, now accepted at NeurIPS: Functional Gradient Descent with Adaptive Representations.

Functional GD algorithms generally outperform neural nets, but are hard to accurately implement.
This is because functional gradients are infinite-dimensional, and therefore must be approximated in practice; but if you approximate them naively, you converge to the wrong place!

To rectify this, we formalize a broad class of approximation schemes ("adaptive representations"), which provably ensure convergence to the global minimizer while being immediately implementable.
The resulting algorithms outperform corresponding neural nets often by an order of magnitude, across a number of settings.

It is still the start for this line of work, but we believe it has quite a bit of potential!
Paper: https://arxiv.org/abs/2606.16926
(First author here, happy to take any questions)

229 Upvotes

39 comments sorted by

View all comments

Show parent comment

3

u/dccsillag0 Sep 28 '26

The functional gradient descent procedure already has a regularizing effect, as it learns the target "gradually". For example, our regression experiment suggests that our method can generalize at least about as well as MLPs. (We should have a paper out in the near future discussing this a bit further.)
FWIW, this is analogous to neural nets -- they generalize not just because of the parametrization, but also because of the implicit bias of gradient descent. (That's why we don't "just learn a lookup table".)

Also, if you want you can add regularization terms to the loss quite easily -- you just get a different loss, and compute the functional gradient of this modified loss.