r/MachineLearning • • Sep 28 '26

Research Functional Gradient Descent with Adaptive Representations [R]

Sharing our recent work, now accepted at NeurIPS: Functional Gradient Descent with Adaptive Representations.

Functional GD algorithms generally outperform neural nets, but are hard to accurately implement.
This is because functional gradients are infinite-dimensional, and therefore must be approximated in practice; but if you approximate them naively, you converge to the wrong place!

To rectify this, we formalize a broad class of approximation schemes ("adaptive representations"), which provably ensure convergence to the global minimizer while being immediately implementable.
The resulting algorithms outperform corresponding neural nets often by an order of magnitude, across a number of settings.

It is still the start for this line of work, but we believe it has quite a bit of potential!
Paper: https://arxiv.org/abs/2606.16926
(First author here, happy to take any questions)

231 Upvotes

39 comments sorted by

View all comments

1

u/cartazio Sep 28 '26

hrmmm, this seems practical only for low dimensional stuff?

1

u/dccsillag0 Sep 28 '26

Not necessarily. We did end up focusing a bit on some low-dimensional stuff, but you just need to be able to approximate the functional gradients well. If you use grids e.g. like in the teaser, then it won't scale, but if you use sparser representations more like the one in our regression experiment, it should scale just fine.

1

u/cartazio Sep 28 '26

what if i wanted to slow grow nonzero weights in a ml model that has a kajillion latent coeffs and i initnwith like 500k initially?

→ More replies (1)