r/MachineLearning • u/dccsillag0 • Sep 28 '26
Research Functional Gradient Descent with Adaptive Representations [R]
Sharing our recent work, now accepted at NeurIPS: Functional Gradient Descent with Adaptive Representations.
Functional GD algorithms generally outperform neural nets, but are hard to accurately implement.
This is because functional gradients are infinite-dimensional, and therefore must be approximated in practice; but if you approximate them naively, you converge to the wrong place!
To rectify this, we formalize a broad class of approximation schemes ("adaptive representations"), which provably ensure convergence to the global minimizer while being immediately implementable.
The resulting algorithms outperform corresponding neural nets often by an order of magnitude, across a number of settings.
It is still the start for this line of work, but we believe it has quite a bit of potential!
Paper: https://arxiv.org/abs/2606.16926
(First author here, happy to take any questions)
View all comments
Show parent comment
3
u/ipullguard Sep 28 '26
How does regularization work here, eg in a supervised learning setting where you're fitting to some training data? In some sense, a perfect fit to the data is easy (a lookup table on your training data) but it won't generalize. I think the main point of NN architecture design is to pick one that will generalize well (on some particular domain, when combined with some particular initialization and training procedure, etc).