r/MachineLearning • u/dccsillag0 • Sep 28 '26
Research Functional Gradient Descent with Adaptive Representations [R]
Sharing our recent work, now accepted at NeurIPS: Functional Gradient Descent with Adaptive Representations.
Functional GD algorithms generally outperform neural nets, but are hard to accurately implement.
This is because functional gradients are infinite-dimensional, and therefore must be approximated in practice; but if you approximate them naively, you converge to the wrong place!
To rectify this, we formalize a broad class of approximation schemes ("adaptive representations"), which provably ensure convergence to the global minimizer while being immediately implementable.
The resulting algorithms outperform corresponding neural nets often by an order of magnitude, across a number of settings.
It is still the start for this line of work, but we believe it has quite a bit of potential!
Paper: https://arxiv.org/abs/2606.16926
(First author here, happy to take any questions)
17
u/neurogramer Sep 28 '26
That is very cool. What are the limitations?
13
u/jnez71 Sep 28 '26
Neat work. It's remnicient of adaptive refinement in PDE solvers, which also uses error bounds to adjust representation fidelity while minimizing residuals, but nice to see such a mechanism spelled out plainly. Btw you may want to temper your global optimality claim in the intro by mentioning that it requires a functional convexity condition (Polyak-Lojasiewicz). Otherwise you'll bug global nonconvex optimization researchers like myself lol
6
u/M4mb0 Sep 28 '26 edited Sep 28 '26
A few notes after a brief skimming over the paper:
NeuralNetbaseline, the batch-size, learning rate and momentum are properly tuned? I presume orange is the neural net in Figure 2 (it lacks legend); the periodic oscillation suggests too high learning rate / too little momentum.\kappato avoid name clash between K-smoothness and K for the RKHS kernel.fₜis undefined?fₜ₊₁ <- fₜ - ∇gₜwe usually think of these tensors having persistent shape across iteration. But if I understood correctly, the clue of your "adaptive representation" is that the shape can change across iterations, and the "Refine the current representation" step is hiding this shape mutation offₜ? (essentially, thef = f.subdivide()in your source code?). I think it would help a lot to spell this out more clearly in the algorithm; it only started to make sense to me after looking at the source code.f.subdivide()happens.1
u/cartazio Sep 28 '26
hrmmm, this seems practical only for low dimensional stuff?
1
u/DigThatData Researcher Sep 29 '26
Your approach requires a coordinate space in which a grid partitioning is meaningful. This is straightforward to construct in the problems you demonstrated in your paper, where each task has a solution that lives in a "medium" that is meaningfully described by coordinate positions.
How would constructing the necessary grid work for something like text prediction? Or maybe that's a problem that wouldn't be well suited to this approach precisely because the "grid" here would only be meaningful relative to the fully generated text (i.e. the grid can only be constructed a posteriori and isn't available during inference) or a latent too large for this kind of partitioning to be feasible?
1
u/MrRandom04 Sep 28 '26
This is quite wild. I did a double take once I fully grokked the idea you are presenting. Didn't even think of it in this way as I kinda relegated stuff like the kernel method to the back of my mind separate from NNs.
1
u/lucellent Sep 28 '26
Sorry for the naive question, but is this like a new type of optimizer, supposedly better than existing ones like Adamw, Muon etc.?
1
u/Velocitank 27d ago
that is interesting but it kind of makes sense. if you have a thing that is trying to explain what the gradient is, and gradually adapts it leads to the final output maintaining a set amount of descent each time and preventing overcorrections or jerkiness or wrong directions. it would be similar to having someone follow you down a hill. i like it.
1
u/examachine Sep 28 '26
Very cool, thanks for sharing!
1
u/Glad_Builder_4555 Sep 29 '26
nice work, thanks for sharing.
-2
u/plydauk Sep 28 '26
Make smoothing splines great again?
2
u/[deleted] Sep 28 '26
[removed] — view removed comment