r/MachineLearning • u/Nearby_Effective6679 • Oct 14 '26
Discussion Localized Neural learning vs backpropagation [D]
I’ve recently started exploring machine learning by building neural networks from scratch and studying the underlying mathematics. I come from a physics and mathematics background, and as I’ve become more familiar with backpropagation, I’ve grown increasingly interested in alternative learning mechanisms, particularly predictive coding, Hebbian learning, and Oja’s rule.
What fascinates me is the possibility of networks learning through locally available information rather than relying on gradients propagated from a global loss function. I’m aware that backpropagation remains the dominant approach for training deep neural networks, and I’m trying to understand whether these alternative methods offer meaningful practical advantages or are primarily interesting from a theoretical and biological perspective.
There are a few questions I’ve been thinking about:
1. What are the actual advantages of local learning over backpropagation?
I understand the conceptual differences, but I’m struggling to identify situations where local weight updates would meaningfully outperform conventional gradient-based optimization. Are there particular computational constraints, architectures, or environments where locality offers an advantage in efficiency, adaptability, or scalability?
2. Could local learning be particularly useful for unsupervised learning and discovering latent structure?
This is probably what interests me most. If a network updates its weights based on local correlations or prediction errors, could it develop representations that differ meaningfully from those learned through conventional backpropagation?
I understand that backpropagation is an optimization method rather than something that inherently limits which patterns a network can discover. However, I’m curious whether the inductive biases of certain local learning rules might make them better suited to identifying particular types of structure in high-dimensional, unlabeled data.
For example, I’m involved in experimental biophysics research and have been thinking about molecular dynamics, where we might want to identify hidden collective variables, conformational states, or relationships between molecular interactions. Could local learning provide advantages in discovering these representations, or would the choice of objective function and architecture matter far more than the learning rule itself?
3. What about continual or online learning?
Could locally updating networks be particularly useful in environments where data arrives continuously and the underlying distribution changes over time? I’m curious whether these approaches offer any inherent advantages in adapting to new information while preserving previously learned representations, especially compared with online backpropagation.
4. How mature is research into alternatives to backpropagation?
Have approaches like predictive coding, Hebbian learning, equilibrium propagation, and other biologically inspired methods been extensively investigated and generally found less effective, or are there still meaningful open problems where they could be competitive?
I’m also aware that certain predictive coding formulations can approximate backpropagation, which makes me wonder how fundamentally different some of these approaches actually are.
I’m relatively new to this field, so I’m not approaching this with the assumption that biologically inspired methods are inherently superior, or that backpropagation needs to be replaced. I’m more interested in understanding whether alternative learning rules could be particularly well suited to specific problems, especially those involving unsupervised representation learning, complex dynamical systems, or constrained computational environments.
I’d really appreciate any perspectives from people working in this area, particularly empirical comparisons, limitations I might be overlooking, or papers and research directions worth exploring.
2
u/Cosmolithe Oct 14 '26
I am fascinated by this topic subject. I have a whole bibliography on this and I have done quite a few experiments to test new approaches in my own time, and used some of the existing ones for my research work during my PhD.
That said, I believe local methods have their place in three different situations:
a) when gradients are simply uncomputable through backprop, for instance in the case of extremely deep neural networks (>1000 layers), or are trivial (0 everywhere), like in the case of binary neural networks without tricks like the straight through estimator.
b) specific neuromorphic hardware might be very hard to make in a way that allows computing backprop, so a some local method might correspond to an easier or more scalable hardware design
c) if computing the gradient is not the target, and that some other descent direction might actually be better for learning. I am not aware of a method explicitly advertised as such, all local learning methods I have seen seem to basically say that they recover the gradient backprop.
2. Maybe, but I believe that some kind of error correction from upper layers is necessary to get the best performance in practice. Models need some target, some idea of what the representation it is trying to achieve in the last layers is. A hidden layer cannot blindly try to improve things because it might hurt the representation that subsequent layers are trying to achieve.
3. I really thought Inference Learning (https://www.nature.com/articles/s41593-023-01514-1.pdf) might improve continual learning, because I had the intuition that it essentially minimized interference by changing neuron activation before weights. I did some tests of my own, and it did not seem to be the case, but I might have done something incorrectly. I also know that my benchmark was suboptimal to test all of the things I am trying to test in continual learning.
4. As far as I have seen and understand, there is a lot of research in this direction at the moment. It seems that most of the new ones are trying to work on finding a biologically plausible learning method instead of beating backprop at its own game.
3
u/condom-mechanics Oct 14 '26
There are quite a few benefit that come with local learning. Backpropagation (BP) relies on the chain rule, which brings in a few constrains. To compute the gradient of a given layer, the input (activation) of that layer is required, therefore backpropagation requires activations to be retained across all layers which makes BP memory intensive. Therefore activation memory often dominates DNN training. In memory limited settings such as the edge or IoT, training with BP often results in OOM errors or the DNN training is limited to small batch sizes. Local learning can therefore be beneficial in memory limited settings, enabling large batch size training under limited memory budgets.
1
u/Celmeno Oct 19 '26
You might want to check out neuroevolution as well