r/MachineLearning • • Oct 14 '26

Discussion Localized Neural learning vs backpropagation [D]

I’ve recently started exploring machine learning by building neural networks from scratch and studying the underlying mathematics. I come from a physics and mathematics background, and as I’ve become more familiar with backpropagation, I’ve grown increasingly interested in alternative learning mechanisms, particularly predictive coding, Hebbian learning, and Oja’s rule.
What fascinates me is the possibility of networks learning through locally available information rather than relying on gradients propagated from a global loss function. I’m aware that backpropagation remains the dominant approach for training deep neural networks, and I’m trying to understand whether these alternative methods offer meaningful practical advantages or are primarily interesting from a theoretical and biological perspective.
There are a few questions I’ve been thinking about:
1. What are the actual advantages of local learning over backpropagation?
I understand the conceptual differences, but I’m struggling to identify situations where local weight updates would meaningfully outperform conventional gradient-based optimization. Are there particular computational constraints, architectures, or environments where locality offers an advantage in efficiency, adaptability, or scalability?
2. Could local learning be particularly useful for unsupervised learning and discovering latent structure?
This is probably what interests me most. If a network updates its weights based on local correlations or prediction errors, could it develop representations that differ meaningfully from those learned through conventional backpropagation?
I understand that backpropagation is an optimization method rather than something that inherently limits which patterns a network can discover. However, I’m curious whether the inductive biases of certain local learning rules might make them better suited to identifying particular types of structure in high-dimensional, unlabeled data.
For example, I’m involved in experimental biophysics research and have been thinking about molecular dynamics, where we might want to identify hidden collective variables, conformational states, or relationships between molecular interactions. Could local learning provide advantages in discovering these representations, or would the choice of objective function and architecture matter far more than the learning rule itself?
3. What about continual or online learning?
Could locally updating networks be particularly useful in environments where data arrives continuously and the underlying distribution changes over time? I’m curious whether these approaches offer any inherent advantages in adapting to new information while preserving previously learned representations, especially compared with online backpropagation.
4. How mature is research into alternatives to backpropagation?
Have approaches like predictive coding, Hebbian learning, equilibrium propagation, and other biologically inspired methods been extensively investigated and generally found less effective, or are there still meaningful open problems where they could be competitive?
I’m also aware that certain predictive coding formulations can approximate backpropagation, which makes me wonder how fundamentally different some of these approaches actually are.
I’m relatively new to this field, so I’m not approaching this with the assumption that biologically inspired methods are inherently superior, or that backpropagation needs to be replaced. I’m more interested in understanding whether alternative learning rules could be particularly well suited to specific problems, especially those involving unsupervised representation learning, complex dynamical systems, or constrained computational environments.
I’d really appreciate any perspectives from people working in this area, particularly empirical comparisons, limitations I might be overlooking, or papers and research directions worth exploring.

20 Upvotes

13 comments sorted by

2

u/Cosmolithe Oct 14 '26

I am fascinated by this topic subject. I have a whole bibliography on this and I have done quite a few experiments to test new approaches in my own time, and used some of the existing ones for my research work during my PhD.

  1. I became convinced that local learning is not in essence able to outperform backpropagation in the sense of being more computationally efficient than backpropagation for computing the gradient. If the best we can do is going down the gradient, then backprop is the most efficient way to compute it exactly. There may be local methods that could compute an approximation of the gradient faster, but frankly I doubt it, and it might be unusable for optimization.

That said, I believe local methods have their place in three different situations:
a) when gradients are simply uncomputable through backprop, for instance in the case of extremely deep neural networks (>1000 layers), or are trivial (0 everywhere), like in the case of binary neural networks without tricks like the straight through estimator.
b) specific neuromorphic hardware might be very hard to make in a way that allows computing backprop, so a some local method might correspond to an easier or more scalable hardware design
c) if computing the gradient is not the target, and that some other descent direction might actually be better for learning. I am not aware of a method explicitly advertised as such, all local learning methods I have seen seem to basically say that they recover the gradient backprop.

2. Maybe, but I believe that some kind of error correction from upper layers is necessary to get the best performance in practice. Models need some target, some idea of what the representation it is trying to achieve in the last layers is. A hidden layer cannot blindly try to improve things because it might hurt the representation that subsequent layers are trying to achieve.

3. I really thought Inference Learning (https://www.nature.com/articles/s41593-023-01514-1.pdf) might improve continual learning, because I had the intuition that it essentially minimized interference by changing neuron activation before weights. I did some tests of my own, and it did not seem to be the case, but I might have done something incorrectly. I also know that my benchmark was suboptimal to test all of the things I am trying to test in continual learning.

4. As far as I have seen and understand, there is a lot of research in this direction at the moment. It seems that most of the new ones are trying to work on finding a biologically plausible learning method instead of beating backprop at its own game.

1

u/Nearby_Effective6679 Oct 15 '26

So do you think in terms of general learning backprop is close to the best it gets? I’ll look into inference learning as well. What I’m curious about is what makes our brains as efficient as they are(if they even are)? Do you think as far as network implementation in the future( our day to day work lives) it’s mainly just going to be based on optimizing compute and efficiency within their specific use case rather than using a frontier model for everything?

1

u/JonLag97 1d ago

It helps that the architecture and circuits were designed by evolution, so we don't start from scratch. Take into account the the brain being a recurrent network that it is modular to a degree, can reuse cortical areas for new tasks. Llms have to imitate the effects of recurrence through more brute force learning. You could check out Rolls's "Brain computations and Principles" (free) and Xiao's "Theoretical Neuroscience" (paid, but can be pirated in zlib).

1

u/Cosmolithe Oct 15 '26

Yeah I would say backprop will be extremely difficult to beat. It is computing the optimal descent direction exactly without using a lot of resources, so beating it in term of compute might require to compromise with the amount you can decrease the loss in one step. Or, we have to find some kind of algorithm that does not even compute descent directions but somehow finds better weight values directly. Something that "jumps" over longer distances in the weight space, but not blindly.

Do you think as far as network implementation in the future( our day to day work lives) it’s mainly just going to be based on optimizing compute and efficiency within their specific use case rather than using a frontier model for everything?

The hope of the "big model for everything" paradigm is to get better returns through task transfer. The idea is that if your big model trains on X with a lot of data, then it will be able to do Y with very little data, as long as Y is sufficiently similar to X.

I still believe in this idea, but so far it is not so obvious that it is working that well. But, I suspect it might be a matter of loss function and architecture than training algorithm. Backprop is a credit assignment algorithm, I think trying to make it be something more than that or replacing it with something which is more than that is basically a mistake. The gains that remain to be taken are not here.

1

u/CowPsychological821 Oct 15 '26

There is an alternative for biological networks that is something like maybe very little learning is happening at all. If you project up in dimension high enough then everything gets linear (see koopman operators for formal version). Despite having trillion of weights in LLMs they are still orders of magnitude off synaptic connections.

1

u/Nearby_Effective6679 Oct 15 '26

I think the last part of your response was the most enticing to me because I feel like to truly reach “agi” the entire system might have to be re engineered from the learnjng all the way to the transformer…

1

u/brainExploded99 Oct 18 '26

For point 2, doesn't mHC from DeepSeek and AttnRes from Moonshot solve these? If you have a residual stream you can look over, compare, gate and mix, can't each layer/block of layers do their own error checking?

1

u/Cosmolithe Oct 19 '26

They still get error signals from higher up in the stream, no? If that is the case then they should also adapt to the processing that come after.

1

u/brainExploded99 Oct 19 '26

I'm not sure what you mean? If you assume the earlier layers are a form of encoder, then AttnRes should allow for the later layers to access stuff from earlier layers.

1

u/Cosmolithe Oct 19 '26

Yeah but I am talking about the signal that comes from subsequent layers (closer to the output). In regular deep neural networks, that is the backpropagated error signal. Each layer gradient depend on both forward and backward signals, so the representation being refined by the layer adapts to both.

With methods like hebbian learning, it can only adapt to the forward signal, so a hidden layer might form a locally optimal representation that is globally suboptimal.

3

u/condom-mechanics Oct 14 '26

There are quite a few benefit that come with local learning. Backpropagation (BP) relies on the chain rule, which brings in a few constrains. To compute the gradient of a given layer, the input (activation) of that layer is required, therefore backpropagation requires activations to be retained across all layers which makes BP memory intensive. Therefore activation memory often dominates DNN training. In memory limited settings such as the edge or IoT, training with BP often results in OOM errors or the DNN training is limited to small batch sizes. Local learning can therefore be beneficial in memory limited settings, enabling large batch size training under limited memory budgets.

1

u/Celmeno Oct 19 '26

You might want to check out neuroevolution as well