Learning Without Backpropagation: Hebb, STDP and Local Learning Rules

Backpropagation has a property that is easy to miss because it is so convenient: it is not local. To adjust a synapse buried three layers deep, the algorithm needs to know how much that synapse contributed to an error measured at the output, which means an error signal has to travel backwards through the network along the exact same weights it just travelled forwards through. Every weight must be used twice, in two directions, and the backward copy must stay perfectly synchronised with the forward one.

Nothing in a brain does this. A synapse has access to the activity of the two neurons it connects and to whatever is diffusing nearby, and to nothing else - no report of what happened at the far end of the cortex, no transposed copy of any weight matrix. Yet brains learn continuously, from streaming data, on roughly twenty watts. So something purely local must be enough for a great deal of what learning does, and the search for what has produced a family of rules simpler and older than backpropagation - and increasingly relevant now that chips are being built to run them.

Hebb's Postulate

In 1949 the psychologist Donald Hebb proposed a mechanism for how experience could leave a physical trace in neural tissue [1]. His formulation was verbal and carefully hedged: when one cell repeatedly takes part in firing another, some growth process or metabolic change occurs in one or both cells such that the first cell's efficiency in firing the second is increased.

It is almost always quoted as "cells that fire together wire together", which is snappier and slightly wrong. Hebb never wrote it - the phrase is a much later paraphrase - and it drops the directional asymmetry that turns out to matter enormously. Hebb wrote about one cell taking part in firing another, which is a causal claim, not a statement about correlation. That distinction is dormant for forty years and then becomes the whole story.

Written as a learning rule, the postulate is about as simple as an equation gets. For a synapse with weight $w$ connecting a presynaptic neuron with activity $x$ to a postsynaptic neuron with activity $y$:

where $\eta$ is a learning rate. Every quantity in that expression is available at the synapse itself. There is no error term, no target, no reference to any other synapse and no backward pass. Whatever else is wrong with it, it is exactly the kind of rule biology could plausibly implement.

It was also, for more than twenty years, entirely speculative. Hebb proposed the mechanism without any evidence that synapses could change their strength at all. That arrived in 1973, when Bliss and Lømo applied brief high-frequency stimulation to a pathway in the rabbit hippocampus and found the response to a single test pulse stayed elevated for hours afterwards [2]. Long-term potentiation gave Hebbian learning a physical substrate and turned a psychological conjecture into a question about molecules.

Why the Rule Cannot Work As Written

Run it and it destroys itself. If $x$ and $y$ are positively correlated then $\Delta w$ is positive, which makes $y$ larger next time, which makes $\Delta w$ larger still. The rule is pure positive feedback with nothing opposing it. Weights grow without bound, every neuron saturates, and the network ends up in a state that reflects nothing about its input.

There is a second problem hiding behind the first: the rule can only ever strengthen. Nothing in it weakens a synapse, so nothing is ever unlearned and no synapse can specialise by giving something up. A memory system that can only add is not a memory system.

Both problems have the same shape, and it recurs throughout this site: a positive feedback loop that discovers structure needs a counterweight, or it runs away with itself. Pheromone trails have evaporation; Physarum tubes have a decay term. The question is what that counterweight can look like when the only information available is local.

Oja's Rule: A Local Rule Computing a Global Statistic

Erkki Oja's answer in 1982 was to add a decay term proportional to the postsynaptic activity squared, which normalises the weight vector as it grows [3]:

The added term is still local - it uses only this synapse's own weight and the postsynaptic activity - but it acts as a brake that grows with the neuron's output. The weight vector converges to unit length instead of exploding.

What it converges to is the genuinely surprising part. A neuron running Oja's rule ends up with a weight vector aligned to the first principal component of its input distribution: the single direction along which its inputs vary most. Principal component analysis is normally introduced as an operation on a covariance matrix, something you compute over an entire dataset held in memory. Here it falls out of one neuron adjusting one synapse at a time using only quantities available at that synapse, with no dataset, no matrix and no global view. A local rule computes a global statistic, and nothing in the neuron knows that this is what it is doing.

BCM: One Synapse, Both Directions

Oja's rule stabilises magnitude but still only ever moves a synapse in one direction for a given input. Bienenstock, Cooper and Munro proposed a rule in the same year in which the sign of the change depends on how active the postsynaptic neuron has recently been [4].

The mechanism is a sliding threshold. Postsynaptic activity above it strengthens the active synapses and activity below it weakens them, and the threshold itself moves with the neuron's own recent average activity. A neuron that has been firing a lot raises its bar, becoming harder to strengthen and easier to weaken; a quiet neuron lowers its bar and turns eager to potentiate. This is homeostasis implemented inside a learning rule, and it solves both of Hebb's problems at once - runaway strengthening is self-limiting, and depression comes for free rather than needing its own mechanism.

Spike Timing: From Correlation to Causality

Everything above treats neural activity as a rate - a number for how active a cell is. Real neurons emit discrete spikes, and once their timing is taken seriously a sharper rule becomes available. The experiments that pinned it down came in the late 1990s. Markram and colleagues paired presynaptic and postsynaptic spikes at controlled intervals and found that the sign of the resulting change in synaptic strength depended on their order [5]. Bi and Poo mapped the effect systematically across a range of intervals in cultured hippocampal neurons, producing the asymmetric curve now known as the STDP window [6].

The result is spike-timing-dependent plasticity, and its logic is worth stating carefully. If the presynaptic spike arrives before the postsynaptic neuron fires, the synapse is strengthened. If it arrives after, the synapse is weakened. The size of the change falls off roughly exponentially with the interval, over a window of tens of milliseconds:

with a mirror-image term of opposite sign for , where $\Delta t$ is the postsynaptic spike time minus the presynaptic one and $\tau$ is around 20 ms in most preparations.

That asymmetry is the entire point. A synapse whose input reliably arrives just before its target fires is a synapse that plausibly helped cause that firing, and it gets stronger. A synapse whose input arrives just after has been ruled out as a cause, and it gets weaker. Simultaneity is not rewarded; precedence is. This is a local rule that approximates causal inference, using nothing but the arrival times of two spikes - and it is what Hebb actually wrote, rather than what he is usually quoted as writing.

What Local Rules Are Good At

Left to run on unlabelled input, STDP does something useful without being told to: neurons become selective for recurring patterns. A group of neurons exposed to handwritten digits, with lateral inhibition so that they compete rather than all learning the same thing, will individually specialise for particular strokes and shapes, and the population reaches respectable classification accuracy on MNIST with no labels involved in the learning at all [7]. Labels are needed only at the very end, to read off which neuron corresponds to which digit.

The practical appeal is what this makes possible in hardware. A local rule needs no stored activations, no backward pass and no separate training phase, so it can run on the same chip that is doing inference, continuously, while the system is deployed. Intel's Loihi implements programmable synaptic plasticity in silicon for exactly this reason [8]: the learning happens where the data already is, which is the same argument that motivates neuromorphic architecture generally.

Where Local Rules Fall Short

It would be a much better story if local rules simply replaced backpropagation. They do not, and the reason is precise rather than vague.

They do not solve credit assignment. STDP tells a synapse whether it helped cause its own neuron to fire. It says nothing about whether that neuron firing was useful - whether it moved the system towards recognising a face, or catching a ball, or anything else. Backpropagation's expensive non-locality buys precisely this: an answer to "how did this weight affect the outcome I care about". A purely local rule is structurally incapable of asking that question, because the outcome is not local.

The features are unsupervised, so they are not task-aware. Oja's rule finds the direction of greatest variance. That is a reasonable guess at what matters, and it is frequently the wrong one - the largest source of variance in a set of photographs might be overall brightness, which is exactly the thing you want to ignore.

And the results reflect this. On hard benchmarks, networks trained with STDP remain well behind equivalent networks trained with backpropagation. The gap has narrowed and the energy comparison is far kinder, but it is real and it has not closed.

The interesting work sits in between. Feedback alignment showed that the backward pass does not need the transposed forward weights at all: random fixed feedback weights still support learning, because the forward weights adapt to align with whatever feedback they are given [9]. That removes the most biologically indefensible requirement in one move. Other approaches - predictive coding, target propagation, equilibrium propagation - try to reach the same place by having local activity differences carry the error signal rather than a dedicated backward pass. A review of where this leaves things argues that the brain probably does perform some form of approximate credit assignment, and that finding its mechanism is a live research problem rather than a solved one [10].

Why It Matters

The gap between Hebb's rule and backpropagation is a good illustration of a trade the rest of this site keeps running into. Backpropagation buys accuracy with global information, and pays for it in memory, in a separate training phase, and in an architecture no physical substrate implements for free. Local rules buy continuous, on-device, low-power learning, and pay for it by having no idea what the task is.

Neither is obviously the right answer, and biology is not straightforwardly on one side. The evidence suggests brains do something more sophisticated than plain STDP and considerably less exact than backpropagation - some approximation of credit assignment built out of mechanisms that never stop being local. Working out what that approximation is remains one of the more interesting open problems shared between neuroscience and machine learning, and unusually, progress on it would matter to both.