Neural Networks: From Perceptron to Deep Learning

The human brain contains roughly 86 billion neurons, each connected to thousands of others via synaptic junctions. When computer scientists speak of "neural networks" they are invoking this biological metaphor - however loosely. This page traces the intellectual arc from a single hand-crafted neuron in 1943 all the way to the architecture behind today's large language models, with three live simulations so you can see the theory firing in real time.

Part I - The Single Neuron

In 1943, neurophysiologist Warren McCulloch and mathematician Walter Pitts published a landmark paper proposing a mathematical model of the neuron [1]. Their formal neuron was binary: it either fired or it did not, depending on whether the weighted sum of its inputs exceeded a threshold. Despite gross simplification, the McCulloch–Pitts neuron captured the essential computational idea - thresholded summation.

Frank Rosenblatt (1958) extended this into the perceptron - a single-layer machine that could adjust its weights via a supervised learning rule to classify linearly separable patterns [2]. The perceptron takes each input, multiplies it by a learned weight, adds a bias, and squashes the result with a sigmoid activation (σ) to produce a continuous output between 0 and 1.

Simulation: The Perceptron

This neuron has been pre-trained as an AND gate: it outputs ≈ 1 only when both inputs are 1. Toggle the inputs to see the weighted sum and output change in real time.

The excitement around the perceptron was short-lived. In 1969, Minsky and Papert rigorously demonstrated that a single layer cannot learn functions whose inputs are not linearly separable [3]. The canonical example - XOR - sent neural network research into a decade-long winter.

Part II - The XOR Problem

XOR (exclusive or) outputs 1 when exactly one input is 1. Plot the four truth-table rows on a plane: you cannot draw a single straight line separating the two 1s (at (0,1) and (1,0)) from the two 0s (at (0,0) and (1,1)). No matter what weights and bias you assign, a single neuron will always misclassify at least one case.

The solution is to add a hidden layer. The hidden neurons learn intermediate representations - essentially warping the input space into one where a final neuron can draw a separating line. In the network below, two hidden neurons act as NAND and OR gates respectively; their outputs feed an AND-like output neuron, producing XOR.

Simulation: XOR Network

A pre-trained 2→4→1 network. Line thickness and colour show connection weight (blue = positive, red = negative). Drag the sliders to explore the decision boundary.

The training algorithm that makes multi-layer networks learnable is backpropagation: a method for computing gradients through every layer and using those gradients to nudge all weights toward lower error. Rumelhart, Hinton & Williams made the technique widely known in 1986 [4]. With backpropagation, networks with hidden layers could finally be trained at scale.

Part III - Going Deeper

A single hidden layer is theoretically sufficient to approximate any continuous function (the universal approximation theorem), but in practice, depth matters enormously. Each additional layer learns a more abstract representation of the previous layer's output. In image recognition, early layers detect edges; middle layers combine edges into textures and shapes; deep layers recognise faces or objects. The same hierarchy emerges in language models: word patterns, phrases, grammar, world knowledge.

Simulation: Deep Network

Three hidden layers (2→5→4→3→1, 45 connections total). The weights are random - the point is the shape of the computation, not the task. Watch how the signal transforms as it passes through each layer.

Scaling this pattern up - more layers, more neurons, and vastly more training data - is what produces modern large language models. A GPT-style model is structurally this same architecture with hundreds of layers and billions of weights, plus one key addition: the attention mechanism, which lets every position in a sequence look at every other position before the layer-by-layer transformation begins. The result is that the network can relate distant words and concepts before compressing them into its representation.

Biological Plausibility: How Close Is the Analogy?

Modern artificial neural networks diverge from biology in important ways. Biological neurons communicate via discrete action potentials (spikes) rather than continuous activations; synaptic plasticity follows Hebbian and spike-timing-dependent rules rather than global gradient descent; and no neuroscientist has found a plausible cortical implementation of backpropagation.

Spiking Neural Networks (SNNs) attempt to close this gap by modelling individual spikes and their timing. Maass (1997) showed that SNNs are computationally more powerful than rate-coded networks in certain regimes [5]. SNNs run efficiently on neuromorphic hardware such as Intel's Loihi chip, which mimics the event-driven, low-power operation of biological neural tissue.

What Biology Still Teaches Us

Despite their differences, biological neural systems remain a wellspring of architectural inspiration. Convolutional neural networks (CNNs) were partly motivated by Hubel and Wiesel's discovery of oriented edge detectors in the visual cortex [6]. Attention mechanisms in transformers bear a functional resemblance to selective attention in primate cognition. And the study of circuit motifs in simple organisms like C. elegans continues to inspire network architecture research. The artificial network is a pale shadow of the biological original - but it is a shadow cast by the same light.