How a neural network recognises a digit, step by step

On the page How a neural network recognises digits a real, if small, neural network is running. We take one handwritten 3 from the MNIST test set and follow it through the network step by step. Every number in this article is a real value of that network, not an illustration.

Step 1: a picture becomes numbers

The network does not see a picture the way we do. It gets a row of numbers, one for each pixel of a 28 × 28 picture: 784 numbers in all. A white pixel is 0, a black one is 1, and the grey edges of the strokes are somewhere in between. Of the 784 pixels of our digit, 132 have some ink; the rest are zeros.

Before a drawing from the page reaches the network, it is prepared the way the MNIST digits once were: shrunk to fit a 20 × 20 box and moved so that the centre of its ink sits in the middle. A small drawing and a large one look the same to the network.

Step 2: a neuron adds up

The first layer has 32 neurons. Each of them looks at every pixel and multiplies it by a weight of its own. The weights of one neuron form a pattern: positive weights (blue on the page) are places where ink raises its sum, negative ones (orange) places where ink lowers it. The neuron then adds everything up, adds its bias and passes the result on only if it is positive. Otherwise it passes on zero and stays off. This rule is called ReLU.

For our digit, neuron 31 shines the most. Ink on its blue places added +6.07, ink on its orange places -2.56, and its bias is +0.11. The sum is 3.62, and since it is positive, the neuron passes it on.

Of the 32 neurons of the first layer, 14 light up. Each reacts to a different short stroke in a different place, as their patterns on the page show. These numbers are all the network "sees": no eyes, only weighted sums.

Step 3: strokes become larger shapes

The second layer has 16 neurons, and it no longer sees pixels. Its input is how brightly the neurons of the first layer shine, and it calculates in exactly the same way: multiply by a weight, add up, add the bias, turn a negative sum into zero. It combines simple strokes into larger parts of a digit. Of the 16 neurons of the second layer, 8 light up for our digit.

What exactly each neuron of the second layer "means" cannot be put simply. The network made up these in-between steps itself while it learned, and nobody named them.

Step 4: ten sums become one answer

The last layer has 10 outputs, one for each digit. Each adds up the neurons of the second layer with its own weights. The output for the digit 3 has a sum of 14.97; the next highest is the output for 5, with 4.38.

The softmax function then turns all ten sums into shares that add up to 100 %. The higher the sum, the larger the share. The digit 3 gets more than 99.9%, so the network picks it as its answer.

How the network learned its weights

The network has 25,818 learned numbers in all: 25,760 weights and 58 biases. Nobody set them by hand. At first they were close to random, and the network guessed blindly.

Then it was given 60,000 handwritten digits with the right answers. After every batch of 64 digits, we measured how wrong it was and nudged each weight a little in the direction that makes the error smaller. This is called backpropagation and gradient descent. The network went through the whole set 30 times.

So that the network also copes with digits drawn with a mouse, every training digit was moved, turned, slanted and made thinner or thicker a little at random on every pass. The network never saw exactly the same picture twice.

How often it is wrong

We tested the network on 10,000 test digits it never saw while learning. It answered 9,780 of them correctly, or 97.8%.

Digit	Test digits	Right	Most common mistake
0	980	98.9%	8 (3×)
1	1,135	98.9%	2 (4×)
2	1,032	98.2%	7 (6×)
3	1,010	98.0%	5 (12×)
4	982	97.5%	9 (13×)
5	892	97.8%	3 (11×)
6	958	98.3%	0 (4×)
7	1,028	97.9%	2 (12×)
8	974	97.2%	2 (4×)
9	1,009	95.3%	3 (14×)

The network does best with 0 (98.9% right) and worst with 9 (95.3%). Its most common mistake in the whole test set: it answered 3 instead of 9, 14 times.

A small network against larger ones

The paper in which LeCun and colleagues presented MNIST in 1998 compares many methods on the same test digits. A simple linear classifier, where every pixel leads straight to the ten outputs, was wrong 12% of the time. A network with two hidden layers of 300 and 100 neurons made 3.05% errors, and 2.5% when it also learned from artificially distorted digits. The convolutional network LeNet-5, with 60,000 parameters, was wrong only 0.95% of the time.

Our network has 25,818 parameters and is wrong 2.2% of the time. A convolutional network uses the same small stroke detector at every place in the picture, so it does not have to learn a stroke separately for each position. The networks that recognise speech or write text today work on the same principle of weighted sums, only they are many orders of magnitude larger.

Why it misreads your drawing

• It only knows digits like the MNIST ones. They were written by employees of the US Census Bureau and by American high school students. A 1 with a long upstroke or a crossed 7 may do worse.
• It cannot say "I don’t know". It always picks one of the ten digits.
• A mouse draws differently from a pen. Strokes made with a mouse or a finger look different from the pen strokes on paper that MNIST was made from.

When the network misreads your digit, look at step 2 on the page to see which neurons of the first layer lit up, and at step 4 to see which outputs they pushed up. Often you can see straight away which stroke led it astray.

Try it yourself on the page How a neural network recognises digits: draw a digit and click any neuron.