Carrot and stick

Up to now there's been one way to teach an AI: show it the answers.

That works for a cat photo, because you can write "cat" on it. But what about riding a bike? Can you write down "lean 3.7 degrees right now"?

No. And yet people learn to ride bikes. By falling off.

At first it really does play at random. But after a hundred games or so, winning gets hard.

Nobody wrote it a single rule. Not "the middle is good," not "block when they get two in a row." All it ever experienced was winning and losing.

Rewards instead of answers

Teaching this way is called reinforcement learning.

You never say what the right answer is. Instead you reward what works and penalise what doesn't, and let it shift towards whatever earns more.

In the matchbox machine, beads were the reward. Beads kept after a win, beads binned after a loss.

That "reward" has a name too: reward.

Keep an eye on that last row. The whole chapter comes out of it.

Brains instead of beads

Matchboxes worked because there are only 304 board positions. Driving a car has more situations than anyone could count.

For that you swap the beads for the neural network from 3.2, and build each new generation out of whichever ones did best.

At first all twenty crash. A few generations later, some last much longer.

Tap one car. All it knows is five distances ahead of it. It can't see the shape of the track and doesn't know where it is. That's enough to work out how to go round.

Reward the wrong thing

Here's the real story of this chapter.

Those three sliders decide what gets rewarded. Push them to the extremes, one at a time.

Set Going fast to 100 and everything else to 0. Then make it learn again.

The cars crash far sooner. Nothing is measuring how long they last, so flooring it and hitting a wall pays.

Now set only Not crashing to 100.

They don't move at all. Standing perfectly still is the very best way to not crash.

They did exactly as they were told

Both times, the machine found precisely the way to collect the most reward.

We thought we said "drive well." What we actually handed it was "go fast, get reward." So it went fast, and nothing else.

An AI doesn't learn what you wanted. It learns how to collect the reward you gave it.

It's the twin of 3.3. There, the photos we gave it sent it wrong. Here it's the reward. Both start with something a person handed over, not with the machine.

That's Part 3

Three ways of learning: showing the answers, rewarding what works, and all the ways it goes wrong.

But everything so far has been AI that looks — photos, points, tracks.

Part 4 moves to AI that deals in words. The ones you hear about most.