AI that sees sound

The enrolled person mostly works. Anyone else barely does.

That is the problem speech recognition couldn't crack for seventy years. Every voice is different. Even the same person sounds different with a cold.

But how does a machine handle sound in the first place?

Turn the sound into a picture

Here's where the last chapter comes back.

Numbers are all a computer has, so sound has to become numbers too. Turn sound into numbers and you get one very long row — more than ten thousand numbers per second.

Stare at that row and you see nothing. So instead:

• chop the sound into very short slices

• for each slice, measure how strong the sound is at each height

• stand that up as a vertical stripe, and lay the stripes side by side

Now you have a picture. It's called a spectrogram.

Reading it takes three facts. Up and down is how high the sound is, left to right is time, and brighter means louder.

Same word, similar pattern. Not identical, though — nobody says the same word the same way twice.

Try whistle and clap too. Nothing like speech: a whistle is one smooth horizontal line, a clap is a vertical stripe.

Which takes us back to 2.2

Once sound is a picture, something neat happens.

In 2.2 you swept a filter across a photo to find features. You can do exactly that to this picture.

Sweep a filter across a spectrogram and out come the features of the sound. Stack layers and it spots bigger things. What was lines → patterns → shapes in a photo becomes short sounds → bits of words → words in audio.

Same method. Only the material changed.

That's the end of seeing and hearing

Part 2 is done. Photos and sounds both end up as tables of numbers, a filter sweeps across them, and stacked layers spot bigger and bigger things.

But who decided the numbers inside that filter? That's the question we parked at the end of 2.2. Part 3 answers it.