Cutting text into pieces

Everything up to Part 3 was AI that looks. Photos, points, tracks.

Now we move to AI that deals in words. The start is a bit odd, though. Computers don't know letters.

It's the same story as 2.1. A computer turned photos into numbers to see them. Text is no different.

But text has one extra step. You have to cut it first.

Why cut it up

You can't turn "This photo is a cat" into a single number. There's no way to number every sentence in the world.

Cut it into single letters and you go too far the other way. Split "photo" into p h o t o and the meaning is gone.

So you cut somewhere in between. Things that turn up together a lot stay joined; everything else comes apart.

One of those pieces is called a token.

The Korean side shattered

Same meaning, and the counts differ several times over.

The shape of it is stranger still. Every English piece is a readable word, while most of the Korean side is crumbs you can't read.

That's because single characters broke into several pieces. Take one piece on its own and it isn't a letter.

No. It was the rule

Press Made from Korean text.

It flipped. Now it's English that comes apart, one letter at a time.

The two rules were built exactly the same way. Only one thing differs — which text they were built from.

A cutting rule keeps whole only what it saw a lot of. Everything else ends up as crumbs.

And the AI in wide use today is mostly that first row. It was built from far more English than anything else.

Type in your own name

Go back to the lab and put your own name in the first box. Try both rules.

Did your name stay in one piece?

If it never appears in this guide, the Korean rule broke it up too. A rule only knows what it saw a lot of.

What's wrong with more pieces

An AI does one round of maths per piece. Twice the pieces, twice the maths.

So saying the same thing is slower and more expensive in some languages than others. It's easy to miss, and it adds up.

Now that it's cut

The sentence is now a row of numbered pieces.

In the next chapter we do exactly one thing with them. Guess the piece that comes next.

Making a machine very, very good at only that is where today's AI came from.