"Tokens" are the greatest Jedi mind trick of this decade.

"Tokens" are the greatest Jedi mind trick of this decade.
People talk about them like they're obvious.
Spoilers: they're not.

So what is a token?

They're not characters.
They're not words.
They're not sentences.

Sometimes, but not always, tokens are one character.
- "A"
- "?"
- "$"

Sometimes, but not always, tokens are a whole word.
- "the"
- "and"
- "because"

Sometimes it’s only part of a word.
- "under" + "standing"
- "play" + "ground"
- "inter" + "national"

... and sometimes a single emoji, a line break, or even a space can become its own token.

The exact definition also changes depending on which model you're using. Nope, they're not interchangeable.

Yet we've built an entire vocabulary around them.
- "How many tokens?"
- "What's your token limit?"
- "Can we reduce token usage?"

Weirder still, token count isn't a measure of complexity:
- Some incredibly difficult tasks require very few tokens.
- Some trivial ones require a lot.

But still, we squabble about them...
- We celebrate when we save a few thousand of them.
- We panic when we use too many.
- We price software with it.

My guess: Our future will criticize us for mistaking tokens for what mattered.