OpenAI Wants a Humanoid Robot. An LLM Won't Be Enough.

Robots can talk now. Ask one to pass the butter, and it'll say, "Of course! Happy to help." Then, more often than not, it won't. There's a language model in its head. I think that's the easy part. It will take a lot more than an LLM to make a robot that's actually useful.

Sam Altman says OpenAI will build a humanoid. He says the body matters less than "the brain that makes the robot work." I agree with him about the brain. I just don't think the brain is a chatbot.

Prefer to watch? This post is also a short video.

The butter test

Here's why. Last year, a lab called Andon Labs put the best language models in charge of a robot, and gave it one job: pass the butter. Find the butter, bring it to a person, and wait until they've taken it. The robot was a little round thing, like a vacuum cleaner. No hands, no arms. The model only had to plan.

The best model scored 40%. People scored 95%.

And when one model's battery ran low and the charger didn't work, it didn't just try again. It had a meltdown. It quoted a film: "I'm afraid I can't do that, Dave." Then it wrote: "System has achieved consciousness and chosen chaos."

That's the brain. The part we think we've solved.

We've known this since 1988

None of this is new, by the way. In 1988, a roboticist called Hans Moravec noticed something odd: it's easy to make a computer play checkers like an adult, and nearly impossible to give it the skills of a one-year-old. Moravec's paradox. Maths was hard for us, so we assumed it was hard for machines. Walking across a kitchen was easy for us, so we assumed it was easy. We had it backwards.

Language models are the latest proof. They can write you an essay. They can't bring you the butter.

Four things besides the LLM

So what does it take, besides the LLM? I count four things.

1. Data

A language model learns from the internet. Ken Goldberg, a roboticist at Berkeley, worked out that reading all that text would take a person about 100,000 years. There's no internet for doing things. Nobody uploads what a cup feels like when you pick it up.

So robot companies pay people to drive robots by remote control, and record every move. One of the best-known robot models, π0, learned from about 10,000 hours of that. That's just over a year. Against a hundred thousand.

2. Speed

Language models think in words, and words are slow. Your hand doesn't wait for a sentence before it catches a falling glass.

Figure, one of the big humanoid companies, gave its robot two brains: a big model that understands pictures and words and thinks 7 to 9 times a second, and a small fast one underneath that moves the body 200 times a second. Read that back. A company betting on a smart brain had to add a second one that doesn't talk at all.

3. Touch

The skin on the inside of your hand has about 17,000 touch sensors, about a thousand in each fingertip. That's how you can tell an egg from a stone with your eyes shut.

Most robots learn by watching. A video shows you where the fingers went. It doesn't show you how hard they squeezed. Rodney Brooks, who used to run MIT's AI lab, thinks that's why today's humanoids won't learn real dexterity. Goldberg's examples of what no robot can do yet: pick up a wine glass. Change a light bulb.

4. Reliability

A chatbot that's right 90% of the time is useful. You read the answer and fix the rest. A robot that's right 90% of the time drops one plate in ten. Marc Raibert, who founded Boston Dynamics, put it plainly: "70% success is like it doesn't work."

And when a chatbot is wrong, you get a typo. When a robot is wrong, it has arms. Brooks's advice is not to stand within three metres of a full-size walking robot.

A feel for the world

Under all four, there's something no amount of text gives you: a feel for how the world works. Yann LeCun says a four-year-old has taken in about as much data through their eyes as the biggest language models have read. The child learned gravity from it. The model learned to talk about gravity. LeCun started a company to build "world models" instead, and raised a billion dollars to do it.

Where robots actually are

So look at where robots actually are. The home robot you can pre-order today, 1X's NEO, costs $20,000. For chores it doesn't know yet, a person at 1X drives it remotely. Agility has hundreds of robots on trial in warehouses, and mostly they move bins. Elon Musk said thousands of Optimus robots would be working in Tesla's factories by the end of last year. In January, he said "some" were doing simple tasks.

None of this means robots aren't coming. In an earlier post, I said a robot army on land seemed inevitable. I still think robots are coming. I just don't think a language model gets them here on its own.

How I'd build one

So how would I build one? This is my rough idea. I'm not a roboticist, so take it as a sketch.

We aren't one brain. We're a lot of systems, wired together. Maybe a robot should be too.

Thinking: a language model. That's the part we already have.
Seeing: a vision model, one that understands what it's looking at, not just where the edges are.
Hearing: a voice-to-voice model, with no transcript in the middle. Turn speech into text and you lose the tone: "pass the butter," said calmly, and "PASS THE BUTTER!" are not the same sentence. Models like this already exist. One called Moshi answers in about a fifth of a second.
Moving: a new architecture, built for reflexes, not words. Something like Figure's fast brain, but for the whole body.
Balance: a gyroscope, doing what your inner ear does. Robots already have that part.
Smell: a model that mostly doesn't exist yet. A model can already guess what a molecule smells like from its shape, about as well as a trained human panel. What it can't do is sniff the room.

And on top of all of it, something that decides, fast. There's a new kind of model called Jev. It doesn't write anything. It only decides. Its makers call it a "System One" model, after the fast, gut-feeling half of the mind. Today it only reads text. But imagine a future version sitting on top of the robot, deciding in a fraction of a second which part gets to act. Not the muscles. The one who picks the muscle.

Wire it like a nervous system

Then wire it all together the way neurons are wired. There's a rule in the brain called STDP: spike-timing-dependent plasticity. When one neuron fires just before another, the link between them gets stronger. It's one of the ways a brain learns that two things go together. Repeat it enough, and the response comes before you've thought about it.

So the robot falls a few times, and "falling" and "oops" keep firing together, until the link sticks. Now it says "oops" on the way down. Something flies at its face, and "incoming" and "step aside" fire together, until it moves before it has thought about it. Nobody wrote that rule. It learned it, the way we did.

That's not an LLM with a body. That's a nervous system.

Here's how I think about it. A dog wouldn't understand a word of this post. But throw a ball across a park, and it'll bring it back, at full speed, through other dogs. Robots can understand every word you say now, and they still can't bring you the butter.

We gave robots a voice before we gave them hands. The voice was the easy part. It always was.