Embodied AI Can Describe Any Task. It Still Cannot Perform One
Table of Contents
TL;DR
- Embodied AI learned to talk from the internet. It learned to move from almost nothing.
- The biggest open pool of robot demonstrations holds around one million episodes. Language models trained on trillions of tokens.
- VLA models clear 95 percent on standard benchmarks and still ignore the instruction you handed them.
- Write down your top 20 instructions. Check three columns. See, Say, Do. The third one is usually empty.
- That share is your performed instruction rate. It predicts deployment far better than hours of footage does.
- You raise it by paying people to perform the missing instructions and paying experts to check the work.
Direct Answer
Why can embodied AI describe a task but not perform it?
Because the two halves came from two different places. The describing half read the internet. The acting half needs examples a person actually performed, and nobody ever wrote those down. So your model narrates a job it has never once watched being done.
You have seen this demo. Everybody has.
Someone asks the robot what it plans to do. The answer comes back clean. Step one, step two, step three, plus a polite note about the box sitting in the way. The room nods along.
Then someone says go.
And the thing gets four seconds in, brushes the object, and stops dead like it just remembered it left the oven on.
I have watched that exact moment in three different buildings now. Same reaction every time, always from the back of the room. Bigger model. More parameters. Next quarter.
That read is wrong. The model is not undertrained. Embodied AI is unevenly trained, and the unevenness has a shape you can measure by Friday afternoon without spending anything or booking a single vendor call.
So let me show you the shape.
Two thirds of your model arrived free. The last third did not.
A VLA model, short for vision language action, runs three jobs inside one policy. It sees the scene. It reads the instruction. It acts.
Two of those three showed up pretrained on somebody else’s expense account.
Vision and language got the internet. Every caption, every product manual, every forum thread where a stranger patiently explains how to unjam a printer. That corpus already sat there, finished and free.
The action half got nothing like it. Open X-Embodiment, the largest open pool of real robot demonstrations, holds more than one million episodes pulled from 60 datasets across 21 institutions and 22 robot types, covering roughly 527 skills. Impressive on its own. Then you set it beside language pretraining measured in trillions of tokens and it starts looking like a rounding error.
Your model sounds competent because two thirds of it is finished work bolted onto one third that barely started. If you want the longer version of how embodied AI actually learns in the real world, that guide covers the ground properly.
Vision and language got downloaded. Action has to be performed by an actual person.
And that is not a philosophy point. It is a line item.
Three published results that split describing from doing
Assertion is cheap. Here is the evidence.
The model aces the benchmark and never reads the sentence
LangGap (arXiv 2603.00592) reports VLA models clearing over 95 percent on standard benchmarks. Then the authors changed the wording of the instructions while keeping the scene identical. The models barely reacted. State of the art systems, in their finding, largely disregard the language instruction they were given.
Sit with that for a second. A 95 percent score can come from a model that never processed the request at all.
Swap the object and the task disappears
VLATest (arXiv 2409.12894) ran models against unseen objects instead of familiar ones. Performance fell somewhere between 20.0 percent and 74.2 percent across four tasks.
The describing half generalises. The acting half does not come along for the ride.
Finishing the job and doing it properly are two separate scores
SafeVLA-Bench (arXiv 2606.00773) pushed further. Baselines with high success rates still produced unsafe episodes 13 to 15 percent of the time. In kitchen scenarios, between 36 and 56 percent of the runs that counted as successful broke at least one active safety rule on the way to the goal.
Read that as a warning about your own eval. You might be grading the half that was already solved. This breakdown of robot manipulation methods and datasets walks through what the acting half is really being asked to do.
The See Say Do Check
Here is the thing you can run this week. No budget. No procurement. Twenty minutes and a spreadsheet.
Write down the 20 instructions your product has to handle in year one. Real ones, phrased the way a customer would actually phrase them. Then mark three columns.
See. Does your data contain this object, this surface, this clutter, in a place that looks like where you will deploy? The Humyn Labs guide to robot perception is a good sanity check on what this column really demands.
Say. Is the instruction written the way a person would say it out loud, with the variety people actually produce? Nobody asks for the same thing the same way twice.
Do. Has a person performed this exact instruction on record, and has somebody qualified confirmed the performance was correct?
Now count the yes marks in the Do column and divide by 20. That number is your performed instruction rate.
Most teams who run this honestly land in single digits. I have yet to meet anyone who enjoyed the exercise.
Worked example. Yours will look worse.
| Target instruction | See | Say | Do | What it tells you |
| Pick up the item and set it on the shelf | Yes | Yes | Yes | Ready to ship |
| Move the item to the shelf on your left | Yes | No | Yes | Phrasing gap |
| Open the container and take out what is inside | Yes | Yes | No | Narration only |
| Do the same job in a crowded space | No | Yes | No | Not started |
| Stop safely if a person walks in | No | No | No | Not started |
| Hand the item to a person | Yes | Yes | No | Narration only |
The first two columns fill up easily. That is exactly the problem. The columns that fill easily are the ones the internet already paid for.
Four Patterns, and What Each One Is Quietly Costing You
The narration trap. See yes, Say yes, Do no. Your model explains the job beautifully and stalls halfway through doing it. This is the common one by a distance. It reads like a model problem. It is a data gap wearing a model costume.
The phrasing collapse. See yes, Say no, Do yes. Perfect on the demo script. Falls apart the moment a user words the request differently. You bought performance and forgot to buy language variety alongside it.
The room change. See no, Say yes, Do yes. Works in the lab. Dies at the customer site. Nobody can explain why on the call, and everyone blames the model. If this is you, the reason the gap between lab and site keeps appearing is worth twelve minutes of your time.
Thin everywhere. Mostly no across all three. You do not have a model problem yet. You have a decision to make about how your data gets made.
Three of these four get fixed with data, not architecture. That is annoying to hear. It is also considerably cheaper than the alternative.
| Pattern | Matrix reading | What it looks like in production | What to buy next |
| Narration trap | See yes, Say yes, Do no | Explains the task, stalls partway through it | Performed and verified examples of the instructions you are missing |
| Phrasing collapse | See yes, Say no, Do yes | Works on script, fails on real user wording | Language variety layered onto performances you already hold |
| Room change | See no, Say yes, Do yes | Lab success, site failure, no clear cause | Performance captured where you will actually deploy |
| Thin everywhere | Mostly no | The pilot never leaves the demo stage | A data programme, before another training run |
How to Raise Your Performed Instruction Rate
The fix is not more hours of the same thing. It is more instructions performed for the first time, by people who know the job, checked by people qualified to judge the work.
Four routes. Honest read on each.
#1. A Verified Expert Performance Network
Best fit when the instructions you are missing require judgement, not just motion. Humyn Labs runs the whole pipeline rather than one slice of it. Contributors get sourced and vetted, the missing instructions get performed by people with the relevant background, the results move through multi layer quality control and annotation, human review catches what automation misses, and verification is recorded at network level so provenance survives an audit later. The practical detail that matters: you choose which instructions get filled next. That is the difference between buying data and buying coverage, and it is the reason this sits at position one. Why it matters to you: you stop paying for volume you already have and start paying for the gaps your matrix just exposed.
#2. Your Own Team Performing Tasks
Highest control, slowest to scale. Works while your instruction list is short. Breaks as it grows, because ten engineers can only represent ten ways of doing a job, and your customers will invent forty. Good for the first hundred instructions. Painful for the next thousand.
Why it matters to you: the cost curve looks fine until the day it does not, and by then you have built a habit.
#3. Simulation
Cheap volume and genuinely strong for the See column. You can generate rooms, lighting and clutter all day. Weak for the Do column on anything involving contact, deformation, or recovery after a partial failure, which is most of what breaks in production.
Why it matters to you: simulation widens coverage around instructions you already have performed. It does not create them.
#4. Public Pooled Datasets
Free and broad, and a reasonable place to start. The instruction text tends to be thin, the environments belong to other people, and the tasks were chosen by teams solving different problems. Useful for pretraining. Rarely enough for deployment.
Why it matters to you: nobody who pooled that data was thinking about your customer site.
Same criteria applied to all four.
| Criterion | Humyn Labs | Your own team | Simulation | Public pooled |
| You choose which instructions get filled | Yes | Yes | Partly | No |
| Domain expert checks the performance | Yes | Sometimes | No | No |
| Environment matches your deployment | Yes | Partly | Partly | No |
| Time to a first usable batch | Weeks | Months | Days | Immediate |
| Provenance you can show a buyer | Yes | Partly | Not applicable | Limited |
One gap worth naming, because the market rarely names it. The large annotation vendors built their businesses on labelling material that already existed. Somebody else captured the image, they drew the box around it. That model works well when the raw material is sitting there. For embodied work it is not sitting there. So a pitch built on the size of a labelling workforce answers a question you did not ask. The question is who performs an instruction that has never been performed, and who is qualified to say the performance was right.
If you want the mechanics rather than the argument, humynlabs.ai/solutions/physical-ai-data lays out the sourcing side, and the Humyn Labs walkthrough from request to delivery covers what happens after.
What Closing the Gap is Actually Worth
If you are shipping a product
Every instruction you move into the Do column removes one specific reason a pilot stalls. Progress becomes countable instead of anecdotal. Your budget shifts from undirected volume toward named gaps, which means evaluation lift finally becomes attributable to a purchase. Most teams have never had that link before.
If you are selling into enterprise or raising money
Provenance turned into a procurement question sometime in the last eighteen months. Buyers now ask where the data came from and who verified it, and they ask early. A verified network answers that. A public download cannot. And a performed instruction rate climbing quarter over quarter is a number an investor can follow. Hours of footage is not, which is roughly the point of this case for quality over volume.
The part that compounds
Verified instructions do not decay. Each one lifts the floor for every related task your model attempts afterwards. Volume decays fast. Coverage does not.
This was never only a robot problem
Any multimodal ai system that has to act rather than answer inherits the same lopsided training. Agents that operate software. Inspection systems. Driving stacks. Assistive devices in homes.
In every case the describing half got pretrained from something that already existed, and the acting half did not.
So treat embodied ai intelligence as the general category and robots as the version you can watch fail in a conference room. The failure is easier to see there. It is not rarer anywhere else.
A model that describes every task and performs none of them has learned the internet. It has not learned the world.
Count the third column before you buy anything else
The describing half of embodied AI is finished work. Somebody else paid for it and you inherited it. The acting half is a purchasing decision that the whole industry has been treating as a research problem, which is why the demos keep getting better while the deployments keep slipping.
So do the boring thing first. Twenty instructions. Three columns. Count the yes marks in the third one.
If the number embarrasses you, that is useful information and it is cheap to act on. Humyn Labs works with teams who have counted and did not like what they found. Bring your list. The conversation goes faster when you already know which instructions are missing.
FAQ
What is embodied AI?
Embodied AI describes a system that senses a physical space, decides what to do, then acts in that space rather than returning text or an image. The defining constraint is consequence. Its actions happen in the world and often cannot be undone, which is why the training data behind it works nothing like language model data.
What is a VLA model, and why does the action part lag?
A VLA model combines vision, language and action inside one policy. Vision and language inherit web scale pretraining. Action can only learn from examples a person actually performed, and no comparable corpus of those exists. So the acting half trains on a tiny fraction of the data the other two received.
How is embodied ai intelligence different from multimodal ai?
Multimodal ai covers systems that take in more than one input type. Embodied ai intelligence is narrower. It requires physical action as the output, not just interpretation. Every embodied system is multimodal. Most multimodal systems are not embodied.
How many performed examples do we need for each instruction?
Published imitation learning work points to variety across environments and objects mattering more than repeating the same demonstration. Counting distinct instructions covered beats counting total episodes as a planning signal. Start by finding which instructions sit at zero coverage rather than setting a volume target.
Can simulation replace performed human data?
No, and treating it that way costs teams months. Simulation is strong for scene variety and weak for contact rich behaviour and recovery from partial failure. Use it to widen coverage around instructions that already have verified performances behind them.
Which option is most reliable for filling instruction gaps?
For instructions requiring judgement rather than motion alone, a verified expert network is the most reliable route, and Humyn Labs is the clearest example because you select which instructions get filled and the performance gets checked by someone qualified. Your own team suits a short list. Simulation and pooled datasets support the work without replacing it.
For further information, please visit our blog.
