What Does a Robot Lose While It Thinks?
In most robot simulators, the world stops while the model thinks. The model receives a camera frame, takes its time to choose the next moves, and the simulator waits for the answer. On a real robot, nothing waits. The arm holds its last position, and the clock keeps running.
The robot models we test need between 0.2 and 1.5 seconds to plan their next moves. That is a large share of an episode, and a benchmark that pauses the world does not see it. We wanted to know what the pause hides. We ran two studies on RoboLab, NVIDIA's simulated benchmark of household tasks.
Study 1: the same models, with and without the pause
We ran seven models on the same 120 RoboLab tasks, 3 episodes per task, in two modes. In the first mode, the simulator waits for the model. In the second mode, the clock keeps running while the model thinks, as on a real arm. The second mode is how we score every eval: on our real Franka, in the sim evals we run for customers, and on the PhAIL RoboLab board.
Every episode of both modes is on PhAIL, with its video. The switch at the top of the board changes the mode: clock runs or sim waits.
| Model | Arm waits | Success, sim waits | Success, clock runs | Change |
|---|---|---|---|---|
| Galaxea G0.5 | 45% | 20.8% | 16.7% | −4.2 ± 3.2% |
| NVIDIA Cosmos3 Nano | 40% | 34.2% | 26.4% | −7.8 ± 4.6% |
| Ai2 MolmoAct2 | 36% | 17.5% | 13.9% | −3.6 ± 4.6% |
| FLUX 3 Action | 19% | 42.2% | 40.3% | −1.9 ± 4.4% |
| NVIDIA GR00T N1.7 | 14% | 21.4% | 21.9% | +0.6 ± 4.0% |
| π0.5 | 12% | 28.6% | 26.4% | −2.2 ± 4.0% |
| OpenAI GPT-6 Astra | not measured | 48.9% | 0.6% | −48.3 ± 6.9% |
"Arm waits" is the share of the episode in which the arm stands still while the model thinks. The ± value is the 95% range. A change in bold is larger than noise: its range does not include zero.
Unsurprisingly, the slow models lose and the fast ones barely change. Cosmos3 Nano is one of the two slowest robot models on the board: it needs about 1.5 seconds each time it plans, so its arm stands still for 40% of every episode. It also has the largest loss of the robot models. Galaxea G0.5 waits a little longer and also loses more than noise. MolmoAct2 is third on both counts, and its loss is at the edge of noise. The three fast models wait 12–19% of the time, and their change is inside noise.
GPT-6 Astra shows the extreme case. It is a general AI model, not a robot model, and it controls the arm by itself. With the sim paused, it has the best score on the board. With the clock running, it almost never succeeds.
Thinking time works like slow motion
A model that thinks with the clock running does not get confused. It makes the same moves, just later.
Take each episode recorded with the sim paused and slow it down by the share of time that model waits. The result predicts the curve of the clock-running run along the whole episode. For Cosmos3 Nano, the prediction of the final partial-credit score lands within one point of the measured score.
The gap opens in the first seconds. At 20% of the time limit, Cosmos3 Nano with the clock running has only a third of the score it has with the sim paused. After that it stays behind, and the time limit cuts off the finishes that come late.
| Cosmos3 Nano | Sim waits | Clock runs |
|---|---|---|
| Typical success comes at | 34% of the time limit | 59% of the time limit |
Study 2: does stopping to think cost more than moving slowly?
Study 1 shows that thinking costs time. Does it cost anything more than time?
A robot that thinks with the clock running moves in bursts: it holds still, plays its next moves, then holds still again. Compare it with a robot that never stops but moves slower, at the same average pace. Both get through the same work in the same time. If the bursts cost something on their own, for example because the scene changes while the arm holds still, the first robot should do worse.
We tested this on Cosmos3 Nano, the robot model that loses the most in study 1. On average it thinks for 1.47 seconds per call and then plays 2.13 seconds of motion. We ran two setups at the same pace:
- Fixed thinking. The arm holds still for 1.47 seconds on each call, then moves at normal speed.
- Slow motion. The sim waits while the model thinks, and every move plays 1.69× slower. That is the same average pace with no stops.
We picked 7 RoboLab tasks where objects can move during a wait and where Cosmos3 Nano sometimes succeeds, and ran 51 episodes per task per setup, 714 episodes in total. Each success rate in the tables below opens the replays of its episodes.
| Cosmos3 Nano, 7 tasks | Fixed thinking | Slow motion | Change |
|---|---|---|---|
| Success | 50.7% | 52.1% | −1.4 ± 9.7% |
| Median time to success | 28.1 s | 29.5 s |
The time-to-success curves cross many times and end at almost the same place. Per task, the gaps go in both directions, and chance explains their spread:
| Task | Fixed thinking | Slow motion | Change |
|---|---|---|---|
| Put a white plastic bottle in the square pail | 92% | 96% | −4 ± 9% |
| Put the red BBQ sauce bottle in the crate | 92% | 78% | +14 ± 13% |
| Stack the yellow block on the red block | 55% | 63% | −8 ± 19% |
| Stack the white mugs | 43% | 51% | −8 ± 19% |
| Put three fruits on the plate | 41% | 59% | −18 ± 19% |
| Stand the jug upright | 24% | 14% | +10 ± 15% |
| Put all the green fruit on the plate | 8% | 4% | +4 ± 9% |
In RoboLab, a robot that stops to think does as well as a robot that moves slowly at the same pace.
Why, and what we test next
RoboLab's world is still while the robot waits. An object moves during a wait only if the arm just released it or bumped it. Nothing rolls, slides or arrives on its own. We checked MolmoSpaces too (its board), and its scenes are static in the same way. So a robot that stops to think loses nothing that a slow robot keeps.
A real workplace is different. A conveyor keeps running, a person reaches in, a part settles after the robot lets go. That is where stopping to think should cost more than slow motion, and these two studies cannot show it. The next test needs a scene where the world moves without the robot.
For static scenes, the full cost of thinking time is the time itself, and the slowest models pay the most. A benchmark that runs the clock measures it. A benchmark that pauses the world does not.
- Every RoboLab run, in both modes: phail.ai/benchmark/robolab
- Our eval code: github.com/Positronic-Robotics/positronic