In most robot simulators, the world stops while the model thinks. The model receives a camera frame, takes its time to choose the next moves, and the simulator waits for the answer. On a real robot, nothing waits. The arm holds its last position, and the clock keeps running.

In simulation the clock stops while the model thinks. On a real robot the arm waits and the clock keeps running, so the same motion takes more time.

The robot models we test need between 0.2 and 1.5 seconds to plan their next moves. That is a large share of an episode, and a benchmark that pauses the world does not see it. We wanted to know what the pause hides. We ran two studies on RoboLab, NVIDIA's simulated benchmark of household tasks.

Study 1: the same models, with and without the pause

We ran seven models on the same 120 RoboLab tasks, 3 episodes per task, in two modes. In the first mode, the simulator waits for the model. In the second mode, the clock keeps running while the model thinks, as on a real arm. The second mode is how we score every eval: on our real Franka, in the sim evals we run for customers, and on the PhAIL RoboLab board.

Every episode of both modes is on PhAIL, with its video. The switch at the top of the board changes the mode: clock runs or sim waits.

Model Arm waits Success, sim waits Success, clock runs Change
Galaxea G0.5 45% 20.8% 16.7% −4.2 ± 3.2%
NVIDIA Cosmos3 Nano 40% 34.2% 26.4% −7.8 ± 4.6%
Ai2 MolmoAct2 36% 17.5% 13.9% −3.6 ± 4.6%
FLUX 3 Action 19% 42.2% 40.3% −1.9 ± 4.4%
NVIDIA GR00T N1.7 14% 21.4% 21.9% +0.6 ± 4.0%
π0.5 12% 28.6% 26.4% −2.2 ± 4.0%
OpenAI GPT-6 Astra not measured 48.9% 0.6% −48.3 ± 6.9%

"Arm waits" is the share of the episode in which the arm stands still while the model thinks. The ± value is the 95% range. A change in bold is larger than noise: its range does not include zero.

Unsurprisingly, the slow models lose and the fast ones barely change. Cosmos3 Nano is one of the two slowest robot models on the board: it needs about 1.5 seconds each time it plans, so its arm stands still for 40% of every episode. It also has the largest loss of the robot models. Galaxea G0.5 waits a little longer and also loses more than noise. MolmoAct2 is third on both counts, and its loss is at the edge of noise. The three fast models wait 12–19% of the time, and their change is inside noise.

GPT-6 Astra shows the extreme case. It is a general AI model, not a robot model, and it controls the arm by itself. With the sim paused, it has the best score on the board. With the clock running, it almost never succeeds.

Change in success rate against the share of each episode the arm waits, for six robot models. Cosmos3 Nano and Galaxea G0.5 wait longest and lose more than noise.

Thinking time works like slow motion

A model that thinks with the clock running does not get confused. It makes the same moves, just later.

Take each episode recorded with the sim paused and slow it down by the share of time that model waits. The result predicts the curve of the clock-running run along the whole episode. For Cosmos3 Nano, the prediction of the final partial-credit score lands within one point of the measured score.

The gap opens in the first seconds. At 20% of the time limit, Cosmos3 Nano with the clock running has only a third of the score it has with the sim paused. After that it stays behind, and the time limit cuts off the finishes that come late.

Cosmos3 Nano Sim waits Clock runs
Typical success comes at 34% of the time limit 59% of the time limit
Cosmos3 Nano success rate over the time limit. With the sim paused it reaches 34%. With the clock running it falls behind in the first seconds and ends at 26%.

Study 2: does stopping to think cost more than moving slowly?

Study 1 shows that thinking costs time. Does it cost anything more than time?

A robot that thinks with the clock running moves in bursts: it holds still, plays its next moves, then holds still again. Compare it with a robot that never stops but moves slower, at the same average pace. Both get through the same work in the same time. If the bursts cost something on their own, for example because the scene changes while the arm holds still, the first robot should do worse.

We tested this on Cosmos3 Nano, the robot model that loses the most in study 1. On average it thinks for 1.47 seconds per call and then plays 2.13 seconds of motion. We ran two setups at the same pace:

  • Fixed thinking. The arm holds still for 1.47 seconds on each call, then moves at normal speed.
  • Slow motion. The sim waits while the model thinks, and every move plays 1.69× slower. That is the same average pace with no stops.

We picked 7 RoboLab tasks where objects can move during a wait and where Cosmos3 Nano sometimes succeeds, and ran 51 episodes per task per setup, 714 episodes in total. Each success rate in the tables below opens the replays of its episodes.

Cosmos3 Nano, 7 tasks Fixed thinking Slow motion Change
Success 50.7% 52.1% −1.4 ± 9.7%
Median time to success 28.1 s 29.5 s

The time-to-success curves cross many times and end at almost the same place. Per task, the gaps go in both directions, and chance explains their spread:

Task Fixed thinking Slow motion Change
Put a white plastic bottle in the square pail 92% 96% −4 ± 9%
Put the red BBQ sauce bottle in the crate 92% 78% +14 ± 13%
Stack the yellow block on the red block 55% 63% −8 ± 19%
Stack the white mugs 43% 51% −8 ± 19%
Put three fruits on the plate 41% 59% −18 ± 19%
Stand the jug upright 24% 14% +10 ± 15%
Put all the green fruit on the plate 8% 4% +4 ± 9%

In RoboLab, a robot that stops to think does as well as a robot that moves slowly at the same pace.

Why, and what we test next

RoboLab's world is still while the robot waits. An object moves during a wait only if the arm just released it or bumped it. Nothing rolls, slides or arrives on its own. We checked MolmoSpaces too (its board), and its scenes are static in the same way. So a robot that stops to think loses nothing that a slow robot keeps.

A real workplace is different. A conveyor keeps running, a person reaches in, a part settles after the robot lets go. That is where stopping to think should cost more than slow motion, and these two studies cannot show it. The next test needs a scene where the world moves without the robot.

For static scenes, the full cost of thinking time is the time itself, and the slowest models pay the most. A benchmark that runs the clock measures it. A benchmark that pauses the world does not.