What is physical AI evaluation?

Why physical AI evaluation is hard

Different environments, different architectures, tiny samples. Most claims you read are unfalsifiable.

The question is how good these models actually are on real commercial tasks – not demos, not simulation, not success rates on 10 tries.

Robot evals run ~20 trials with a single success rate and no confidence intervals. At that scale, today's top models are statistically indistinguishable – most reported differences are noise. On the same data, four reasonable scoring choices produce three different "winners."

We know sim and real scores correlate. What we don't know is how much a sim rank is worth on real hardware, and we don't think anyone does.

How we do it

We test robot AI models on real robots in our lab. Real hardware. Blind runs. Public data.

• One real robot, a Franka FR3.
• The operator is blind to the model, and every try is on video.
• We measure what operations people care about: units per hour and time between failures.
• A distributional methodology that separates close models with ~30x fewer trials.
• The benchmark, the dataset and the code are public.

See the results on PhAIL, our public leaderboard for physical AI. Every run on it has full video and telemetry. To test your model without publishing the results, ask us for a private evaluation.