A 10 trial demo is not evidence. Seven successes out of ten gives a 95% Wilson interval of 39.7% to 89.2%, a 49.5 point band, and a policy whose true success rate is 70% shows 9 or 10 out of 10 about 14.9% of the time, so the impressive demo is a routine event. The published work that evaluated policies properly spent 1,800 rigorously controlled real-world trials at 50 rollouts per task per policy per condition and 200 per condition in simulation, and Open X-Embodiment ran 3,600 evaluation trials across 6 robots to support its claims. Grader disagreement is part of your error bar: a QA round covering roughly 27% of nearly 2,700 real-world rollouts found a 2.31% discrepancy on scored success and 6.25% on the overall rubric question. Benchmark numbers do not substitute: a June 2026 audit showed a 0.09B parameter probe with no language encoder scoring at or near reported state of the art on LIBERO, and most reported gains there not provably statistically significant. Detecting a 10 point difference between a 60% and a 70% policy at 80% power needs about 356 trials per arm, which is why sequential testing and a simulation screen exist. Start this week by recomputing your last acceptance demo as a Wilson interval and putting the two numbers side by side.
Seven successes out of ten trials. That is the number under most robot policy demos, and it supports almost nothing. Run the Wilson score interval on 7 out of 10 at 95% confidence and you get 39.7% to 89.2%: a 49.5 point band that contains "fails one attempt in two" and "nearly production ready" at the same time. How many trials it takes to evaluate a robot policy is an ordinary sample size question, and the honest answer is far larger than the count in the video.
Nothing about the policy changes as you add trials. Only the evidence does. The same 70% point estimate measured over 50 trials narrows to 56.2% to 80.9%, and over 200 trials to 63.3% to 75.9%. If your acceptance decision would flip anywhere inside the first band, you have not measured the thing you are about to buy.
The second problem is that the demo you were shown was probably not cherry-picked, and it still misleads. A policy whose true success rate is 70% produces 9 or 10 successes in a 10 trial block about 14.9% of the time. A true 80% policy runs the table, 10 for 10, about 10.7% of the time. Nobody has to cheat for a mediocre policy to look excellent in a short block, which is why the trial count belongs in the acceptance clause rather than in the post-mortem.
What a 10 trial block can and cannot distinguish
import math
def wilson(successes, trials, z=1.96):
p = successes / trials
denom = 1 + z * z / trials
centre = (p + z * z / (2 * trials)) / denom
half = (z / denom) * math.sqrt(p * (1 - p) / trials + z * z / (4 * trials * trials))
return max(0.0, centre - half), min(1.0, centre + half)
print(wilson(7, 10)) # (0.3968, 0.8922)
print(wilson(35, 50)) # (0.5625, 0.8089)
print(wilson(140, 200)) # (0.6329, 0.7590)Two things follow. The band narrows with the square root of the trial count, so the fifth hundred trials buys a fraction of what the first fifty did. And a 10 point total width at a 70% rate needs roughly 320 trials per condition, which is the number to have in mind before anyone promises a percentage in a slide.
A 10 trial block is not useless. It is a screen. A policy that goes 3 for 10 has a 95% interval of roughly 10.8% to 60.3% and is genuinely broken, and killing it after 10 trials saves a week of floor time. Use short blocks to eliminate, never to select.
At 10 trials each, a head to head between two policies that genuinely differ by 10 points is close to a coin flip. That is the arithmetic behind every bake-off that reversed itself the following week.
The formal version uses a two-proportion power calculation at 80% power and a 5% two-sided false positive rate, computed with the standard normal approximation:
Read the last row before you plan anything. A 5 point improvement in the 70s is not detectable at any trial count a hardware programme will fund. If your roadmap depends on proving 5 point deltas on real robots, the roadmap is the thing to change, not the evaluation.
One policy: how wide is the interval
Every number below is the Wilson score interval at 95% confidence, z set to 1.96, computed for this post. It is four lines of code, and running it on your own last demo is the single fastest way to see the problem. Use Wilson rather than the textbook normal approximation. At the counts robot evaluation actually runs, and especially near 0 or near a clean sweep, the simple formula produces intervals that run past 100% and understate uncertainty exactly where you care most.
Two policies: ranking costs far more than characterising
This is where most evaluation budgets break. Characterising a single policy to a usable width is expensive. Deciding which of two policies is better is much more expensive, because the noise in both estimates compounds. Run the comparison directly. Give a policy with a true 60% success rate and a policy with a true 70% success rate the same number of trials, and count how often the worse one ties or wins.
| Trials at a 70% observed rate | 95% Wilson interval | Total width |
|---|---|---|
| 10 | 39.7% to 89.2% | 49.5 points |
| 20 | 48.1% to 85.5% | 37.4 points |
| 30 | 52.1% to 83.3% | 31.2 points |
| 50 | 56.2% to 80.9% | 24.6 points |
| 100 | 60.4% to 78.1% | 17.7 points |
| 200 | 63.3% to 75.9% | 12.6 points |
| 500 | 65.8% to 73.9% | 8.0 points |
| Trials per policy | Chance the 60% policy ties or beats the 70% policy |
|---|---|
| 10 | 40.5% |
| 20 | 30.7% |
| 50 | 17.1% |
| 100 | 7.9% |
| 200 | 2.0% |
| Difference you want to detect | Trials per policy per condition |
|---|---|
| 50% vs 70% | 93 |
| 70% vs 90% | 62 |
| 70% vs 80% | 294 |
| 65% vs 75% | 329 |
| 60% vs 70% | 356 |
| 90% vs 95% | 435 |
| 70% vs 75% | 1,251 |
What a defensible evaluation actually cost
A July 2025 evaluation of multitask manipulation policies ran 1,800 rigorously controlled real-world trials as its headline experiment. The design is worth copying literally: 50 rollouts per task per policy per condition in the real world, and 200 rollouts per task per policy per condition in simulation. Across the whole programme that came to nearly 2,700 real-world rollouts.
Note what the unit is. Not 50 rollouts for the policy. Fifty rollouts per task, per policy, per condition. Four tasks, two policies and three conditions is 24 cells, and 24 cells at 50 rollouts is 1,200 hardware trials before anyone writes a sentence about which policy won.
The pattern is not unique to that study. Open X-Embodiment states plainly that its conclusions rest on 3,600 evaluation trials across 6 different robots. Evaluation at this standard is a line item comparable to data collection, and it is usually missing from the plan entirely. We sized the collection side of the same budget in how much data you need to train a robot policy; the evaluation side is the half that gets discovered late.
The grader is part of your error bar
Success on a manipulation task is not read off a sensor. Somebody watches a video and decides whether a near miss that self-corrected counts, whether the object landed inside tolerance, whether the operator touched the cell.
That same July 2025 evaluation put a number on the ambiguity. A quality assurance round covering roughly 27% of the nearly 2,700 real-world rollouts found a 2.31% discrepancy on scored success and a 6.25% discrepancy on the overall rubric question. Two competent graders looking at identical footage disagreed on the binary outcome for about one rollout in forty three, and on the broader judgement for about one in sixteen.
Put that beside your claimed improvement. A 20 point delta survives 2.31% grader noise without difficulty. A 3 point delta does not, and reporting it to three significant figures is worse than reporting nothing. The general problem of scoring outputs that have no single right answer is the same one we work through in testing AI systems where there is no right answer, and the same discipline applies here: a written rubric, an independent scorer, and a re-scored sample with the disagreement rate published alongside the result.
Three rules make the grading defensible. Write the rubric before the first rollout, including the near-miss and operator-intervention cases, since a rubric written after you have seen the failures is a rubric fitted to them. Have someone who did not train the policy do the scoring. Re-score a fixed fraction, 25% is a reasonable target given the published precedent, and report the discrepancy rate as part of the result rather than in an appendix. The tradeoffs between human scoring and automatic metrics are worked out in human evaluation versus automated metrics.
A benchmark number is not acceptance evidence
If a vendor answers your trial-count question with a benchmark score, the answer is not a smaller version of what you asked for. It is a different claim.
An audit published in June 2026 examined LIBERO, CALVIN, SimplerEnv, RoboCasa and RoboTwin 2.0 against four named failure modes: shortcut solvability, insufficient statistical power, progressive overfitting, and dataset-specific performance. Two of its findings should change how you read a datasheet.
First, a 0.09B parameter probe with no language encoder at all scores at or near reported state of the art on LIBERO. A model that cannot read the instruction matches models that can, which means the benchmark is substantially solvable by a shortcut rather than by following language. Second, most reported gains on LIBERO are not provably statistically significant, so the leaderboard ordering is partly noise that everyone has agreed to read as progress.
The CALVIN result is the most operationally useful of the set. Randomising block poses within the training range, not outside it, drops performance for every tested policy. The policies had memorised specific configurations rather than learned the task, and the diagnostic that exposed it is a perturbation any buyer can demand: move the fixture within the range you already agreed on, and re-run.
Run those four diagnostics against whichever simulated benchmark a vendor quotes at you, and let the result decide how much weight the score carries. Even a benchmark that survives all four is still not an acceptance test, because none of these benchmarks contains your gripper, your camera baseline, your control rate or your parts.
Sequential testing: stop early without inflating false positives
The trial counts above assume a fixed sample chosen in advance. Everyone wants to stop as soon as the answer is obvious, and the naive version of that is exactly how false positives get manufactured: re-checking a fixed-sample p-value after every block and stopping the moment it dips under 0.05 pushes the real error rate well past 5%.
The fix is to commit to a sequential test before the first rollout. A policy comparison method published in March 2025 was built for this setting, starting from the observation that feasible real-world sample sizes are 10 or 50. It reduces the number of evaluation trials by up to 32% versus baseline sequential methods and saved more than 160 simulation rollouts in a multi-task comparison, with the stopping rule doing the work rather than a smaller trial budget.
The July 2025 evaluation used the same family of tools, running pairwise sequential tests across policies with a Bonferroni correction to hold a 95% confidence level across the whole comparison. That correction is not optional bookkeeping. Comparing k policies means k(k-1)/2 pairwise tests, and the family-wise error rate climbs fast:
Six candidate checkpoints compared pairwise without correction gives you a better than even chance of declaring a winner that is not one. The correction costs trials, which is another reason to shortlist in simulation and bring two or three candidates to hardware, not six.
| Policies compared | Pairwise tests | Chance of at least one false positive, uncorrected | Bonferroni alpha per test |
|---|---|---|---|
| 2 | 1 | 5.0% | 0.0500 |
| 3 | 3 | 14.3% | 0.0167 |
| 4 | 6 | 26.5% | 0.0083 |
| 6 | 15 | 53.7% | 0.0033 |
Size the simulation screen and the hardware gate separately
Simulation rollouts and hardware trials are not interchangeable units, and the 4 to 1 ratio in the published design (200 simulated rollouts per condition against 50 real ones) reflects cost, not equivalence.
The asymmetry is the point: a simulated benchmark can reject a candidate cheaply and cannot certify one, because the simulator carries a bias you cannot quantify without paired real trials. Work published in October 2025 formalises exactly that pairing, treating policy evaluation as a prediction-powered inference problem that uses paired real and simulation evaluations to correct simulator bias and return confidence intervals instead of point estimates. Its motivating observation is the one this whole post is about: robot policies are usually evaluated on a small number of hardware trials with no statistical assurances at all.
If you are still choosing where the rollouts come from in the first place, the cost structure behind each source is in teleoperation versus simulation versus human video.
| Simulation screen | Hardware gate | |
|---|---|---|
| Purpose | Eliminate candidates, find the shortlist | Certify the policy that ships |
| Rollouts per task per condition | 200 | 50, more if the delta is small |
| Marginal cost | Compute, parallelisable overnight | Floor time, an operator, a reset |
| What it can prove | That a candidate is not worth hardware time | That the deployed policy meets a bound |
| Main failure | Shortcut solvability, sim-to-real bias | Too few trials, ambiguous rubric |
| Correct verdict | Reject | Accept or reject |
Why the evidence has to be produced inside your perimeter
For a regulated production line, this stops being a statistics question and becomes an evidence question, and that changes the design in three concrete ways.
The rollout footage is the evidence. In a pharmaceutical fill line, a medical device cell or a defence supplier's machine shop, the video that a grader scores contains the product, the fixture geometry, the process and often the operator. That footage is the artifact a qualification file is built on, and it cannot go to a vendor cloud for scoring. So the grading pipeline, the rubric tooling and any model-assisted pre-scoring have to run inside the same boundary as the cell. The rest of the physical AI pillar covers where that boundary usually sits.
The condition matrix is site-specific, so the vendor's trial count never transfers. A supplier's 50 rollouts per condition were run in the supplier's cell with the supplier's lighting, parts and fixture tolerances. Your conditions are different, and every condition you add multiplies the trial budget. Fifty rollouts across four tasks and three conditions is 600 hardware trials on your floor, and that number belongs in the commissioning schedule as line downtime rather than appearing as a surprise during qualification.
The confidence bound is what the auditor reads. A validation file that records "the policy succeeded in 47 of 50 trials, 95% lower bound 83.8%, rubric v3, 25% re-scored with a 2% grader discrepancy" is defensible years later. A file recording "94% success rate" is a number with no attached uncertainty, and it will not survive a question about how many trials produced it. Build the evaluation harness so those fields are emitted automatically, the same discipline we describe in evals-driven development.
Put the trial count in the acceptance clause
The practical output of all of this is a few lines in a contract and a config file, agreed before an integrator ships anything.
acceptance:
tasks: [tray_load, tray_unload, tote_transfer]
conditions: # every entry multiplies the trial budget
- {lighting: overhead_only, clutter: none, part: sku_a}
- {lighting: overhead_only, clutter: light, part: sku_a}
- {lighting: mixed_daylight, clutter: light, part: sku_b}
trials_per_task_per_condition: 50
pass_rule:
statistic: wilson_lower_bound # not the point estimate
confidence: 0.95
threshold: 0.80 # requires >= 45/50 in a cell
scoring:
rubric: rubric_v3.md # frozen before the first rollout
grader: independent # not the team that trained the policy
qa_rescore_fraction: 0.25
max_grader_discrepancy: 0.05 # else the cell is re-scored, not re-run
stopping_rule: sequential # declared in advance, no peeking
artifacts: [per_rollout_log, rubric_scores, qa_discrepancy, wilson_bounds]The pass_rule line is the one that carries the weight. A threshold on the point estimate is satisfied by 7 out of 10. A threshold on the 95% lower bound at 50 trials requires 45 successes, because 45 out of 50 gives an interval of 78.6% to 95.7% and 44 does not clear 80%. Same words, completely different purchase.
Two clauses save the most arguments later. Fix the rubric and the grader before the first rollout, so nobody negotiates the definition of success after seeing the failures. And write the condition matrix explicitly, because "it works" is a claim about a set of conditions, and the set nobody wrote down is always smaller than the one the buyer imagined.
This week, do one thing: take the last robot policy result somebody showed you, put the raw successes and trials into the Wilson function above, and write the interval next to the headline percentage in the same document. If the interval is wide enough that your decision would change at either end, you do not have a policy problem yet. You have a sample size problem, and it is much cheaper to fix.
FAQ
Quick answers to the questions this post tends to raise.
