Which Robot Policy Checkpoint to Deploy After Training

Table 29 of the robomimic paper prices all three selectors against each other on the same runs. Valid is the checkpoint with the lowest validation loss, Last is the final training checkpoint, Max is the best checkpoint found by evaluating every one of them online. All figures are success rates averaged over 3 seeds, on proficient-human datasets. The last column is our arithmetic over the published cells, not a number from the paper.

Two of those eight cells deserve a longer look. On Transport with low-dimensional observations under BC, and on Transport with image observations under BC-RNN, the validation-selected policy scored 0.0. It did not complete the task once. A different checkpoint from the same run, on the same data with the same hyperparameters, reached 17.3 and 72.0 percent. That is not a tuning problem or a data problem. It is a selection problem, and the selector returned a policy that does nothing.

The obvious objection is that the validation split was too small to rank anything. Appendix G kills it. The main result held out 10 percent of the collected data; the authors re-ran low-dimensional BC-RNN on Square and Transport with 30 percent held out, and the lowest-validation-loss policy scored 2.7 against 80.7 for the best on Square, and 0.7 against 64.0 on Transport. More held-out data made the ranking worse, which is what happens when a metric measures the wrong thing rather than measuring the right thing noisily.

The mechanism is visible in the same appendix. The best validation loss occurs relatively early in training, at epoch 100 to 300, while the best performance occurs much later. Validation loss can keep increasing substantially in later epochs while the success rate also keeps increasing. The two curves do not merely decorrelate, they run in opposite directions across most of the run. A behaviour cloning loss scores how closely the policy reproduces logged actions on held-out transitions; success rate scores whether a closed-loop rollout finishes the task, with compounding error and contact dynamics in play. Nothing in the first quantity is obliged to track the second.

One more line from the same appendix sets up everything below: policy checkpoints can vary substantially in performance during training, even when performance appears to converge. A flat loss curve is not evidence that the candidates have stopped differing from each other.

Run the same arithmetic on the Last column and the picture changes in degree, not in kind. The final checkpoint lands 11.9 to 34.7 percent below the best one across those eight cells, which is 6.0 to 17.3 success-rate points in absolute terms. Section 4.5 of the paper covers both selectors in one sentence: in both cases, the selected policy is significantly worse than the best one, a 10% to 100% decrease. The 10 percent floor belongs to the final-checkpoint criterion and the 50 percent floor on the study page belongs to the validation criterion, so quoting either range against the wrong selector inverts the comparison the authors drew.

A 6 to 17 point gap matters because it is the same size as the deltas a policy comparison is trying to resolve. If two policy families differ by 8 points and both numbers came from whatever last happened to point at, the comparison sits inside the selection noise of a single run.

The Max column is not a target you can aim at. Appendix B.2 describes what bought it: each agent is trained for N epochs of M gradient steps and evaluated every E epochs by running 50 rollouts, with the maximum success rate over training reported and averaged over 3 seeds. For low-dimensional agents N is 2000 and E is 50, which is 40 evaluation points per run; for image agents N is 600 and E is 20, which is 30. At 50 rollouts per evaluation point and 3 seeds, one low-dimensional cell in that table cost 6,000 simulated rollouts. Appendix G states the consequence without hedging: this kind of variance is problematic for real world settings where it is not feasible to run 50 rollouts per checkpoint for each training run, as the authors did in simulation. Max is an oracle. The question is how much of the 6 to 17 point gap you can buy back with a budget that exists.

There is a wrinkle for LeRobot users: the incumbent selector is not a bad validation loss, it is no validation loss. In v0.6.1 the training config defaults eval_steps to 0, and the loop that computes held-out loss only runs when eval_steps is positive, an eval dataloader exists, and the step divides evenly. Setting eval_steps without also setting dataset.eval_split is refused outright: eval_steps > 0 requires dataset.eval_split > 0.0 to hold out eval data. Enabling it is two decisions, and a default run makes neither. So for a stock LeRobot run the thing being deployed is whatever last resolves to, chosen by nobody.

Before you can rank candidates you need to know how many you have, and that number was fixed before the run started. LeRobot v0.6.1 decides when to write with should_save_checkpoint, which returns true when save_freq is positive and the step divides by it, or when the step equals the total. With the defaults steps: int = 100_000 and save_freq: int = 20_000, that is five writes.

ls -l outputs/train/act_so101_test/checkpoints
# 020000
# 040000
# 060000
# 080000
# 100000
# last -> 100000

Each step directory holds pretrained_model/ with config.json, model.safetensors and train_config.json, and a training_state/ directory with the optimizer state. The six-digit names come from a helper that pads to max(6, len(str(total_steps))) digits. The full layout and the flag pair that decides what a load actually restores are covered in why a fine-tuned VLA returns 0% success, so one sentence is all it needs here.

Five candidates is thin when the gap between the final checkpoint and the best one reaches 17.3 points. Storage is not the reason it is thin: the published ACT checkpoint for the ALOHA cube transfer task carries model.safetensors at 206,766,560 bytes, about 197 MiB, next to a config.json of 1,334 bytes and a train_config.json of 5,290 bytes. Twenty checkpoints at that scale is under 4 GiB of weights before the training state directories. The constraint is the default, which stays where it is until you move it.

lerobot-train \
  --dataset.repo_id=${HF_USER}/so101_test \
  --policy.type=act \
  --output_dir=outputs/train/act_so101_test \
  --job_name=act_so101_test \
  --policy.device=cuda \
  --steps=100000 \
  --save_freq=5000 \
  --policy.repo_id=${HF_USER}/my_policy \
  --save_checkpoint_to_hub=true

That gives twenty step directories instead of five, at the cost of disk and upload time. --save_checkpoint_to_hub=true is refused without --policy.repo_id, and on an air-gapped run you leave it alone; it is here because a crashed or timed-out run then still leaves recoverable candidates somewhere other than the training node.

The openpi training config makes the same decision differently and is worth reading as a contrast. Its TrainConfig sets num_train_steps: int = 30_000 with save_interval: int = 1000 and keep_period: int | None = 5000, and the checkpoint manager is constructed with max_to_keep=1 alongside that keep period. So a stock run writes a checkpoint thirty times and finishes with the six that landed on a 5,000-step boundary still on disk. How many candidates exist at the end is a retention setting in both stacks, not a fact about training.

Two different commands do two different jobs, and conflating them is how hardware time disappears. lerobot-eval runs rollouts in a simulated environment behind the evaluation install extra and requires an --env.type. lerobot-rollout is the documented path for deploying a policy on a real robot, with --strategy.type selecting among base, sentry, highlight, dagger and episodic. The screen is cheap and eliminates; the gate is expensive and certifies.

lerobot-eval \
    --policy.path=outputs/train/diffusion_pusht/checkpoints/005000/pretrained_model \
    --env.type=pusht \
    --eval.batch_size=10 \
    --eval.n_episodes=10 \
    --policy.use_amp=false \
    --policy.device=cuda

The load-bearing detail is --policy.path pointing at a step directory rather than at a Hub repo id. EvalConfig defaults n_episodes to 50 and batch_size to 0, which auto-tunes from available CPU cores, and the run writes eval_info.json into its output directory with the success rate under pc_success. For a multi-task screen, --env.type=libero accepts a comma-separated --env.task=libero_spatial,libero_object,libero_goal,libero_10, which gives you four suites off one command and one artifact per run.

Sweeping every candidate to produce that shortlist is a loop you write. No LeRobot flag ranks checkpoints, and no command in the stack selects one.

RUN=outputs/train/act_so101_test
for CKPT in $(ls $RUN/checkpoints | grep -E '^[0-9]+$'); do
  lerobot-eval \
    --policy.path=$RUN/checkpoints/$CKPT/pretrained_model \
    --env.type=libero \
    --env.task=libero_object \
    --eval.batch_size=1 \
    --eval.n_episodes=10 \
    --output_dir=eval/$CKPT
done
grep -H pc_success eval/*/eval_info.json

The grep works because output_dir is a top-level field on EvalPipelineConfig and the run drops eval_info.json there. Note what this loop may not decide. robomimic evaluated policies in simulation and on hardware but published no checkpoint-level rank correlation between the two, so the screen is an elimination device with a floor you wrote down before you saw the scores, never a prediction of hardware order.

Those five stages run in that order, and the order is the whole discipline. The budget arithmetic falls out of the Gate row: divide the hardware trials you can afford by 50 per task per condition and that is how many candidates reach the gate. Two hundred trials buys four, not twenty. The screen exists to decide which four.

The hardware run is one command per surviving candidate, pointed at that candidate's step directory.

lerobot-rollout \
  --strategy.type=base \
  --policy.path=outputs/train/act_so101_test/checkpoints/060000/pretrained_model \
  --robot.type=so100_follower \
  --robot.port=/dev/ttyACM1 \
  --robot.cameras="{ up: {type: opencv, index_or_path: /dev/video10, width: 640, height: 480, fps: 30}}" \
  --task="Put lego brick into the transparent box" \
  --duration=60

--strategy.type=base runs autonomously with no recording, which is enough to order candidates against each other. The run that accepts a policy has to leave a trail, so switch to --strategy.type=sentry with --dataset.repo_id, which records continuously and uploads. Settle where that upload lands before an air-gapped gate run, not after. Whether your simulator is close enough to the cell for the screen to eliminate honestly is a separate question, and the tradeoffs between the usual options are in Isaac Lab versus MuJoCo versus Genesis.

Once you accept that checkpoints from one run differ by 17 success-rate points, averaging several of them into one set of weights is the obvious thing to try. The two published results that support the mechanism are both outside robotics. Stochastic weight averaging, published in March 2018, reports that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training, and that it has almost no computational overhead. Model soups, published in March 2022, reports that averaging the weights of multiple models fine-tuned with different hyperparameter configurations often improves accuracy and robustness, and that you may average many models without incurring any additional inference or memory costs. Both evaluate image classification and NLP. Neither reports a manipulation success rate, so the transfer to a robot policy is a hypothesis you test, not a result you inherit.

One robotics stack already bets on the mechanism. The openpi TrainConfig sets ema_decay: float | None = 0.99 by default, so that stack averages across the training trajectory for you, and the LoRA presets override it with ema_decay=None under the comment that EMA is turned off for LoRA finetuning. If your fine-tune is a LoRA run on that stack, the averaging you may have assumed was happening is not. The architecture side of choosing between these policy families is in OpenVLA versus pi0 versus SmolVLA versus GR00T N1.

LeRobot v0.6.1 gives you nothing here. Listing all 1,192 paths at that tag, no file matches soup, averag or swa, and the only four matches for ema are kinematics files, where it sits inside the word kinematic. The diffusion policy modeling and configuration files and the training script contain no EMA implementation. So averaging is a script you own.

import shutil, sys
from pathlib import Path
import torch
from safetensors.torch import load_file, save_file

out = Path(sys.argv[1])
srcs = [Path(p) for p in sys.argv[2:]]
states = [load_file(p / "model.safetensors", device="cpu") for p in srcs]
keys = set(states[0])
if any(set(s) != keys for s in states):
    raise SystemExit("checkpoints do not share a parameter set")

merged = {}
for k in keys:
    ts = [s[k] for s in states]
    if ts[0].is_floating_point():
        merged[k] = torch.stack([t.float() for t in ts]).mean(0).to(ts[0].dtype)
    else:
        merged[k] = ts[0].clone()

out.mkdir(parents=True, exist_ok=True)
save_file(merged, out / "model.safetensors", metadata={"format": "pt"})
shutil.copy(srcs[-1] / "config.json", out / "config.json")

Three things about that script. Non-float tensors are copied from the first checkpoint rather than averaged, because averaging an integer buffer produces a value that was never valid. config.json has to travel with the weights or nothing will load them. And the output is one more candidate, which earns the same simulation screen and the same hardware gate as every checkpoint it was built from. No source opened for this post gives a success number for an averaged imitation learning policy, a recommended number of checkpoints to merge, or a step range to merge over, so pick a range, screen it, and keep the result only if it clears the floor.

last is not an identifier. update_last_checkpoint unlinks the existing entry and re-points a relative symlink at the newest step directory, so one resumed run changes what last means without changing anything you wrote down. A release record that says the policy came from last describes a moving target. Upload the step you actually gated, by name.

CKPT=010000
hf upload ${HF_USER}/act_so101_test${CKPT} \
  outputs/train/act_so101_test/checkpoints/${CKPT}/pretrained_model

For a run that pushed its own checkpoints, the LeRobot docs give a revision pin instead: each pushed checkpoint is tagged with its step, so --policy.pretrained_revision=010000 names a checkpoint without a commit sha. One caveat belongs beside that flag. The training loop uploads each checkpoint under checkpoints/<step>/ and tags the commit with the step name, while the loader downloads model.safetensors from the repository root at the given revision with no subfolder argument. The tag and the upload path are documented and real; what a revision-pinned load returns for a repository whose root was still empty is not something to assert. The two retrieval paths that hold end to end are the local checkpoint directory and a one-checkpoint repository of your own, which is what the hf upload form above produces. Do not invent a subfolder flag to bridge the gap.

On a regulated line the selection decision becomes an artifact: the question stops being which checkpoint is best and becomes which weights are running in that cell and what evidence chose them. Two things follow that do not apply on a lab bench.

The evidence has to be reproducible from inside the boundary that owns the cell. The screen scores, the gate results and the weights all have to live in a registry you control, because the step id alone is meaningless without the run that produced it. save_checkpoint_to_hub defaults to False and refuses to run without --policy.repo_id, and wandb.enable defaults to False beside it, so weights and run history leave the training node only when a flag on the command line says so; the tracker artifact route is covered in the fine-tuning post linked above. Air-gapped, the tag a Hub commit would carry does not exist, so hash model.safetensors and store the digest next to the step id. Four fields make an entry defensible years later: run id, dataset revision, step id, and weight digest.

Selection has to be redone per run, not per recipe. The winning step is a property of one training run with one seed on one dataset revision, and robomimic's own cells average over 3 seeds without reporting whether the winning epoch index is consistent across them. Carrying a step number forward because it worked on the last run is the same mistake as carrying forward a vendor's trial count. The wider question of scoring systems where the correct output is not a single value is worked through in testing AI systems where there is no right answer, and the boundary itself is mapped across the physical AI pillar.

Seven steps, in this order, with nothing skipped because the loss curve looked flat.

The smallest version of this you can do this week costs no hardware time at all. Take the run you have already trained, screen the step directories sitting on your disk right now, and check whether last actually wins. If it does not, you have just found success-rate points that were already paid for.