SmolVLA Fine-Tuning Returns 0% Success: What to Check First

A run that reached step 20,000 without crashing has eliminated a real set of problems: the optimiser is stepping, the dtype is not producing NaNs, the dataloader is feeding batches, and the video decode path works. That is the entire list.

What a falling loss cannot tell you is whether the action slots line up. modeling_smolvla.py reads original_action_dim from the action feature shape and slices losses to [:, :, :original_action_dim] before masking, so the zero padding out to 32 dimensions is excluded from the reported number while still occupying model action slots. A dimension-order mismatch therefore lands entirely on the robot and never on the curve.

Loss magnitude is also not portable. LeRobot issue #3433, filed against 0.5.2 and now closed, reported that SmolVLA masked padded action timesteps with action_is_pad and then reduced with .mean(), dividing by every element including the zeroed padding, so the reported loss was underestimated in proportion to the padding fraction. The v0.6.1 code carries the fix, computing num_valid = ((~actions_is_pad).sum() * losses.shape[-1]).clamp_min(1) and then loss = losses.sum() / num_valid. Comparing your 0.008 against a number posted on a different version, or at a different chunk_size, tells you nothing.

A hard zero across every episode is a plumbing signal, not a statistics problem; how many trials it takes to tell two working policies apart is a separate discipline, covered in how many trials it takes to evaluate a robot policy. Stop counting and start diffing.

The cheapest check in the stack does not involve a checkpoint. lerobot-replay reads the action column straight out of your dataset and sends it to the robot frame by frame, with no policy in the loop.

lerobot-replay \
    --robot.type=so101_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=my_blue_follower_arm \
    --dataset.repo_id=${HF_USER}/mydataset \
    --dataset.episode=0

--dataset.root points at a local dataset outside $HF_LEROBOT_HOME, and --dataset.fps defaults to 30. Run it on your cleanest demonstration.

If the replay does not accomplish the task on your hardware today, no amount of training will fix it, because the policy's ceiling is the behaviour in that column. The fault is calibration, mounting, a moved fixture, or the recording itself.

Calibration deserves its own line because it lives outside the dataset. LeRobot resolves HF_LEROBOT_HOME to $HF_HOME/lerobot unless overridden, and HF_LEROBOT_CALIBRATION to $HF_LEROBOT_HOME/calibration (the older LEROBOT_HOME is deprecated and now raises if set). Re-running lerobot-calibrate between recording and rollout changes what a joint reading means, and nothing in the checkpoint, the dataset or the loss curve knows it happened. Replay catches that in one episode.

Two flags load a checkpoint and they are not interchangeable. The v0.6.1 docs publish the difference as a table, and it is the highest-yield thing to check in a failing command.

The mechanism sits in src/lerobot/policies/factory.py. In make_policy(), cfg.output_features is always overwritten from your dataset, but cfg.input_features is populated from the dataset only if not cfg.input_features. With --policy.path=lerobot/smolvla_base the input features arrive already populated from the checkpoint, so your dataset's camera and state keys are ignored. That is why a rename map exists.

The reset side is just as expensive. The docs spell out one case for pi0.5: --policy.n_action_steps=10 and --policy.empty_cameras=1 have to be passed explicitly because --policy.pretrained_path loads weights only, and the checkpoint's stored values would otherwise fall back to 50 and 0. The same silent fallback applies to every stored SmolVLA setting. Four fields diverge between the shipped lerobot/smolvla_base config and the current dataclass defaults: pad_language_to is max_length against longest, prefix_length 0 against -1, num_expert_layers 0 against -1, and load_vlm_weights true against false.

That last one is a category of its own. load_vlm_weights defaults to False in the dataclass, with the in-source comment explaining that False is for training the expert from scratch and True is for initialising from pretrained SmolVLA weights. Running --policy.type=smolvla without --policy.load_vlm_weights=true gives you a randomly initialised SmolVLM2 backbone, and under PEFT the validator logs: training SmolVLA from scratch is unlikely to yield good results, set load_vlm_weights=True to fine-tune the existing policy. Published LIBERO commands built on --policy.type=smolvla --policy.load_vlm_weights=true train an action expert from scratch on a pretrained backbone, which is not a fine-tune of the released checkpoint.

For fine-tuning lerobot/smolvla_base on your own arm, start from the documented command:

cd lerobot && lerobot-train \
  --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=${HF_USER}/mydataset \
  --batch_size=64 \
  --steps=20000 \
  --output_dir=outputs/train/my_smolvla \
  --job_name=my_smolvla_training \
  --policy.device=cuda \
  --wandb.enable=true

lerobot/smolvla_base declares its inputs in config.json: observation.state at shape [6], observation.images.camera1, .camera2 and .camera3 each at [3, 256, 256], and action at [6]. If your recording produced observation.images.front and observation.images.wrist, those keys do not exist as far as the loaded config is concerned.

The middle row is the one that produces 0%. Because empty_cameras defaults to 0, the back-fill loop exits immediately, the missing views are dropped, and a two-of-three camera mismatch trains and rolls out with no warning.

The third row is quieter still. pad_vector(vector, new_dim, *, truncate=False) in src/lerobot/policies/common/vla_utils.py zero-pads the last dimension up to new_dim and, with truncate=False (the value SmolVLA uses), returns the tensor unchanged when it is already that wide. It never raises. SmolVLA calls it for state and action against a max_state_dim and max_action_dim of 32, so the declared 6-wide shape is not enforced in that path.

The fix is --rename_map, a JSON dict mapping the key you have to the key the policy wants. It works on lerobot-train and lerobot-eval, and the documented multi-key form is:

--rename_map='{"observation.images.left_hand_camera_rgb": "observation.images.left_hand", "observation.images.right_hand_camera_rgb": "observation.images.right_hand", "observation.images.first_person_camera_rgb": "observation.images.first_person"}'

LeRobot's own documentation is blunt about why the strings matter: it enforces the .images.* prefix for visual features, tells you to make your policy config input_features use the same naming keys as your dataset metadata, and notes that naming keys are encoded inside the normalization statistics layer. These strings are part of the contract, not labels.

To see what the model actually received rather than what you sent, LeRobot ships processor hooks. register_before_step_hook(fn) and register_after_step_hook(fn) both take (step_idx: int, transition: EnvTransition), and transform_features(initial_features) validates the feature contract.

def show_keys(step_idx: int, transition):
    obs = transition.get(TransitionKey.OBSERVATION)
    print(step_idx, sorted(obs.keys()))

preprocessor.register_after_step_hook(show_keys)

Print observation keys at step 0 and you know within a minute whether three cameras or one arrived.

MEAN_STD normalisation in src/lerobot/processor/normalize_processor.py is (tensor - mean) / (std + eps) with eps defaulting to 1e-8, and the inverse is tensor * std + mean. Consider a joint that barely moved during recording, so its std in meta/stats.json is near zero. That dimension is divided by roughly 1e-8 on the way in and multiplied by roughly zero on the way out, so the commanded value collapses to the dataset mean regardless of what the model predicts.

That is the signature described in LeRobot issue #1791, where a multi-task fine-tune produced a robot that stayed at the home position or returned to it slowly, completing no task. A maintainer closed it by pointing at the official LIBERO port and a working SmolVLA LIBERO checkpoint as a reference, not with a root cause, so treat symptom-to-cause here as a hypothesis you test.

One widely repeated claim about this is wrong at v0.6.1. On a fresh fine-tune, lerobot_train.py forces your dataset's statistics over the checkpoint's, setting preprocessor_overrides['normalizer_processor']['stats'] = dataset.meta.stats and the matching postprocessor_overrides['unnormalizer_processor']['stats'], both guarded by if not cfg.resume. You do not need to delete anything. The two genuine hazards are narrower: --resume=true, where the checkpoint's saved statistics stay authoritative by design, and a stale meta/stats.json after you delete episodes or convert a dataset.

The recompute is a dataset operation, not a training flag:

lerobot-edit-dataset \
    --repo_id your_dataset \
    --new_repo_id your_dataset \
    --operation.type recompute_stats \
    --operation.overwrite true

overwrite defaults to false and skip_image_video defaults to true, so the in-place form needs that flag. Then read the warning that follows in the docs, because this is where the day goes: the result lands in $HF_LEROBOT_HOME/your_dataset, not the cache that --dataset.repo_id reads. Train with --dataset.root=$HF_LEROBOT_HOME/your_dataset, or add --push_to_hub true. Recomputing statistics and then training against the old cached copy is common enough to belong on every checklist.

Two related notes. normalization_mapping is VISUAL: IDENTITY, STATE: MEAN_STD, ACTION: MEAN_STD in both the dataclass and the shipped checkpoint, overridable as --policy.normalization_mapping='{"ACTION": "MEAN_STD", "STATE": "MEAN_STD", "VISUAL": "IDENTITY"}'. Because VISUAL is IDENTITY, SmolVLA rescales images from [0, 1] to [-1, 1] itself inside prepare_images for SigLIP, and the normalizer never touches pixels. The files you open to inspect any of this are meta/stats.json, meta/info.json and meta/tasks.parquet; the layout is covered in LeRobot, RLDS and HDF5 as robot data formats.

Two datasets can have identical shapes, keys and statistics and still mean different things.

Relative actions require statistics computed for them, through --operation.type recompute_stats --operation.relative_action true --operation.chunk_size 50 --operation.relative_exclude_joints "['gripper']", with chunk_size matching the policy's. A dataset of delta end-effector poses fine-tuned onto a prior trained on absolute joint targets converges in loss and does nothing useful on the robot: the numbers are internally consistent and mean the wrong thing.

Issue #2259 shows how many of these can stack. The reporter converted 500 RLBench Franka Panda episodes with observation.state of dimension 7, a separate 7-dimensional observation.state.joints, five camera views, and action of dimension 7 as a delta end-effector pose plus gripper. Training ran without issues and measured success was 0%, against roughly 22% for OpenVLA on the same data. A follow-up run cut to three cameras and reached below 0.01 training loss with the same result. No maintainer has published a root cause there, so read it as what a stack of mismatches produces.

The same trap reappears at evaluation. The LIBERO integration exposes --env.control_mode, relative by default and absolute as the alternative, and the documentation warns that different VLA checkpoints are trained with different action parameterizations, so the mode has to match the policy.

Then the instruction. The lerobot/smolvla_base preprocessor pipeline is six steps in order: rename_observations_processor, to_batch_processor, smolvla_new_line_processor, tokenizer_processor, device_processor, normalizer_processor. The tokenizer step is configured with max_length 48, task_key task, padding_side right, padding max_length and truncation true. A long instruction is cut at 48 tokens without a warning, and the tail is usually where the object and the destination are. Keep it short and identical between recording and rollout, and remember that meta/tasks.parquet is the canonical source for the task in the dataset. At rollout the string is yours to get right:

lerobot-rollout \
    --strategy.type=base \
    --policy.path=outputs/train/my_smolvla/checkpoints/last/pretrained_model \
    --robot.type=so101_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=my_blue_follower_arm \
    --robot.cameras="{ front: {type: opencv, index_or_path: 8, width: 640, height: 480, fps: 30}}" \
    --task="Grasp a lego block and put it in the bin." \
    --duration=60

Robot demonstrations are not neutral data. They carry the line layout, the fixture geometry, the part being handled and the cycle time, which is why manufacturing and defence-adjacent buyers treat an episode archive the way they treat CAD. That changes two things about this loop.

The first is egress, and the default is not where most people look. PreTrainedConfig.push_to_hub defaults to True, but the training loop's Hub upload is gated on TrainPipelineConfig.save_checkpoint_to_hub, which defaults to False and refuses to run without --policy.repo_id. Checkpoints do not leave on their own. The live path out is the experiment tracker: the official SmolVLA command includes --wandb.enable=true, and with it the trainer calls log_policy(checkpoint_dir) at every checkpoint, packaging the pretrained_model directory as a model artifact and uploading it unless --wandb.disable_artifact=true is set. An air-gapped run is four decisions: pre-stage lerobot/smolvla_base under $HF_LEROBOT_HOME/hub, set --dataset.root so nothing resolves through the Hub, set --wandb.disable_artifact=true or self-host the tracker, and leave save_checkpoint_to_hub alone. The same pattern governs serving the policy, in air-gapped and offline vLLM deployment, and the cost side is in cloud versus on-premise AI security and cost.

The second is reproducibility. Three fields belong in every run record: which loading flag you used, because it changes at least four config values and the entire feature-name source; the dataset revision, since LeRobot's guidance is to pin --dataset.revision=<commit-sha> because Hub datasets can be re-uploaded, with a content hash over meta/ plus the accepted episode count as the on-premise equivalent; and the calibration file in force at recording and at rollout, because it sits outside everything else and redefines the state vector.

Checkpoints land at <output_dir>/checkpoints/<step>/pretrained_model with a last pointer, and training refuses to start if output_dir exists while resume is false. That refusal is a feature: it stops two runs from sharing a directory and becoming impossible to attribute.

Run these in order of cost, not likelihood. There is no published measurement of how often each causes a 0% run, so anyone ranking them by frequency is guessing.

Before concluding the run was too short, check which recipe you copied. The SmolVLA documentation gives 20,000 steps at batch size 64 and roughly four hours on a single A100; the lerobot/smolvla_base model card gives 100,000 steps at batch size 4. Those are not the same run, and quoting one recipe's expectations against the other's command convinces teams a healthy job is broken.

The stop rule is simple. When those checks come back clean, train ACT on the same episodes: it is far cheaper than a VLA, which is why practitioners keep recommending it in the LeRobot issue threads. If ACT also does nothing, the problem is in the demonstrations or the hardware and no config flag reaches it, so re-record rather than keep tuning. If ACT works and SmolVLA does not, you are back on the binding list with a much smaller search space.

This week, do two things. Replay episode 0 on the arm. Then print the checkpoint's input_features next to your dataset's feature keys and read them side by side, character for character. Together they take under an hour and eliminate the two failure classes no amount of additional training will fix. For the wider set of production decisions, start at the physical AI pillar.