LeRobot Dataset Merge: What Breaks and How to Check

lerobot-edit-dataset is a real console script, mapped in pyproject.toml to lerobot.scripts.lerobot_edit_dataset:main, and it registers nine operation types. Merge is the odd one out: it does not edit a dataset in place, it derives a new one, so it takes its identity from a different pair of flags than every other operation. _validate_config hard-requires new_repo_id on a merge, and handle_merge logs a warning when repo_id or root is set, saying merge uses --new_repo_id and --new_root and that --repo_id and --root are ignored.

Here is the invocation that runs, entirely against local paths:

lerobot-edit-dataset \
    --new_repo_id your-org/cell_a_merged \
    --new_root /data/lerobot/cell_a_merged \
    --operation.type merge \
    --operation.repo_ids "['your-org/cell_a_week01', 'your-org/cell_a_week02']" \
    --operation.roots "['/data/lerobot/cell_a_week01', '/data/lerobot/cell_a_week02']"

That is the complete surface. MergeConfig exposes exactly four fields: repo_ids, roots, concatenate_videos and concatenate_data, the last two defaulting to True. There is no chunk_size, no data_files_size_in_mb and no video_files_size_in_mb on the merge operation, at v0.6.1 or on main. You cannot control file rotation from the CLI, which becomes a problem later in this post.

handle_merge calls merge_datasets in src/lerobot/datasets/dataset_tools.py, a thin wrapper that forwards six arguments to aggregate_datasets in src/lerobot/datasets/aggregate.py. That function performs four rewrites, and only one of them is statistical.

Before any of that, validate_all_metadata compares every source against the first and raises one of three distinct ValueErrors. Those three checks, plus the feature comparison behind the third, are what your merge has to survive.

The fourth and fifth rows are the pair to read together. features_equal_for_merge compares the whole normalized feature dict and returns False on any difference, ignoring the six video.-prefixed encoder keys inside a feature's info sub-dict. Encoder settings are forgiven, stream geometry is not, and the geometry check runs too late to help you.

Run this before the merge, not after. It mirrors the three ValueErrors in the same order and prints the two totals finalize_aggregation will write into info.json:

from lerobot.datasets import LeRobotDatasetMetadata
from lerobot.datasets.feature_utils import features_equal_for_merge

ROOTS = {
    "your-org/cell_a_week01": "/data/lerobot/cell_a_week01",
    "your-org/cell_a_week02": "/data/lerobot/cell_a_week02",
}

metas = [LeRobotDatasetMetadata(rid, root=root) for rid, root in ROOTS.items()]
first = metas[0]

for meta in metas[1:]:
    assert meta.fps == first.fps, f"fps {meta.fps} != {first.fps}"
    assert meta.robot_type == first.robot_type, f"robot_type {meta.robot_type} != {first.robot_type}"
    assert features_equal_for_merge(first.features, meta.features), "feature dicts differ"

tasks = [set(m.tasks.index) for m in metas]
print("union of task strings:", sorted(set().union(*tasks)))
print("expected merged episodes:", sum(m.total_episodes for m in metas))
print("expected merged frames:", sum(m.total_frames for m in metas))

Printing the union of task strings is the point of that snippet. Nothing in the merge path checks task text for near-duplicates, so this is your only warning before "pick up the red block" and "Pick up the red block " become two task indices.

One bookkeeping check is worth running afterwards. finalize_aggregation recomputes info.json totals as sums over the sources, while the dataset-level stats.json is aggregated from pre-merge source statistics with no offsetting and no task relabelling. episode_index and task_index are real features under DEFAULT_FEATURES, so they carry statistics that should agree with the totals.

import numpy as np
from lerobot.datasets import LeRobotDatasetMetadata

meta = LeRobotDatasetMetadata("your-org/cell_a_merged", root="/data/lerobot/cell_a_merged")

print("info.json total_episodes:", meta.info.total_episodes)
print("info.json total_frames:  ", meta.info.total_frames)
print("info.json total_tasks:   ", meta.info.total_tasks)
print("stats.json episode_index max:", np.asarray(meta.stats["episode_index"]["max"]))
print("stats.json task_index max:   ", np.asarray(meta.stats["task_index"]["max"]))

episode_index max should be total_episodes - 1 and task_index max should be total_tasks - 1. Open issue #3788, reopened and unresolved, reports exactly the shape where they do not: two single-task sources whose task strings differ produce merged Parquet rows carrying task_index 0 and 1 while the statistics report a max of 0 and a standard deviation of 0. Its proposed fix, PR #3791, is open and unmerged. The inventory of what each file under meta/ holds is covered separately.

Open issue #2679 has an adjacent mechanism. LeRobotDatasetMetadata.create builds the merged feature dict as {**deepcopy(features), **DEFAULT_FEATURES}, and DEFAULT_FEATURES defines the five bookkeeping features (timestamp, frame_index, episode_index, index, task_index) with only dtype, shape and names, no per-feature fps, so anything the sources carried there is overwritten. Because features_equal_for_merge compares the full dict, a merged dataset can then fail validate_all_metadata when you merge it again with an unmerged one.

Start one level below the merge. A dataset's meta/stats.json is already an aggregation before anything is merged: save_episode in src/lerobot/datasets/dataset_metadata.py does self.stats = aggregate_stats([self.stats, episode_stats]) on every episode and writes the result. And the per-episode inputs are themselves estimates. get_feature_stats instantiates RunningQuantileStats with num_quantile_bins defaulting to 5000 and computes each quantile by linear interpolation inside the bin where the cumulative histogram crosses q * count. Any episode with fewer than two rows skips that path entirely and gets every quantile key set to a copy of the mean.

So a single dataset's stored quantiles are an aggregation of histogram estimates. A merge aggregates those aggregations. DEFAULT_QUANTILES is [0.01, 0.10, 0.50, 0.90, 0.99], giving five keys, and all five go through the same branch, not just the q01 and q99 pair the normalizer reads.

What that branch computes changed on 2026-08-06, and the version boundary matters more than the bug. At tag v0.6.1, aggregate_feature_stats stacks the source quantile values, multiplies by frame counts, sums and divides by the total: a frame-count weighted mean of percentiles. On main, PR #3804 replaced it with np.min for quantile keys at or below the 50th and np.max above it, and the commit's own in-code comment is unambiguous that exact global quantiles cannot be recovered from quantile summaries and that the result is a conservative envelope, bounds across the inputs rather than global quantile estimates.

The repository's own test fixture shows what each version produces. Two sources with 100 and 150 frames:

Six rows, and the two merged columns err in opposite directions. A weighted average of quantiles can never land outside the range of its inputs, so v0.6.1 pulls the band toward the middle: 8.6, sitting between the source spans of 8.0 and 9.0 and covering neither source's own range. The main envelope pushes the other way to 10.0, wider than either source and a factor of 1.16 over the weighted mean. Note the q50 row: because the branch is "at or below the 50th takes the minimum", the stored median of a merged dataset on main is the smaller of the source medians. It is a lower bound labelled as a median.

Neither column is a percentile of the merged frames, and that does not expire when the next patch lands.

v0.6.1 was published 2026-08-03 and is the newest release; PR #3804 merged three days later, so it exists on main and in no released wheel. Main's pyproject.toml declares version 0.6.2, an unreleased development line. Anyone installing from PyPI is on the weighted mean, anyone installing from git main is on the envelope, and neither is on a real quantile merge: PR #3801, which proposes computing true global quantiles via a histogram merge, is open and unmerged.

The check that survives every patch above does not care which aggregation ran. It asks one question: what fraction of frames falls outside the bounds the merged dataset saved for itself?

For a genuine 1st and 99th percentile pair computed over the same frames, roughly 1 percent of frames sits below q01 and roughly 1 percent above q99, so about 2 percent per dimension is the arithmetic expectation. That is not a library constant and not a published threshold; it is what the definition of a percentile implies. Issue #4156, opened 2026-07-27 and still open, reports the measured figures on a merged set: 41.65 percent for state joint 1, 38.74 percent for state joint 6, 42.12 percent for action joint 1 and 38.83 percent for action joint 6. Roughly twenty times the expectation.

The same report isolates the cause. Mean differences between the aggregated and directly recomputed statistics were around 1e-7 to 1e-8, which the reporter reads as confirmation that the problem is specific to quantile aggregation rather than data loading or mean and standard deviation aggregation. That matches the code: mean and standard deviation are exactly recoverable from per-group counts, quantiles are not.

One pass over the numeric columns gives you your own number:

import numpy as np
from lerobot.datasets import LeRobotDataset
from lerobot.utils.constants import ACTION, OBS_STATE

ds = LeRobotDataset("your-org/cell_a_merged", root="/data/lerobot/cell_a_merged")
BLOCK = 50_000

for key in (ACTION, OBS_STATE):
    q01 = np.asarray(ds.meta.stats[key]["q01"], dtype=np.float64)
    q99 = np.asarray(ds.meta.stats[key]["q99"], dtype=np.float64)
    column = ds.select_columns([key])

    outside = np.zeros_like(q01)
    seen = 0
    for start in range(0, ds.num_frames, BLOCK):
        rows = np.asarray(column[start : start + BLOCK][key], dtype=np.float64)
        outside += ((rows < q01) | (rows > q99)).sum(axis=0)
        seen += len(rows)

    pct = 100.0 * outside / seen
    print(key, "outside saved q01/q99 per dim (%):", np.round(pct, 2))

load_stats casts the stored values through cast_stats_to_numpy, so ds.meta.stats[key]["q01"] is already an array with one entry per dimension and the comparison broadcasts per joint. Read the output per dimension, not as an average: in the numbers above, joint 1 and joint 6 differ by about three percentage points, and a mean would hide which axis drifted.

Single digits per dimension, and your merged bounds are close enough to percentiles to train on. Tens of percent, and they are summaries. The check costs one sequential read of two columns.

A wrong q01 and q99 only matter if something reads them, which each policy's default normalization_mapping decides.

Five policies, two exposed. A code search for NormalizationMode.QUANTILES under src/lerobot/policies returns three files: the pi05 and molmoact2 configurations and a processor file for lingbot_va. If you are training pi0, smolvla or act, a bad quantile merge cannot reach your normalizer, though everything about task indices and video files still applies. Which VLA you are fine-tuning decides whether this section is about you.

The exposure is silent by construction. The QUANTILES branch in src/lerobot/processor/normalize_processor.py computes denom = q99 - q01 (substituting self.eps when the denominator is zero) and then 2.0 * (tensor - q01) / denom - 1.0. There is no clamp. A frame outside the saved band normalizes outside [-1, 1] and keeps going. The only thing that raises is a missing key: if q01 or q99 is absent the processor raises a ValueError telling you to update the dataset with augment_dataset_quantile_stats.py. Present-and-wrong raises nothing.

Issue #4156 names the consequence in its own terms: this significantly affects policies using QUANTILES normalization, such as Pi0.5, because a large fraction of normal training samples are incorrectly mapped outside the expected [-1, 1] range. What that does to a final success rate is not something anyone in these trackers measured, so do not expect a bad merge to announce itself as a failed evaluation. The observable is the input distribution and the loss scale.

Video files are not re-encoded during a merge. aggregate_videos copies a source file with shutil.copy when the destination slot is empty, and otherwise calls concatenate_video_files, which writes an .ffconcat list and runs the concat demuxer in stream-copy mode. Rotation to a new destination file is triggered by size, dst_size + src_size >= video_files_size_in_mb, not by chunk arithmetic. chunk_size only feeds update_chunk_file_indices, which rolls the chunk directory when the file index reaches chunks_size - 1. It counts files per directory and says nothing about frames.

Episode timestamps inside a merged file are rebased using accumulated float seconds, each source's offset set to the destination file's running duration via get_video_duration_in_s, which computes float(video_stream.duration * video_stream.time_base).

Open issue #3883, filed 2026-06-27 with no comments, reports a merged dataset whose video reads fail near a file boundary. The reproduction was a merge of 21 datasets with mixed 2-camera and 3-camera footage, and the failing read was past frame 11,723 of a file holding 11,724 frames. The check that fires lives in torchcodec's stream decoder, which validates that a requested frame index is strictly less than the file's frame count and otherwise raises an invalid-frame-index error naming the index, the stream and the bound. With 11,724 frames in the file, index 11724 fails that comparison. Whether the off-by-one originates in float duration drift across the stream copy, in container timebase rounding, or in the reader's own index arithmetic is not established by anything in the tracker, and the merge CLI gives you no rotation knob to experiment with, since MergeConfig has four fields and none of them is a size.

The compatibility check that guards concatenation is the other hazard, and its problem is placement rather than logic. It compares five fields between the destination file and the file being appended, video.height, video.width, video.fps, video.codec and video.pix_fmt, and raises when any differ:

ValueError: Input video /data/lerobot/cell_a_week02/videos/observation.images.top/chunk-000/file-000.mp4 is not compatible with the reference video /data/lerobot/cell_a_merged/videos/observation.images.top/chunk-000/file-000.mp4.

That check runs inside the copy loop, and the source carries a TODO saying to move it before the loop to avoid failing in the middle. So a camera swapped mid-quarter fails your merge after some files are already written, and then the retry cannot run: LeRobotDatasetMetadata.create does obj.root.mkdir(parents=True, exist_ok=False), so a second attempt against the same --new_root raises FileExistsError. Delete the partial output before re-running.

Three things change when the corpus cannot leave your network, and all three are code facts rather than framing.

The repair is an upload. src/lerobot/scripts/augment_dataset_quantile_stats.py accumulates a single running histogram per feature across all episodes instead of aggregating per-episode summaries, and the pi05 and molmoact2 documentation on main recommends it directly, alongside the acknowledgement that recording, resuming and merging all leave meta/stats.json holding a conservative envelope rather than whole-dataset quantiles. It is not a console script: pyproject.toml lists nineteen entry points and this is not one of them, so it runs as a file path. And its argparse surface is hyphenated, --repo-id, which is not the underscored --repo_id that lerobot-edit-dataset takes.

python src/lerobot/scripts/augment_dataset_quantile_stats.py \
    --repo-id your-org/cell_a_merged \
    --root /data/lerobot/cell_a_merged \
    --overwrite \
    --skip-images

--overwrite is mandatory here rather than optional. has_quantile_stats returns True as soon as any feature carries any of the five quantile keys, and a merged dataset always carries all five, so without the flag the script logs that the dataset already contains quantile statistics and returns. The ending is the part to plan around: augment_dataset_with_quantile_stats finishes with dataset.push_to_hub(), then deletes and recreates the Hub tag for CODEBASE_VERSION v3.0, with no flag to suppress it. On a segmented network the call fails and you are left unsure whether anything was written locally; reading the function, write_stats runs before the push, so the local statistics should be on disk, and the acceptance check above is how you confirm that rather than assume it. On a network with egress, running the recommended repair uploads a merged corpus of clinical, defence-supplier or plant-floor teleoperation footage.

The fix a regulated shop needs is not in a released artifact. PR #3804 merged three days after the v0.6.1 cut, so a change-control process that pins to a PyPI version is on the weighted-mean code and stays there until the next release. The choice is pinning a git commit, carrying a one-function patch against aggregate_feature_stats, or moving the affected policies to MEAN_STD. All three are change-control conversations, not a pip install --upgrade.

Merging is also where provenance survives or does not, which is why the rest of the physical AI and robot data pipeline should treat the merged tree as a build output. Nothing in the merge path requires the Hub: --operation.roots and --new_root keep the whole operation on local paths. Training on several datasets without merging is not an escape hatch either. make_dataset in src/lerobot/datasets/factory.py raises NotImplementedError on the multi-dataset branch, and MultiLeRobotDataset sets its statistics with the same aggregate_stats call the merge uses, so it would inherit the identical quantile problem if it were wired up.

The full state of play on the publish date, so you can locate yourself by version rather than by symptom:

Seven symptoms, one of them closed. The rollback discipline for the other six is the same, and it falls out of how aggregate_datasets is written: it never modifies its sources, it writes a new root, and it refuses to write into a directory that already exists. Mount the session roots read-only, pass them with --operation.roots, keep every operator-recorded dataset immutable, and treat the merged tree as a rebuildable derivative with the exact merge command and the LeRobot commit recorded beside it. Then the recovery for a mid-merge codec failure, a stale task index or a suspicious bounds report is always the same move: delete the output and re-derive.

Three things to add this week to whatever script already calls the merge, in order. Put the pre-merge gate in front of it, so an fps or feature mismatch fails in seconds rather than after copying half a terabyte of video. Put the bounds acceptance check behind it, and fail the pipeline above a per-dimension threshold you pick from your own first clean run. Record the LeRobot version alongside the merged tree, because "which aggregation ran" is answerable only from the commit, not from the files. If your merged bounds come back in the tens of percent and you are training pi05 or molmoact2, the single-dataset path is a different repair, covered in recomputing statistics for a single LeRobot dataset.