LeRobot vs RLDS vs HDF5: Which Robot Data Format to Use

The four formats in circulation solve different original problems, and their structures still reflect that.

RLDS is a semantic layer on top of TFRecord and TensorFlow Datasets. A dataset contains episodes, an episode contains an ordered sequence of steps, and every step is a dictionary with mandated keys: observation, action, reward, discount, is_first, is_last, is_terminal. Observation and action are themselves nested dictionaries, which is how a single schema absorbs wildly different sensor suites. Shards run roughly 200 to 500 MB and reads go through tf.data with parallel prefetch. It came out of reinforcement learning, which is why reward and discount are first-class fields most manipulation datasets leave at zero.

HDF5 is a general scientific array container: hierarchical groups, n-dimensional datasets, attributes, chunking and optional compression. Robotics inherited it from the robomimic lineage, and bimanual teleoperation stacks commonly write it. It has no robot-specific schema at all, which is both its flexibility and its problem, since two HDF5 robot datasets from different labs frequently share nothing but the file extension.

LeRobot splits the recording by data type. Tabular data (joint states, actions, timestamps, episode indices) goes into Apache Parquet, one row per timestep. Camera frames are encoded as MP4 video, organised by camera in subdirectories. Metadata lives in meta/info.json (schema and fps), meta/stats.json (normalisation statistics), meta/tasks.jsonl (task mappings), and chunked Parquet under meta/episodes/ holding episode boundaries. Reads go through PyTorch and the Hugging Face datasets library.

ZARR is a chunked, compressed n-dimensional array store, laid out as a directory tree of chunk files with JSON metadata. Diffusion-policy-style training loops adopted it because a replay buffer doing random access into large arrays is exactly what ZARR chunking is built for.

Published measurements across several public robot datasets make the pattern unambiguous. These figures come from a formats-and-storage study, Robo-DM, which measured identical content across formats.

There is one mechanism behind every row. RLDS and HDF5 as conventionally written store camera frames as uncompressed matrices. LeRobot encodes them as video. Robot demonstrations are enormously temporally redundant, a static workcell with one moving arm, which is precisely the case inter-frame video compression was designed for.

The tradeoff is decode cost. Video-backed storage moves work from disk and network to CPU or GPU at load time, and pulling one arbitrary frame from the middle of an encoded clip costs more than slicing an array. For imitation learning, which reads sequential windows of timesteps, that trade is strongly favourable. For a training loop doing uniform random single-frame access, it is not, and that is the case where ZARR or an array format earns its disk.

v2.1 gave every episode its own Parquet file and its own MP4 per camera. Clean, readable, and it fails at scale for a boring reason: a dataset approaching millions of episodes becomes millions of files, and filesystems and object stores both degrade badly under that. Listing a directory becomes slow, syncing becomes slow, and per-file overhead starts to dominate small reads.

v3 concatenates. Tabular data from many episodes goes into larger shared Parquet files, frames from many episodes go into shared MP4 files (episode-0000.mp4, episode-0001.mp4 and so on become file-0000.mp4), and everything is organised into chunked subdirectories. Metadata becomes the relational index: it maps an episode ID back to the file and the byte or frame range holding it. v3 also adds StreamingLeRobotDataset, which reads directly from the Hub without downloading the dataset to disk or loading it into memory.

Upgrading an existing v2.1 dataset is one module invocation:

python -m lerobot.datasets.v30.convert_dataset_v21_to_v30 \
  --repo-id=<HFUSER/DATASET_ID>

The design lesson generalises past LeRobot. Any collection program that writes one file per episode will hit the same wall somewhere between hundreds of thousands and millions of episodes. If you are specifying a delivery format for a multi-year program, ask what happens at 10x your current episode count before the layout is locked in.

Format comparisons routinely measure order-of-magnitude differences in episodes per second between loaders on identical content. The variables that matter:

Profile the loader before scaling the cluster. A pipeline that leaves an expensive GPU at 30% utilisation because it is opening small files is a far more common failure than an actual compute shortage, and the same lesson applies here as in language model serving, where the inference bottleneck is usually not where teams assume.

Conversion between these formats is a solved problem and should never drive the original choice.

The discipline that matters more than the command: keep the raw capture, and treat every training format as a derived artifact. Script the conversion, version the script, and be able to regenerate the training corpus from raw at any time. Programs that treat the converted dataset as the only copy discover the cost the first time they need a channel that the conversion dropped, or need to re-synchronise streams after finding a timestamp bug. This is the same principle we apply to text corpora in cleaning messy business data for AI training: the pipeline is the asset, not its output.

The most common right answer for a new program is LeRobot v3 as the working format plus a parallel RLDS export if the cross-embodiment ecosystem matters to you. The most common wrong answer is picking whatever the first tutorial used and discovering at 100,000 episodes that the layout does not shard.

Format is the easy half. The half that is not recoverable after the fact is schema and synchronisation. Before collection starts, agree in writing:

Those seven lines cost an hour to agree and prevent the failure mode where a delivered corpus is technically in the right format and still untrainable. It is the same reason our robotics data collection programs settle the format, schema and acceptance bar before the first episode is recorded, and do the segmentation, calibration, proprioceptive sync and format conversion on our side rather than shipping footage somebody on your team still has to clean up.

Once the corpus lands, the question becomes how much of it you need and how it should be spread, which we work through in how much data you need to train a robot policy, and where each type of data should come from in the first place, covered in teleoperation vs simulation vs human video. Both sit alongside the rest of our robotics writing in the Physical AI and robotics hub.