Teleoperation vs Simulation vs Human Video for Robot Data

The difference between these sources is not quality. It is which channels exist in the recording at all, and no amount of post-processing creates a channel that was never captured.

Read the first row and the last few together. Teleoperation is the only source that produces action labels on your exact embodiment, which is why it remains the core of every deployed manipulation policy we have worked on. Simulation is the only source with no human in the throughput loop. Human video is the only source where diversity is nearly free, and it is missing the one channel that control needs.

Teleoperation means a human drives your robot through a leader-follower rig, VR controllers or an exoskeleton, and every joint position, torque and camera stream is recorded on the platform you will deploy.

The economics are operator-bound.

The reference point for what a serious teleoperation program looks like at scale is the largest public in-the-wild manipulation dataset: 76,000 teleoperated trajectories, roughly 350 hours of interaction, 564 distinct scenes across 52 buildings, 86 tasks, collected over 12 months by 50 teleoperators at 13 institutions on 18 identical 7-DoF arms. The full methodology is in the DROID paper. Twelve months and 50 operators for 350 hours of interaction is the honest cost of scene diversity through a robot.

Where teleoperation wins. Contact-rich manipulation, tight tolerances, force-sensitive tasks, anything where the physics of the contact is the task, and any policy that must ship on a specific machine. If the deployment target exists and the task involves contact, this is where the majority of the budget goes.

Where it fails. Cost per unit of diversity. Moving the rig to a new room is hours of setup for tens of episodes, and diversity is the axis that drives generalization, as covered in our guide on how much data you need to train a robot policy. Teleoperation also cannot record tasks that are unsafe or on hardware that does not exist yet.

The pricing trap. Per-hour billing puts rejected episodes on your invoice. At a 70% acceptance rate, roughly 900 of 3,000 billed demonstrations never reach the training loop. Pricing per accepted episode moves that risk to the vendor, which is why our robotics data collection programs bill that way, and why quotes below roughly $15 an hour usually exclude the QA rejection and replacement that make the real number.

Egocentric collection means operators wear head-mounted cameras and perform tasks in real environments, across whatever range of scenes, objects and lighting you specify. Modern capture rigs add 3D hand and finger tracking, which is what turns raw footage into something a policy can learn motion structure from.

The scale available here is a different order of magnitude. One public dataset holds 829 hours of egocentric video with paired 3D hand and finger tracking across 194 tabletop tasks, captured on consumer headsets where multiple calibrated cameras and on-device SLAM track every joint of each hand. The methodology is in the EgoDex paper. A larger study trained a vision-language-action model on more than 20,000 hours of action-labeled egocentric human video and reported a log-linear scaling relationship between human data volume and validation loss, published as EgoScale.

The co-training result at the top of this post comes from a framework that aligned human egocentric video with teleoperated robot data in a single policy architecture: EgoMimic. Its scaling finding is the practically useful part. An added hour of hand data beat an added hour of robot data on the same task, which means that at the margin, on a program that already has a teleoperated core, human video is the cheaper way to buy the next increment of performance.

Where egocentric wins. Pre-training scale, task and scene diversity, speed to first delivery (days, not weeks, because there is no rig to build or calibrate), and any program where the target robot has not arrived yet.

Where it fails. No action labels. There is no joint command, no torque, no gripper state in a video of a human hand. It bootstraps manipulation and it will not teach precise control on its own. Transfer also degrades when the head-mounted viewpoint is geometrically far from the robot's cameras, and when the recorded environments do not resemble the deployment environment.

Two distinct things get called synthetic data, and conflating them causes bad decisions.

Physics simulation renders scenes and rolls out trajectories in a simulator with randomised dynamics, lighting and textures. It is the standard approach for locomotion and whole-body control, where domain randomisation transfers well, and for anything unsafe to demonstrate.

Automated demonstration generation takes a small set of real human demonstrations and adapts them to new object poses, instances and scene configurations. One published system generated over 50,000 demonstrations across 18 tasks from roughly 200 source human demonstrations, remaining compatible with multiple simulators and with real robots, described in the MimicGen paper. A follow-up system extended the approach to bimanual dexterous manipulation for humanoids.

A 250x multiplier on human effort is real leverage, and the boundary is sharp:

The rule we apply: generation multiplies coverage of what you can programmatically vary. It does not create physics you never recorded. A corpus of 50,000 generated episodes derived from 200 demonstrations of a rigid-object pick and place is genuinely robust to where the object starts, and knows nothing about what happens when the object is a folded towel. The general form of this tradeoff is covered in when synthetic data works for AI training.

The sim-to-real gap is smallest for locomotion and gross motion and largest for contact-rich manipulation, for the same reason: simulators model rigid-body dynamics well and model friction, deformation and compliance badly.

The comparison that decides budgets is not cost per hour. It is cost per unit of the thing you are actually short of.

The setup column that does not appear: simulation has a large fixed cost in asset creation, scene modelling and reward or task specification, which is why it pays off for programs running many tasks over years and rarely pays off for a single-task pilot.

Programs that work treat these as stages, not as a pie chart.

Stage 1, before the robot exists. Egocentric collection starts within days and needs no rig. Build the visual, object and task priors, weighted heavily here, and specify the environments to match the deployment site rather than accepting generic footage.

Stage 2, once the target embodiment is on site. Teleoperation on the exact robot becomes the priority, and it should carry the majority of the budget for the deployment task, because it is the only source with no embodiment gap. Set the acceptance criteria before the first episode and agree the diversity matrix (scene, object instance, lighting, initial pose, operator) so coverage is tracked weekly instead of discovered at handover.

Stage 3, once a teleoperated core exists. Automated generation multiplies that core over initial states and layouts at near-zero marginal cost, and simulation handles the locomotion, whole-body and unsafe cases. Both are now anchored to real demonstrations rather than to a modelled guess.

Steady state. A teleoperated core for the deployment task, a human-video layer an order of magnitude larger measured in hours, and a generated layer an order of magnitude larger again measured in episodes. Re-run the subset curve after each delivery to find out which layer is actually binding before ordering more of any of them.

Mixing three sources makes the storage format a real decision rather than a detail, because each source arrives in a different container and the merged corpus has to be readable by one loader. That comparison is in LeRobot vs RLDS vs HDF5.

The rest of our robotics data writing, including formats and policy training, sits in the Physical AI and robotics hub.

The question worth answering before signing a collection contract is not which source is best. It is which channel your policy is currently missing. If it is action fidelity on your hardware, no amount of video or simulation fixes it. If it is scene and object coverage, teleoperation is the most expensive way in the world to buy it. Most stalled programs we see have bought a great deal of the first when they needed the second.