On the original UMI authors' cup arrangement task, a handheld gripper collected 111 demonstrations an hour against 35 with SpaceMouse teleoperation and 231 with a bare hand, so the over 3x figure is real but tied to one task and one teleop device. The field's default, in the words of a July 2026 paper, is to use UMI data mainly for pre-training and add a small real-robot teleoperation anchor at post-training. That paper, HiFi-UMI, removed the anchor and matched in-domain teleoperation within -2.5, +3.1 and -0.6 percentage points on three policy backbones, but it did so with 3 mm workspace-local end-effector accuracy, offline stereo-inertial SLAM, measured inter-gripper pose, a microsecond hardware trigger and about ten times as many demonstrations as the teleop baseline (3,200 against 300 per task). An October 2026 localisation benchmark found fisheye visual-inertial odometry without tag factors stayed at centimetre scale while a tag-aided method reached millimetre level. So the purchase decision is about localisation quality and the acceptance check on trajectories, not the gripper. LeRobot v0.6.1 has no UMI importer, so budget a conversion step. This week, ask whoever proposes UMI capture for end-effector trajectory error against an independent reference on a sample of your demonstrations, plus the SLAM success rate per session.
On the original UMI authors' cup arrangement task, UMI gripper data collection produced 111 demonstrations an hour. Teleoperation with a SpaceMouse produced 35, and a bare human hand produced 231. That is where the over 3x figure comes from, it is printed on the project page, and it is the number that puts handheld capture into robot data budgets.
The figure answers a narrower question than the budget asks. A July 2026 paper, HiFi-UMI, describes current practice plainly: robot-free UMI capture scales, and the resulting data is used mainly for pre-training, with a small real-robot teleoperation anchor added at post-training. The paper then shows the anchor can be dropped, but its rig reached 3 mm workspace-local end-effector accuracy with offline stereo-inertial SLAM, a microsecond hardware trigger and measured inter-gripper pose, and it needed about ten times as many demonstrations as the teleoperation baseline.
So the thing to buy is localisation quality and an acceptance check on the trajectories. The gripper is the cheap part, and the camera on it raises a second question on any regulated production floor: where the raw footage goes before it becomes trajectories.
What UMI gripper data collection records, and what it does not
The 2024 UMI paper uses one sensor: a GoPro with a 155 degree fisheye lens, fed raw without undistortion. Two side mirrors on the gripper put implicit stereo views into the same frame. The GoPro's built-in IMU is written into the MP4, fiducial markers on the fingers give a continuous gripper-width reading, and each demonstration is one MP4 file.
A SLAM pipeline built on ORB-SLAM3, with marker-assisted initialisation and a pre-built map, turns that video into a 6-DoF end-effector trajectory. Policies are trained on relative end-effector trajectories, expressed relative to the current pose at each inference step. If you want the theory of why end-effector and joint representations behave differently on hardware, our post on joint space vs task space action spaces covers it.
What a UMI session never contains matters more for a buyer:
This is the difference from egocentric human video. Our comparison of teleoperation, simulation and human video as robot data sources notes that human video carries no action labels. UMI does produce action labels without a robot. What it lacks is labels on your exact embodiment.
The paper's ablations show those design choices are load-bearing. On cup arrangement, 20 trials each, relative trajectories scored 20/20, step-wise delta actions 16/20, absolute actions 5/20, and the same task without the fisheye view dropped to 11/20. Transfer of the same policy to a Franka arm scored 18/20. On dynamic tossing, inference-time latency matching (aligning observation streams to the slowest one and sending actions early to cover robot execution delay) took success from 57.5% to 87.5%. On bimanual cloth folding, giving the policy the relative pose between the two grippers took it from 6/20 to 14/20.
| UMI-style handheld capture | Teleoperation on the deployment robot | |
|---|---|---|
| What is recorded | Wide-angle video, IMU, gripper width; end-effector pose derived by SLAM | Joint positions, commanded actions, camera streams, torques where the robot reports them |
| Embodiment | A gripper-shaped device; robot kinematics unknown at capture | Exact, by construction |
| Action labels | Relative end-effector trajectories | Joint or end-effector actions on your robot |
| Published throughput | 111 per hour on cup arrangement (project page) | 35 per hour with a SpaceMouse on the same task |
| Main dependency | Localisation accuracy and SLAM success | Operator skill and robot uptime |
| Failure modes | SLAM loss in texture-poor scenes, infeasible trajectories, sync drift | Slow, operator fatigue, needs the robot built and in the cell |
| Best use | Scene and object diversity, pre-training, collection before the robot arrives | Post-training anchor, force-sensitive and embodiment-specific skills |
Throughput: what the 3x figure does and does not mean
The project page gives three numbers for the same cup arrangement task: 111 demonstrations an hour with the UMI gripper, 35 with teleoperation, 231 with a bare hand. UMI runs at 48% of bare-hand speed, at roughly 30 seconds per demonstration, and the page says collection can start within 2 minutes at a new location.
Three qualifications travel with those numbers, followed by the one hardware cost figure with a primary source behind it.
The teleoperation comparator was a SpaceMouse
A SpaceMouse is a 3D mouse. It is not a leader-follower rig, a VR controller setup or an exoskeleton, and the published ratio says nothing about those interfaces. The over 3x ratio is a measurement against one device on one task. Our teleoperation post gives its own planning bands for trained operators; do not divide those by 111 to get a ratio, because the tasks, devices and acceptance definitions differ.
Some tasks are only collectable by hand
The paper reports that on dynamic tossing, teleoperation produced no successful demonstrations in 15 minutes. For fast, dynamic motions, the comparison is not 3x; it is whether you get data at all.
Cheaper per demonstration is not cheaper per policy
The in-the-wild cup arrangement run collected 1,400 demonstrations across 30 locations in 12 person-hours, and the resulting policy scored 71.7% (43/60) in unseen environments across training and unseen cups, where a baseline trained only on the narrow domain scored 0%. That is the strongest argument for handheld capture: diversity is cheap. HiFi-UMI's discussion section supplies the counterweight. Without pre-training, its UMI-only post-training used 3,200 trajectories per task against 300 teleoperation trajectories, roughly ten times as many. A demonstration that costs a third as much and is needed ten times as often is not a saving. Size against the policy you need, not the demonstration rate; our guide on how much data you need to train a robot policy covers episode-count sizing.
What the hardware costs
The original paper's bill of materials comes to around $370 at 2024 prices: about $73 for the gripper and $298 for the GoPro and accessories. Quotes for commercial handheld kits circulate at other figures; treat any of them as a vendor price, not a property of the method. The paper also lists the device's weight (780 g), bulk and reduced degrees of freedom as the reasons it runs slower than a bare hand. The line items that decide whether the corpus is usable sit after the hardware. SLAM processing per session, mapping and marker setup per location, an independent reference to measure trajectory error against, kinematic filtering for your robot, conversion into your training format, and the teleoperation anchor you may still need. A budget that prices the gripper and the operator hours and stops there has priced the cheapest part of the programme.
Pre-training data or a replacement for teleoperation: the 2026 evidence
HiFi-UMI asks whether raising capture fidelity, rather than shrinking the real-robot share, can remove the teleoperation anchor. Its system uses head-mounted offline stereo-inertial SLAM, measures the relative pose between the two grippers directly, fires every sensor from a shared microsecond trigger, and mounts two wide-angle cameras per hand covering about 200 degrees.
The results, all the authors' own:
Read the caveats the paper writes about itself before quoting the headline. The evaluation covers four tabletop bimanual tasks (stain wiping, shirt folding, remote insertion, produce sorting). Each task and policy pair got 40 rollouts, so one extra success moves the rate by 2.5 percentage points, and the authors describe the gaps as within sampling noise. Generality to other tasks, embodiments and distribution shifts remains untested. And the fidelity factors were not isolated by a controlled degradation, so the paper does not tell you how much localisation error breaks the result.
That last caveat is the one that governs a purchase. The evidence supports this statement: with millimetre-grade end-effector localisation, tight sync and enough demonstrations, a handheld corpus matched in-domain teleoperation on these four tasks. It does not give a threshold, and it does not say a rig with worse localisation would have done the same.
The boundary between the two methods is also blurring. ActiveUMI, published in October 2025, is a portable VR teleoperation kit with sensorised controllers that mirror the robot's end-effectors plus head-mounted active perception, and policies trained only on its data averaged 70% in distribution and 56% on novel objects and environments across six bimanual tasks. Robot-free capture that is also teleoperation is a reminder to evaluate the data, not the label on the device.
| Result | Figure |
|---|---|
| Workspace-local end-effector accuracy | 3 mm, without external tracking |
| Zero-robot post-training vs in-domain teleop, StarVLA-QwenPI | -2.5 pp |
| Zero-robot post-training vs in-domain teleop, OpenPI-pi_0.5 | +3.1 pp |
| Zero-robot post-training vs in-domain teleop, LingBot-VA | -0.6 pp |
| Best policy on precision insertion | 85%, with teleop collected in the evaluation scene and no UMI data from it |
| Pre-training on 4,000 hours, ten unseen tasks | 41% lower action error |
| Pre-training gain on real-robot success, StarVLA-QwenPI | +18.1 pp |
| Released dataset (HiFi-UMI-2K) | 2,000 hours, each demonstration validated by simulation replay |
Why the teleop anchor stays in the plan: localisation error
The MILD benchmark, published in October 2026, measures end-effector localisation for UMI-style teaching on 86 real sensor sequences across 15 repeated tabletop tasks, each with a robot end-effector reference trajectory, plus a simulated set. Its finding is that visual-inertial and fiducial-aided systems show large differences in both trajectory error and temporal coverage, even on the same nominal task. Fisheye visual-inertial odometry without tag factors stayed at centimetre scale on their recordings, while their tag-aided method, AprilVINS, reached millimetre level. The code and data are announced for release on acceptance, so you cannot run it against your rig yet.
Put that next to the other two figures carefully. The original UMI paper reported a mean absolute trajectory error of 6.1 mm and 3.5 degrees against motion capture, and a mean inter-gripper relative pose error of 10.1 mm. HiFi-UMI reports 3 mm workspace-local end-effector accuracy. MILD reports SE(3)-aligned TCP-relative error. Those are three metrics on three setups, so they cannot be ranked on one scale, and a vendor who quotes one of them as if it were the others is telling you something about their rigour.
What the three papers agree on is that accuracy is a property of the localisation method and its setup, not of the UMI concept. The UMI reference repository's own README calls ORB-SLAM3 "still the most fragile part of UMI pipeline". HiFi-UMI met millimetre-grade accuracy as a condition; it did not prove that condition sufficient elsewhere. Unless your rig's error has been measured against an independent reference and lands in that range, plan the teleoperation anchor.
Decision table: UMI-only, UMI plus a teleop anchor, or teleop-only
Two rows deserve emphasis. The last one is a scope limit: every result cited here comes from tabletop gripper tasks, so treat handheld capture for dexterous hands as unproven. The force row is a physics limit: no amount of SLAM accuracy puts a torque channel into a video.
The middle column is the default for a reason. It lets handheld capture do what it is demonstrably good at, cheap scene and object diversity before the robot is even in the cell, while the anchor on the deployment robot supplies the embodiment, the contact behaviour and the exact camera geometry that the handheld data cannot. Moving a task from the middle column to the left one is a decision you make per task, after you have measured localisation error on that task's demonstrations, not a programme-wide switch.
Camera layout feeds the same decision. A study published in October 2026 (arXiv 2610.14075) found that adding a head view did not consistently improve task completion or execution quality on a bimanual platform when two large field-of-view wrist cameras already saw the objects, grippers and goals. Narrowing the wrist cameras raised the head view's value, and cropped head views gave more consistent gains than the full image. Our post on where to put the cameras for robot imitation learning covers placement in depth; the point here is that a UMI rig's camera layout should match what your deployment robot will see.
| Condition | UMI-only | UMI pre-training plus teleop anchor | Teleop-only |
|---|---|---|---|
| End-effector error measured against an independent reference, millimetre-grade | Candidate, if tasks resemble the published ones | Default | Not needed for this reason |
| Error unmeasured or centimetre-scale | No | Yes | Yes, if diversity is not the bottleneck |
| Tight-tolerance task (insertion, connector seating) | Only with millimetre accuracy and validation on the robot | Yes | Yes |
| Force- or torque-sensitive contact | No, UMI records no force | Anchor carries the contact skill | Yes |
| Device geometry or reach differs from the deployment robot | No | Yes, after kinematic filtering | Yes |
| Fast dynamic motion | Yes, with latency matching | Yes | Hard; a SpaceMouse yielded no tossing demos in 15 minutes |
| Many scenes, objects or sites needed | Strong | Strong | Expensive |
| Robot not delivered yet | Collect now | Collect UMI now, anchor later | Wait |
| Multi-finger dexterous hand | No evidence in the papers above | No evidence | Yes |
Acceptance checklist for a UMI dataset you buy or build
Whether the data comes from an internal team or a contracted collector, put these seven questions in writing before payment.
LeRobot has no UMI importer
The UMI reference code runs SLAM in a Docker image on the collector's machine and writes a Zarr replay buffer, dataset.zarr.zip, which its own training script consumes. LeRobot v0.6.1, released in August 2026, has no UMI importer. The feature request, issue 3157, has been open since March 2026 without a maintainer commitment, and a pull request adding a UMI example was closed unmerged in August. LeRobot did merge a UMI dataset importer in early 2024, in an older data path; it is not in v0.6.1. Representation is the second trap. LeRobot's relative actions at v0.6.1 subtract state from action element-wise and leave the observation state unchanged, while UMI relativises observations too and composes poses as transforms. A pull request to compose end-effector poses in SE(3) is open. Budget a conversion step with its own validation, and read our breakdown of why a VLA fine-tune can score 0% on the robot before the first training run, because an action-representation mismatch trains without an error.
| Check | What to ask for | Why it matters |
|---|---|---|
| SLAM success per session | Share of raw data used and number of dropped demonstrations, per session | The reference pipeline reports both; a low rate means collection technique or scene texture is wrong |
| Trajectory error | Error against mocap, a tracker or a robot end-effector reference on a sample, with the metric named | The only number that predicts whether the anchor can shrink |
| Temporal coverage | Fraction of each demonstration with a valid pose | MILD's second axis; gaps inside a grasp are worse than gaps at the start |
| Synchronisation | Camera, IMU and gripper-width timing and the latency figures used for matching | Latency matching moved tossing from 57.5% to 87.5% in the original paper |
| Kinematic filter | Proof that trajectories were filtered for your robot's reach and limits | The device cannot know your robot at capture time |
| Diversity counts | Scenes, objects and operators, counted | Diversity is what you are paying the handheld premium for |
| Rejection rules | Written per-episode rejection criteria and the rejected count | Keeps bad demonstrations out of training; see which episodes to cut from a LeRobot dataset |
Wearable capture on a regulated site
This is where the on-prem question changes the answer rather than decorating it. A 155 degree fisheye on a gripper, or HiFi-UMI's two wide-angle cameras per hand covering about 200 degrees, records everything around the task: operators' faces and badges, screens, batch labels, neighbouring product, and the layout of the line. On a pharma fill-finish line, a medical device cleanroom or a defence supplier's floor, that footage is a more sensitive artefact than the trajectories you actually want.
The reduction from video to trajectories is a compute step, and it can stay inside the site. The UMI reference SLAM runs in a container on the collector's own machine. HiFi-UMI runs its SLAM offline by design, so the optimiser can use future as well as past observations. Nothing in either pipeline requires a cloud service.
That turns the regulated buyer's decision into four written answers:
Teleoperation has the same camera exposure, but its recordings stay on the robot's own stack inside the cell. Handheld capture can reach the same posture if processing is local. The same questions apply to public corpora. HiFi-UMI-2K is published under CC BY 4.0, which settles the licence, but a licence says nothing about what is visible in the frames; review a sample before it enters a training set that your quality system has to describe. Once the corpus is reduced, keep the training side offline too; our notes on running LeRobot without Hugging Face calls cover that half. More on collection, formats and deployment sits on the physical AI pillar page.
What to do this week
Ask whoever is proposing UMI capture, internal team or vendor, for two numbers: the end-effector trajectory error on a sample of your own demonstrations against an independent reference, with the metric named, and the SLAM success rate per session. If they can produce both and the error is millimetre-grade, a UMI-heavy plan is defensible for tasks like the published ones. If they cannot produce both, plan UMI for pre-training and diversity, and keep a teleoperation anchor on the deployment robot.
FAQ
Quick answers to the questions this post tends to raise.
