Where to Put the Cameras for Robot Imitation Learning

Three independent groups, two simulators and two real arms, measuring the same thing.

The close-laptop row survives at 19/30 because the task is coarse: a rough estimate of where the lid is still produces a working push. The three tasks needing the gripper at a specific pose go to near zero. Task tolerance decides how much viewpoint error you can absorb, not policy architecture.

The mechanism paper is blunt: "policies without extrinsics often infer camera pose using visual cues from static backgrounds in fixed scenes; this shortcut collapses when workspace geometry or camera placement shifts." That is view-invariant policy learning with camera conditioning. The RLBench rows come from a view-synthesis system, the Franka rows from a sim-to-real image translation study trained on 35,294 real fixed-camera observations plus 59,520 simulated.

Two reference layouts are fully verifiable, and both are more conservative than the specs circulating on vendor pages.

The ACT and ALOHA rig used four Logitech C922x webcams at 480x640 RGB, two on the wrists of the follower robots and two at the front and the top, recording at 50 Hz with 50 demonstrations per task. It has no ablation over camera count or resolution, so treat it as a working reference, not an optimum.

The LeRobot maintainers publish the other: preferably two views, the leader arm out of frame, the only moving objects being the follower arm and the manipulated items, at least 480x640 or 720p, approximately 30 FPS. Their recording guidance is blunter. "We suggest recording at least 50 episodes, with 10 episodes per location. Keep the cameras fixed and maintain consistent grasping behavior throughout the recordings." And steal their placement test: you should be able to do the task yourself by only looking at the camera images.

The wrist figures come from the robomimic study of offline human demonstrations, whose authors called wrist images and image randomization together "the difference between a proficient and non-proficient real-world policy". Transport is the two-arm task.

The view-count figures come from a multi-camera view scaling study testing 1, 3 and 5 views, which reports equivalent performance with 3 to 5 times fewer expert trajectories than a single-view baseline. On a real FANUC CRX-10iA with two cameras and 50 demonstrations, dual-view training moved success from 0.45 to 0.75 at 25 trials. That is the evidence behind one line in how much data you need to train a robot policy: a wrist camera and an extra angle beat another thousand episodes.

Capture resolution is not a quality dial, it is a token budget. The ACT backbone is a ResNet18 that divides each spatial dimension by 32, so 480x640 becomes a 15x20x512 feature map, 300 tokens per camera, and four cameras give a 1200x512 sequence.

That is arithmetic on the divide-by-32 rule and the quadratic cost of attention, not a measured benchmark, and it explains the most common LeRobot support thread. A report opened in October 2025 describes an SO-101 arm on an RTX 3060 with 8GB of VRAM, 49 episodes and about 39,000 frames across two 1080p views, ACT at roughly 26 million parameters trained for 100k steps, asking whether those images can be downscaled to 480p for larger batches. No maintainer answered, so take it as evidence of the constraint, not advice: at 1080p across two views, batch size is set by the camera spec.

High-resolution capture therefore buys nothing at the model input while costing storage, bandwidth and dataloader time every epoch; the GPU cost of that checkpoint choice is in OpenVLA vs pi0 vs SmolVLA vs GR00T N1. The ACT row is the harder constraint, the ACTConfig docstring stating that "Right now we only support all images having the same shape", so resolution is a rig-wide decision taken once.

Frame rate carries a subtler trap. lerobot-record paces the loop at --dataset.fps and reports what it achieved per episode and per session, down to the share of loop time spent in the observe step. A frame's timestamp is derived from its index, so a session that ran at 25 Hz still produces a dataset claiming 30, the recorded motion simply faster than reality. Cameras are usually what pushes those ticks over budget.

The direct fix is to stop making the policy guess. Conditioning on camera extrinsics through Plucker embeddings of per-pixel rays improved all three tested architectures on all six tasks.

All figures are success rates in percent. Gains are largest where the base policy was weakest, the signature of removing a failure mode rather than adding capability.

The rig-design consequence is one sentence from that paper: "to achieve the same performance, training without camera pose conditioning requires several times more cameras compared to training with pose conditioning." The supporting figure has a logarithmic camera-count axis and no numeric table behind it, so the multiplier is not recoverable, but extrinsics are cheaper than cameras.

One implementation detail matters if you build this: random cropping was applied to both the images and the Plucker maps. Crop the image without the ray map and you have taught the policy a lie.

Conditioning also couples to the action space: Delta End-Effector Pose actions measured best, ahead of Absolute Joint Position and Absolute End-Effector Pose, and the view-scaling group independently ranked camera-space actions above base-space above end-effector-space. Specify a relative pose in the collection contract, not absolute joint targets. There is no LeRobot flag for any of this.

Sometimes the camera has already moved, or three lines each frame the workspace differently. View augmentation buys back part of the loss, as successes over trials aggregated across three shifted viewpoints:

Augmentation recovers a real fraction of the collapse and never all of it: stack cups goes from 1/30 to 13/30, a large relative gain and still a policy you would not ship. The ranking is not uniform either, with VISTA ahead on three of four tasks and MANGO winning only on stack cups. Benchmark on your own tasks.

A newer line of work drops the calibration requirement. A calibration-free view-robust VLA published in July 2026 states the plant engineer's version: "Real-world robot deployment rarely maintains the training-stage camera setup, where cameras often experience repositioning or remounting depending on actual scenarios." It predicts a camera-centric end-effector action plus a 6-DoF hand-eye matrix and composes them geometrically. That is where the field is heading, not a production option yet.

What is definitely not the answer is LeRobot's built-in augmentation. The transforms are ColorJitter, SharpnessJitter and Identity, configured through ImageTransformsConfig (defaults enable=False, max_num_transforms=3) and applied at training time only. All of it is photometric, so --dataset.image_transforms is a lighting measure, never a viewpoint one.

The geometry that was true during collection must still be true at inference, and you must be able to prove it. LeRobot gives you the capture half and none of the record-keeping half.

lerobot-find-cameras opencv     # or: lerobot-find-cameras realsense

lerobot-record \
  --robot.type=so101_follower \
  --robot.port=/dev/tty.usbmodem585A0076841 \
  --robot.id=cell_a_follower \
  --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: /dev/video10, width: 640, height: 480, fps: 30}}" \
  --teleop.type=so101_leader \
  --teleop.port=/dev/tty.usbmodem58760431551 \
  --teleop.id=cell_a_leader \
  --display_data=true \
  --dataset.repo_id=acme/cell-a-tray-load \
  --dataset.single_task="Load the tray into the fixture" \
  --dataset.num_episodes=50 \
  --dataset.fps=30 \
  --dataset.root=/srv/robotdata/cell-a-tray-load \
  --dataset.push_to_hub=False

Two details are load-bearing. The camera keys become observation keys, so name them by location (front, top, left, right, wrist), never by device. And --dataset.push_to_hub=False with an explicit --dataset.root keeps the footage on the machine, which otherwise lands in ~/.cache/huggingface/lerobot/.

During recording, the left arrow or r re-records the current episode, right arrow or n ends it early, ESC or q stops and encodes. Use the re-record key whenever anyone bumps the rig. Afterwards, --dataset.episodes keeps a list of indices and --dataset.exclude_episodes drops more on top of it, documented as episodes to drop when corrupt or heterogeneous.

{
  "dataset": "acme/cell-a-tray-load",
  "rig_revision": "cellA-r3",
  "cameras": {
    "front": {
      "mount": "kinematic base plate, cell frame column B, torqued fasteners",
      "intrinsics_file": "calib/front_intrinsics_2026-08-10.yaml",
      "extrinsics_base_to_cam": "calib/front_extrinsics_2026-08-10.yaml",
      "verified_at_episodes": [0, 25, 49]
    },
    "wrist": {
      "mount": "printed bracket, link 6, two M3",
      "intrinsics_file": "calib/wrist_intrinsics_2026-08-10.yaml",
      "extrinsics_ee_to_cam": "calib/wrist_handeye_2026-08-10.yaml",
      "verified_at_episodes": [0, 25, 49]
    }
  }
}

camera_spec:                       # extends the schema clauses in the format contract
  views:
    - {name: front, mount: fixed_frame,   purpose: scene_layout}
    - {name: wrist, mount: link6_bracket, purpose: contact_detail}
  identical_across_views: true     # ACT requires one shape for the whole stack
  width: 640
  height: 480
  fps: 30
  geometry_record:
    intrinsics: per_camera_per_session
    extrinsics: per_episode        # nothing in LeRobotDataset v3 stores this
    format: sidecar_json_keyed_by_episode_index
  remount_tolerance:
    method: measured               # no published standard defines a number
    procedure: shift_in_known_increments_until_acceptance_fails
    recorded_in: acceptance_report
  revalidation_trigger:            # any of these voids the acceptance evidence
    - camera_moved_or_replaced
    - bracket_or_fixture_modified
    - workspace_background_changed
    - lens_or_focus_adjusted

Where the data comes from at all is a separate axis in teleoperation vs simulation vs human video. Camera geometry constrains all three the same way.

In a German plant, Betriebsverfassungsgesetz Section 87 paragraph 1 number 6 gives the works council co-determination over the "Einführung und Anwendung von technischen Einrichtungen, die dazu bestimmt sind, das Verhalten oder die Leistung der Arbeitnehmer zu überwachen." It applies whether or not monitoring is the intent, so mount design, framing and retention get negotiated before collection starts, and re-aiming a camera can reopen the agreement. The EU anchor is the EDPB guidance on processing personal data through video devices, adopted in January 2020.

The rig can also end up inside the safety argument. Under the EU AI Act, Annex I Section A item 1 is Directive 2006/42/EC on machinery, so a vision policy acting as a safety component of a covered machine takes the Article 6(1) high-risk route, which Article 113 dates from 2 August 2027, and the rig and its geometry record become technical documentation. Under the 2025 revision of ISO 10218, a camera bracket inside the safeguarded space is a modification to the robot application that pulls the cell's risk assessment back into scope.

The footage also cannot leave the building. A camera aimed at a cell records part geometry, fixture design, process sequence, cycle time and sometimes the operator, which in pharma, defence supply and medical device manufacturing is among the most sensitive data on site. Everything above runs next to the cell: lerobot-record writes to --dataset.root, --dataset.push_to_hub=False keeps it there, and the sidecar never touches a network. The cost side is in cloud versus on-premise AI security and cost, the contractual side in data privacy restrictions on AI training, and the Physical AI and robotics hub covers the boundary.

Photograph your rig with a scale reference in frame, measure the extrinsics of every camera against the robot base, and write them into a JSON sidecar keyed by episode index. That is an afternoon, and it is what you will wish you had the first time somebody moves a bracket.

Then find your own breaking point. Take a policy you already trained, shift one third-person camera in known increments (2 cm, 5 cm, 10 cm, then 10 and 20 degrees of yaw) and run 20 rollouts at each step. The increment where success crosses your acceptance bar is the re-mount tolerance for your next contract.