Egocentric Data for Imitation Learning

Quick facts

Resolution

1080p baseline; stereo 2160p on request for depth and 3D-hand fidelity

Field of view

≥120° horizontal, with hands and manipulated objects kept in frame

Mount

Head-mounted (glasses or head-rig), not chest or handheld — preserves the gaze-aligned manipulation viewpoint a policy learns from

Sensors

RGB, IMU (head motion), optional 25-joint-per-hand 3D pose, optional gaze, optional stereo depth

Labels

frame-aligned action/skill segments; per-take task success and failure flags; object and grasp-state annotations

Volume

80–400 accepted demonstration-hours per task family (pilot batch in days)

What imitation learning needs from egocentric data

An imitation learning dataset is a set of task demonstrations a policy learns to clone; egocentric human demonstrations are a cheaper, faster-to-scale alternative to teleoperation-rig data. The catch is licensing — the strongest open egocentric demonstration corpora are non-commercial or no-derivatives, so commercially-usable, action-labeled human demos generally require custom capture.

The capture settings this covers:

• Bimanual kitchen manipulation — pouring, chopping, opening jars, loading a dishwasher: the contact-rich two-hand tasks humanoid policies still fail at.
• Tool use and light assembly — driving screws, using a drill, threading cables, seating connectors: dexterous sequences with clear success and failure states.
• Deformable and articulated-object handling — folding laundry, opening drawers and doors, manipulating bags, cloth, and cabling under natural clutter.
• Retail and warehouse pick-and-place — grasping varied SKUs, bin-to-bin transfers, and packing at realistic pace and lighting.
• Repeated multi-take demonstrations of one target skill (10–50 takes per task) to give behavior-cloning the sample density it needs.

Why imitation learning needs first-person human video

Apple's EgoDex learns dexterous manipulation directly from large-scale egocentric human demonstrations, turning first-person human video into a first-class imitation-learning signal rather than a curiosity ^[1]

EgoScale shows manipulation success rate scaling log-linearly with the volume of diverse egocentric human-demonstration data, so demonstration hours — not just model size — are now the lever labs are racing to pull ^[2]

The industry consensus is that real-world demonstration data, not compute, is the bottleneck holding humanoid and manipulation policies back — the AoE line of work frames scalable egocentric collection as the fix — which is why every serious lab is now buying human demonstrations ^[3]

Capture and delivery spec

Every imitation learning capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.

Open imitation learning datasets

The 5 open corpora most relevant to imitation learning are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. Only 1 of the 5 is permissively licensed for commercial use — which is the whole reason custom capture exists.

Open datasets vs Truelabel custom capture

The commercially-usable-AND-demonstration-labeled intersection is empty: the annotation gold standard for imitation learning is EgoDex's per-hand pose data, and it ships non-commercial and no-derivatives ^[4], while the one Apache-2.0 corpus at scale, Egocentric-10K, has no action labels or hand pose at all. Custom capture is the only route to demonstrations you can legally train and ship a policy on.

Spec control: imitation learning needs action signals — hand pose, head pose, IMU — frame-aligned to RGB and organized around your policy's objects and skills. Open corpora fix the task list (kitchen, tabletop, life-logging); custom capture lets you specify the demonstration taxonomy, the number of takes per skill, and the success criteria your behavior-cloning run actually optimizes.

Exclusivity and provenance: a public corpus is downloaded by every competitor, and NC/scraped data leaves legal risk baked into your weights. A commissioned demonstration set can be delivered exclusively, with per-clip consent and provenance so the training license is defensible.

The sim objection does not hold yet: frontier labs are buying and pretraining on real human egocentric demonstrations precisely because human-to-robot transfer works and diverse real contact dynamics remain hard to simulate — EgoDex and EgoScale show the same for imitation learning directly ^{[4] [2]}.

Imitation learning: by the numbers

The figures below are specific to imitation learning egocentric data and anchor the comparisons above.

• EgoDex: 829 hours, 338,000 demonstrations across 194 tasks, 25-joint 3D pose per hand — all under a non-commercial, no-derivatives license.
• Physical Intelligence reported ~2x improvement in robot skill acquisition from adding egocentric human demonstrations (Dec 2025).
• EgoScale: manipulation success rate scales log-linearly with diverse egocentric human-demonstration hours.
• Egocentric-10K: 10,000 Apache-2.0 hours across 2,138 workers and 87 factories — but zero demonstration or action labels.
• Teleoperation demonstration rigs run past $50k per seat; head-mounted human capture reaches comparable demonstration volume at a fraction of the cost.

How Truelabel captures imitation learning data

Truelabel runs imitation learning programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to 80–400 accepted demonstration-hours per task family (pilot batch in days), delivered as H.265 clips plus per-episode JSON (actions, poses, success flags), HF-streamable and LeRobot-compatible episode format. Go deeper via how egocentric data licensing works, what egocentric data is, industrial first-person capture, kitchen manipulation capture, and warehouse pick-and-place capture.

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

External references and source context

1. EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video

   EgoDex learns dexterous manipulation directly from large-scale egocentric human demonstrations.

   arXiv ↩
2. EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data

   EgoScale supports the log-linear scaling of dexterous-manipulation success with diverse egocentric human-demonstration data.

   arXiv ↩
3. AoE: Always-on Egocentric Human Video Collection for Embodied AI

   AoE frames scalable, low-cost collection of egocentric human demonstrations as the answer to the real-world data bottleneck for humanoid and manipulation policies.

   arXiv ↩
4. EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video

   The EgoDex paper documents a large-scale egocentric manipulation corpus for learning dexterous imitation from first-person human video.

   arXiv ↩
5. EgoDex: code and dataset release

   The EgoDex code/dataset release documents its 25-joint-per-hand egocentric manipulation data and its non-commercial, no-derivatives license.

   Apple
6. EgoMimic: Scaling Imitation Learning via Egocentric Video

   EgoMimic demonstrates scaling imitation learning directly from egocentric human video paired with limited robot demonstrations.

   EgoMimic (Georgia Tech)
7. EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks

   EgoLive is a large-scale egocentric dataset of real-world human tasks whose commercial license terms are unverified.

   arXiv
8. Egocentric-10K dataset card and license

   The Egocentric-10K dataset card supports the Apache-2.0 license and the 10,000-hour / 2,138-worker / 87-factory scale, and that it ships no action labels.

   Hugging Face
9. HOT3D: egocentric hand and object tracking in 3D

   HOT3D provides mocap-grade 3D hand and object pose ground truth under a non-commercial research license.

   Meta Reality Labs
10. Project Go-Big: Internet-Scale Humanoid Pretraining and Direct Human-to-Robot Transfer

    Figure's Project Go-Big supports large-scale egocentric human video as a pretraining substrate for humanoid manipulation.

    Figure

FAQ