Sensor · Egocentric data
Ego-Exo Paired Video Datasets
An ego-exo dataset pairs synchronized first-person (head-mounted) and third-person (external) video of the same activity, so a model can learn view-invariant representations and exo-to-ego transfer. The largest paired corpora — Ego-Exo4D, EgoExoLearn, Charades-Ego and Assembly101 — are signed-license, research-only or non-commercial, so hardware-synchronized custom capture with two rigs is the only clean route to commercially-usable paired data.
Quick facts
- Resolution
- 1080p @ 30fps per view baseline; 2160p exo for global body pose, stereo ego optional
- Field of view
- ≥120° ego for the full reach envelope; framed exo coverage of the body and workspace
- Mount
- Head-mounted ego rig + hardware-synced exo camera — fixed tripod, multi-cam ring, or a second wearer, chosen per transfer target
- Sensors
- Ego RGB (baseline), Exo RGB (one or more views), IMU (head), Gaze (optional), Timecode / genlock sync track
- Labels
- Cross-view synchronization track (per-pair timecode offset); Shared action / keystep segments aligned across both views; Ego-exo spatial correspondence (points or regions) where captured
- Volume
- 40–150 accepted paired hours per program; calibration pilot returns in days
Key papers
Hard citations for the claims above. Each entry pairs a specific number with the paper that reports it.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
1,286 hours, 740 participants. Ego-Exo4D pairs simultaneously-captured egocentric and exocentric video of skilled activity from 740 participants across 13 cities — 1,286 hours with multichannel audio, eye gaze, 3D point clouds, camera poses, and IMU.
120 hours. EgoExoLearn pairs egocentric task-execution video with exocentric demonstration video across 120 hours of daily-life and specialized-laboratory scenarios, with high-quality gaze and multimodal annotations, to model asynchronous ego-to-exo procedural learning.
Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities
4,321 videos, 8 exo + 4 ego views. Assembly101 is a multi-view procedural-activity dataset of 4,321 videos of people assembling and disassembling 101 take-apart toy vehicles, recorded simultaneously with 8 static and 4 egocentric cameras and annotated with over 1M fine-grained action segments and 18M 3D hand poses.
What ego-exo paired data contains
An ego-exo dataset pairs synchronized first-person (head-mounted) and third-person (external) video of the same activity, so a model can learn view-invariant representations and exo-to-ego transfer. The largest paired corpora — Ego-Exo4D, EgoExoLearn, Charades-Ego and Assembly101 — are signed-license, research-only or non-commercial, so hardware-synchronized custom capture with two rigs is the only clean route to commercially-usable paired data.
The capture settings this covers:
- A bimanual assembly or repair step filmed at the same instant from the worker's head-cam and a tripod facing the bench — the identical insert-and-fasten motion from both viewpoints, locked to the same frame.
- Demonstration-following: an instructor's first-person view of a procedure paired with a learner's third-person view of the same run, the way cooking, lab, and clinical protocols are actually taught.
- Dexterous, occlusion-heavy work — soldering, plating food, dry-lab suturing — where the exo camera recovers exactly what the closing hand hides from the ego view.
- Whole-body manipulation (lifting, carrying, loading, crouching) where a second wearer or a fixed exo rig captures the posture and reach envelope a head-cam alone never sees.
- Tool handovers and two-person coordination, framed from one participant's ego plus an external camera that keeps both people and the shared object in shot.
- Skilled motion for rehab or sports analysis, where the exo view supplies global body pose and the ego view supplies gaze and attention on the same take.
Why robotics and AI labs need ego-exo paired data
Apple's EgoDex pretrains dexterous manipulation on large-scale egocentric human video, the ego half of the paired ego-exo capture a cross-view policy learns from. [1]
EgoLive shows large-scale egocentric human demonstrations of real-world tasks lifting manipulation policies, so paired ego-exo footage of the same task directly improves the robot skill it targets. [2]
AoE frames scalable, low-cost collection of egocentric human video of manual tasks as an answer to the data scarcity a paired ego-exo corpus is built to close. [3]
Capture and delivery spec
Every ego-exo paired capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.
| Spec | Detail |
|---|---|
| Resolution | 1080p @ 30fps per view baseline; 2160p exo for global body pose, stereo ego optional |
| Frame rate | 30fps baseline, hardware-synced across ego and exo; 60fps for fast bimanual work |
| Field of view | ≥120° ego for the full reach envelope; framed exo coverage of the body and workspace |
| Mount | Head-mounted ego rig + hardware-synced exo camera — fixed tripod, multi-cam ring, or a second wearer, chosen per transfer target |
| Sensors | Ego RGB (baseline), Exo RGB (one or more views), IMU (head), Gaze (optional), Timecode / genlock sync track |
| Labels | Cross-view synchronization track (per-pair timecode offset); Shared action / keystep segments aligned across both views; Ego-exo spatial correspondence (points or regions) where captured; Per-hand pose and object states; Best-exo-view selection flags per keystep |
| QA gates | Ego-exo sync within tolerance (sub-frame timecode); Both views cover the action at every keystep; Exo framing keeps the wearer and the workspace in shot; Stability / no motion blur at the contact moments; Per-clip consent artifact for every wearer and bystander |
| Delivery | H.265 per view + cross-view sync JSON (timecode offsets, shared segments, correspondence), Hugging Face-streamable; consent artifacts per clip |
| Volume | 40–150 accepted paired hours per program; calibration pilot returns in days |
Open ego-exo paired datasets
The 4 open corpora most relevant to ego-exo paired are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. None of them are cleanly licensed for commercial model training — which is the whole reason custom capture exists.
| Dataset | Size / scale | Sensors | License | Commercial use | Gap |
|---|---|---|---|---|---|
| Ego-Exo4D | ~1,286 h · 740 participants · 13 cities | Synchronized ego (Aria: RGB + gaze + IMU) + multi-camera exo RGB | Ego-Exo4D signed research license | Conditional | The largest and best-annotated paired corpus, but the signed license and a fixed task/city set mean you benchmark on it, not ship a product trained on it. |
| EgoExoLearn | ~120 h of ego + exo demonstration-following | Ego RGB + exo RGB | Research-only | No | Pairs ego and exo of the SAME task recorded at DIFFERENT times — asynchronous by design, so it teaches procedure-following, not frame-synchronized correspondence. |
| Charades-Ego | 68,536 paired first/third-person clips · 7,860 videos · 157 action classes | Ego RGB + exo RGB | Non-commercial research (Charades license) | No | Scripted, acted home activities filmed by crowd workers — great for cross-view action recognition, but the behavior is staged and the license bars commercial training. |
| Assembly101 | ~513 h · 101 toy-vehicle (dis)assembly sequences · 8 exo + 4 ego views | Multi-view exo RGB + egocentric monochrome + 3D hand poses | Non-commercial research | No | A genuinely multi-view ego+exo rig, but the objects are toy vehicles and the NC license excludes any shipping product. |
Open datasets vs Truelabel custom capture
Every large paired corpus is off-limits commercially. Ego-Exo4D ships under a signed research license, EgoExoLearn is research-only, and Charades-Ego and Assembly101 are non-commercial. You can prototype exo-to-ego transfer on them; you cannot train a policy you intend to ship.
The open pairs are asynchronous or scripted. EgoExoLearn deliberately records the ego and exo of a task at different times, and Charades-Ego is acted home activity — neither gives you the hardware-frame-synchronized pairs a correspondence or view-invariant model actually needs. Custom capture genlocks both rigs to the same timecode.
The exo rig is a design decision the open datasets already froze for you. You get no say in whether the third-person view is a fixed tripod, a multi-camera ring, or a second wearer, nor in its framing. Custom capture lets you spec the exo geometry to the exact transfer you are training.
Paired footage doubles the consent surface — two participants, two viewpoints, usually faces. Custom capture attaches a per-clip consent artifact for every wearer and bystander and can grant exclusivity, so the same paired footage is not also feeding the lab you are competing with.
Ego-exo paired: by the numbers
The figures below are specific to ego-exo paired egocentric data and anchor the comparisons above.
- Ego-Exo4D: ~1,286 h from 740 participants across 13 cities, under a signed research license
- EgoExoLearn: ~120 h of asynchronous ego-exo demonstration-following (paired by task, not by frame)
- Charades-Ego: 68,536 paired first/third-person clips across 7,860 videos and 157 action classes, non-commercial
- Assembly101: 101 toy-vehicle assemblies captured across 8 exocentric + 4 egocentric views (~513 h)
- TrueLabel ego-exo programs: hardware-synced dual-rig capture at sub-frame timecode, 40–150 accepted paired hours per program
How Truelabel captures ego-exo paired data
Truelabel runs ego-exo paired programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to 40–150 accepted paired hours per program; calibration pilot returns in days, delivered as H.265 per view + cross-view sync JSON (timecode offsets, shared segments, correspondence), Hugging Face-streamable; consent artifacts per clip. Go deeper via what egocentric data is, egocentric data licensing, imitation learning, cross-embodiment transfer, VLA training data, and teleoperation data.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex is a large-scale egocentric video corpus built to pretrain dexterous manipulation from first-person human video.
arXiv ↩ - EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks
EgoLive is a large-scale egocentric dataset of real-world human tasks used to lift manipulation policies.
arXiv ↩ - AoE: Always-on Egocentric Human Video Collection for Embodied AI
AoE frames scalable, low-cost collection of egocentric human video of manual tasks as an answer to embodied-AI data scarcity.
arXiv ↩ - Ego-Exo4D project site
Ego-Exo4D is a large-scale synchronized ego-exo dataset (~1,286 h, 740 participants, 13 cities) released under a signed research license.
ego-exo4d-data.org - EgoExoLearn: bridging egocentric and exocentric skill learning
EgoExoLearn pairs ~120 h of egocentric and exocentric demonstration-following video recorded asynchronously, under research-only terms.
OpenGVLab - Charades-Ego: paired first- and third-person activity videos
Charades-Ego provides 68,536 paired first/third-person clips across 7,860 scripted home-activity videos under a non-commercial license.
Allen Institute for AI (PRIOR) - Assembly101: A Large-Scale Multi-View Video Dataset
Assembly101 captures 101 toy-vehicle (dis)assembly sequences across 8 exocentric and 4 egocentric views with 3D hand poses, non-commercial.
Assembly101 project - EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
EgoScale reports dexterous-manipulation performance scaling with the volume and diversity of egocentric human data.
arXiv
FAQ
What is an ego-exo dataset?
It is a collection where every clip exists as a synchronized pair: a first-person (head-mounted) view and a third-person (external) view of the same activity, ideally locked to the same frame. The pairing is what lets a model learn view-invariant representations and exo-to-ego transfer — mapping what a third-person demonstration looks like onto the first-person stream a robot actually perceives. Ego-Exo4D is the reference example, built with keystep and cross-view correspondence annotations for exactly this.
How are the ego and exo streams time-synchronized at capture, and to what tolerance?
Hardware sync, not best-effort alignment in post. Every rig shares a timecode or genlock track, and the delivered sync JSON records the per-pair offset so you can trust frame-to-frame correspondence at contact moments. Sub-frame tolerance is the QA gate; clips that drift are rejected before delivery.
Static tripod exo, or a second wearer?
Either, and it's a spec decision you make. A fixed tripod or multi-camera ring gives you stable global geometry for body-pose and correspondence work; a second wearer gives you a moving, naturalistic third-person view that tracks the action. We pick the exo configuration around the transfer you're training rather than shipping a frozen rig.
What does paired ego-exo train that ego-only video can't?
Three things ego-only misses: view-invariant encoders that recognize the same action from either viewpoint, exo-to-ego transfer that maps third-person human demonstrations onto a robot's first-person stream, and occlusion recovery where the exo view sees what the hand hides in ego. It's the same human-demonstration-to-robot bridge that EgoMimic and Physical Intelligence's human-to-robot work exploit, and manipulation performance scales with the volume of that human data.
Why not just use Ego-Exo4D?
Because its terms and its scope both fight a shipping product. Ego-Exo4D is a signed research license over a fixed set of tasks and 13 cities, so it's superb for benchmarking view-invariant methods but not something you can legally train a deployed policy on — and it won't match your objects, your workspace, or your exo geometry. Custom capture closes both gaps.
How is consent handled when two people and their faces are in frame?
Every clip carries a consent artifact for each wearer, and bystander handling is defined up front with framing and blur rules applied per your retention policy. Paired capture is where provenance matters most — two viewpoints multiply the PII surface — so the signed consent chain the open corpora rarely document is attached to every pair we deliver.
Looking for ego-exo dataset?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request ego-exo paired data