Use case · Egocentric data
Egocentric Video for World Models
World model training data is first-person video paired with the actions that produced it — ego-motion, hand pose, and frame-aligned action labels — so a model can learn how a scene changes when an agent acts. Action-conditioned POV video at commercial scale barely exists in the open, so world-model labs increasingly commission custom egocentric capture.
Quick facts
- Resolution
- 1080p baseline; stereo 2160p/60 for depth-sensitive world-model work
- Field of view
- ≥120° horizontal — wide enough to keep hands and manipulated objects in frame
- Mount
- Head-mounted (glasses or head-rig), not chest or handheld, for a true first-person viewpoint
- Sensors
- RGB, IMU (head motion / ego-motion), Optional gaze, Optional depth or stereo
- Labels
- Frame-aligned action segments; Ego-motion / head-pose track; Hand pose (optional 3D joints)
- Volume
- Pilot 40–120 accepted hours; production programs scale into the thousands of hours
Key papers
Hard citations for the claims above. Each entry pairs a specific number with the paper that reports it.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
3,670 hours, 74 locations. Ego4D spans 3,670 hours of daily-life first-person video from 931 camera wearers across 74 locations in 9 countries, collected under consenting-participant privacy and de-identification standards.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
1,286 hours, 740 participants. Ego-Exo4D pairs simultaneously-captured egocentric and exocentric video of skilled activity from 740 participants across 13 cities — 1,286 hours with multichannel audio, eye gaze, 3D point clouds, camera poses, and IMU.
EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
829 hours, 194 tasks. EgoDex pairs 829 hours of egocentric video across 194 tabletop tasks with 3D hand and finger tracking captured on Apple Vision Pro — the largest and most diverse dexterous human-manipulation dataset to date.
What world models needs from egocentric data
World model training data is first-person video paired with the actions that produced it — ego-motion, hand pose, and frame-aligned action labels — so a model can learn how a scene changes when an agent acts. Action-conditioned POV video at commercial scale barely exists in the open, so world-model labs increasingly commission custom egocentric capture.
The capture settings this covers:
- Everyday first-person navigation through homes, offices, and streets so the model learns how a scene reshapes as the wearer moves through it
- Object manipulation runs — pick, place, pour, open, close — captured with synchronized hand pose so each action sits next to its visual outcome
- Long-horizon multi-step tasks (cook a meal, assemble a shelf, tidy a room) that expose cause and effect over minutes rather than a two-second clip
- Deliberate action-outcome pairs — push a door, flip a switch, drop an object — that hand a world model clean interventional signal instead of ambiguous correlation
- The same tasks repeated across operators, lighting, and geographies so the learned dynamics generalize past a single building
Why world models needs first-person human video
World-foundation platforms like NVIDIA Cosmos serve as backbones for world and action models that are post-trained on robotics and autonomous-vehicle data — so more first-person, action-conditioned footage directly buys a better downstream policy. [1]
Apple's EgoDex — 829 hours of egocentric video built specifically to learn dexterous manipulation — shows that first-person human video transfers into robot manipulation skill, so labs building action-conditioned world models now treat egocentric human data as a primary training substrate, not a nice-to-have. [2]
The AoE line of work reframes the embodied-AI bottleneck as data, not model architecture: scalable, low-cost collection of egocentric human video is the answer to the scarcity of diverse real-world POV footage with the actions that produced it. [3]
Capture and delivery spec
Every world models capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.
| Spec | Detail |
|---|---|
| Resolution | 1080p baseline; stereo 2160p/60 for depth-sensitive world-model work |
| Frame rate | 30 fps baseline; 60 fps for fast-motion clips and prediction targets |
| Field of view | ≥120° horizontal — wide enough to keep hands and manipulated objects in frame |
| Mount | Head-mounted (glasses or head-rig), not chest or handheld, for a true first-person viewpoint |
| Sensors | RGB, IMU (head motion / ego-motion), Optional gaze, Optional depth or stereo |
| Labels | Frame-aligned action segments; Ego-motion / head-pose track; Hand pose (optional 3D joints); Object-state transitions; Natural-language action descriptions |
| QA gates | Hands and the manipulated object stay in frame; Stable, non-nauseating head motion; Resolution and FOV floor met; Action-label to video alignment verified; Consent artifact present for every clip |
| Delivery | H.265 clips plus per-clip JSON metadata (action segments, ego-motion, timestamps); Hugging Face-streamable |
| Volume | Pilot 40–120 accepted hours; production programs scale into the thousands of hours |
Open world models datasets
The 4 open corpora most relevant to world models are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. Only 1 of the 4 is permissively licensed for commercial use — which is the whole reason custom capture exists.
| Dataset | Size / scale | Sensors | License | Commercial use | Gap |
|---|---|---|---|---|---|
| EgoVid-5M | 5M action-annotated egocentric clips | RGB + action/kinematic annotations | Annotation layer open; source video under Ego4D terms | Conditional | Built for egocentric video generation, not action-conditioned control; commercial use is gated by Ego4D's signed license on the underlying frames. |
| Egocentric-10K | 10,000 h · 2,138 workers · 87 factories | Head-mounted RGB only | Apache-2.0 | Yes | Raw factory-floor video with no action labels, no ego-motion track, and no documented wearer consent chain — permissive, but unstructured for world-model training. |
| Nymeria | In-the-wild egocentric + full-body motion capture | Aria RGB + IMU + motion capture | Non-commercial research | No | Gold-standard motion ground truth, but the license bars any model you intend to sell. |
| Aria Everyday Activities | ~7.3 h of everyday activities | Aria RGB + IMU + eye tracking | Non-commercial research | No | Small and research-only — useful as a benchmark, not as a training substrate. |
Open datasets vs Truelabel custom capture
Almost no annotated egocentric corpus is commercially licensed. EgoVid-5M's action labels are open but the frames inherit Ego4D's signed-license terms; Nymeria and Aria Everyday Activities are non-commercial research only. Train a world model on them and you ship a model you cannot legally sell.
World models need action-conditioning, not just pixels. Passive POV video teaches appearance; a world model needs the paired ego-motion, hand pose, and action labels that let it learn 'if the agent does X, the scene becomes Y.' Custom capture bakes IMU, head-pose, and hand-pose sync into every clip — most open corpora do not.
Spec and taxonomy control. Open corpora froze their resolution, FOV, and label schema years ago. A custom program lets you set the 1080p/30 (or stereo 2160p/60) baseline, the ≥120° FOV, and your own action taxonomy so the training distribution matches the environments your model has to generalize to.
Exclusivity and a real consent chain. Custom footage is captured under a documented consent chain and can be held exclusive, so a competitor cannot train on the exact distribution you paid to create.
World models: by the numbers
The figures below are specific to world models egocentric data and anchor the comparisons above.
- EgoVid-5M packs 5 million action-annotated egocentric clips assembled specifically for egocentric video generation
- EgoVid-5M is the wedge in one dataset: its action labels are open, but its frames inherit Ego4D's signed-license terms
- Nymeria pairs Project Aria glasses with in-the-wild full-body motion capture — the motion ground truth world models want but can't commercially license
- World-model training needs action-conditioned POV (RGB + ego-motion + hand pose), not the passive video most open corpora ship
How Truelabel captures world models data
Truelabel runs world models programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to Pilot 40–120 accepted hours; production programs scale into the thousands of hours, delivered as H.265 clips plus per-clip JSON metadata (action segments, ego-motion, timestamps); Hugging Face-streamable. Go deeper via egocentric data licensing, what egocentric data is, industrial egocentric video capture, egocentric kitchen video, and egocentric warehouse video.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- Physical AI with World Foundation Models | NVIDIA Cosmos
NVIDIA Cosmos world foundation models serve as backbones for world and action models post-trained on robotics and autonomous-vehicle data.
NVIDIA ↩ - EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex is a large-scale egocentric video corpus built to learn dexterous manipulation, evidence that first-person human video transfers into robot manipulation skill.
arXiv ↩ - AoE: Always-on Egocentric Human Video Collection for Embodied AI
AoE frames scalable, low-cost collection of egocentric human video as the answer to embodied-AI data scarcity.
arXiv ↩ - EgoVid-5M
The EgoVid-5M project page is the official reference for the dataset's scale and its egocentric video-generation purpose.
EgoVid-5M project - EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation
The EgoVid-5M paper supports the 5-million action-annotated-clip figure and the annotation-layer-open / video-under-Ego4D-terms licensing characterization.
arXiv - Egocentric-10K
The Egocentric-10K dataset page is the reference for its head-mounted RGB, factory-floor, unlabeled scope.
Hugging Face - Egocentric-10K dataset card and license
The Egocentric-10K dataset card is the source for the Apache-2.0 license and the 10,000 h / 2,138 workers / 87 factories figures.
Hugging Face - Nymeria: egocentric full-body motion dataset
Nymeria is the reference for in-the-wild egocentric video paired with full-body motion capture under a non-commercial research license.
Meta / Project Aria - Aria Everyday Activities (AEA)
Aria Everyday Activities is the reference for a small (~7.3 h) non-commercial everyday-activity egocentric benchmark.
Meta / Project Aria - EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex supports the claim that dense (25-joint) hand annotation from egocentric video is tractable at scale, backing the monocular-to-3D-actions answer.
arXiv - EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
EgoScale supports the claim that scaling diverse real egocentric human data drives manipulation performance, backing the anti-simulation argument.
arXiv
FAQ
What is world model training data?
It is diverse, first-person video paired with the actions that produced it — ego-motion, hand pose, and frame-aligned action labels — so a model can learn how an environment responds when an agent acts. Passive POV video alone teaches appearance; the action-conditioning is what turns it into world model training data.
Why can't we just train on EgoVid-5M or Egocentric-10K?
You can experiment, but you can't safely ship. EgoVid-5M's action labels are open while the underlying frames inherit Ego4D's signed license, and Egocentric-10K is Apache-2.0 yet carries no action labels, ego-motion, or documented consent chain. Neither gives you commercially-clean, action-conditioned data at the spec a production world model needs.
Can you get 3D actions from monocular first-person video?
Yes, when you capture the right side channels. We sync IMU and head-pose — and, where needed, hand pose — to the RGB stream so each clip carries the ego-motion and action signal a world model conditions on, instead of trying to recover it from pixels after the fact. Work like EgoDex shows 25-joint hand annotation from egocentric video is tractable at scale.
Won't simulation solve the data problem?
Sim is great for cheap rollouts, but it can't originate the messy long-tail dynamics of real homes, hands, and objects — which is exactly why world-model labs keep buying real POV video. Real capture anchors the distribution and sim augments it; EgoScale's results point the same way, that scaling diverse real egocentric data is what moves the needle.
Do we get exclusive rights, or will the same footage be sold to competitors?
Custom capture can be held exclusive to you, under a documented consent chain, so the distribution you pay to create is not resold. That is the core reason labs commission capture instead of scraping open corpora — you control the spec, the taxonomy, and who else is allowed to train on it.
Looking for world model training data?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request world-model training data