Egocentric data hub
Egocentric Data for Physical AI
Egocentric data is first-person, head-mounted video used to train physical-AI and robotics models. This directory indexes egocentric datasets by environment (factory, home/ADL, retail, driving) and by task (hand-object interaction, hand tracking, gaze, activity recognition), and compares the open corpora — almost all non-commercial — against custom, consented capture.
Quick facts
- Resolution
- 1080p @ 30fps baseline; stereo 2160p @ 60fps and 4K available for pose-critical work
- Field of view
- ≥120° horizontal, head-mounted, to keep both hands and the work surface in frame
- Mount
- Head-mounted (helmet, cap, or glasses rig) — never chest or handheld
- Sensors
- RGB, IMU / head pose, Eye gaze (optional), Depth (optional), Audio (optional)
- Labels
- Frame-aligned action segments; Object and object-state annotations; 2D/3D hand pose (MANO / 25-joint)
- Volume
- Pilot batch in days; 40–500+ accepted hours per program
Key papers
Hard citations for the claims above. Each entry pairs a specific number with the paper that reports it.
EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
829 hours, 194 tasks. EgoDex pairs 829 hours of egocentric video across 194 tabletop tasks with 3D hand and finger tracking captured on Apple Vision Pro — the largest and most diverse dexterous human-manipulation dataset to date.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
3,670 hours, 74 locations. Ego4D spans 3,670 hours of daily-life first-person video from 931 camera wearers across 74 locations in 9 countries, collected under consenting-participant privacy and de-identification standards.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
1,286 hours, 740 participants. Ego-Exo4D pairs simultaneously-captured egocentric and exocentric video of skilled activity from 740 participants across 13 cities — 1,286 hours with multichannel audio, eye gaze, 3D point clouds, camera poses, and IMU.
What egocentric data captures
Egocentric data is first-person, head-mounted video recorded while a person performs a real task — the viewpoint a robot sees the world from. This directory indexes it by environment and by task. The capture settings the family spans:
- Assembly and repair on real factory lines, filmed head-mounted while hands manipulate parts and tools
- Homes during unscripted activities of daily living — cooking, cleaning, tidying, self-care
- Retail floors and stockrooms: shelf stocking, picking, checkout, and customer assistance
- Driver point-of-view and in-cabin capture that stays head-mounted rather than dashboard-fixed
- Fine-grained hand-object interaction: grasps, tool use, and bimanual manipulation with objects in frame
- Gaze-aligned task video where fixation and saccade streams are synchronized to first-person RGB
Why physical-AI labs need egocentric data
Apple's EgoDex — a large-scale egocentric video corpus built to learn dexterous manipulation — shows first-person footage is training signal, not just pretraining filler. [1]
EgoLive-scale corpora of real-world human tasks show frontier teams treating first-person human video as the base layer of the training stack. [2]
NVIDIA's Cosmos world-foundation-model platform serves as the backbone for world and action models post-trained on robotics and autonomous-vehicle data. [3]
The AoE line of work reframes the shortage of real-world interaction data as the bottleneck holding humanoids back — and scalable egocentric capture is the cheapest way to close it. [4]
EgoScale demonstrated that dexterous-manipulation performance keeps climbing as you scale diverse egocentric human data, which is exactly why labs are now buying it by the hour. [5]
Capture and delivery spec
Every egocentric data capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.
| Spec | Detail |
|---|---|
| Resolution | 1080p @ 30fps baseline; stereo 2160p @ 60fps and 4K available for pose-critical work |
| Frame rate | 30fps standard; 60fps for fast manipulation and gaze work |
| Field of view | ≥120° horizontal, head-mounted, to keep both hands and the work surface in frame |
| Mount | Head-mounted (helmet, cap, or glasses rig) — never chest or handheld |
| Sensors | RGB, IMU / head pose, Eye gaze (optional), Depth (optional), Audio (optional) |
| Labels | Frame-aligned action segments; Object and object-state annotations; 2D/3D hand pose (MANO / 25-joint); Gaze fixations; VLA-formatted episodes |
| QA gates | Hands-in-frame coverage; Head-mount stability / shake; Field-of-view and framing; Exposure and motion blur; Per-clip consent artifact present |
| Delivery | H.265 clips + per-clip JSON metadata, Hugging Face-streamable, with a signed consent and provenance manifest |
| Volume | Pilot batch in days; 40–500+ accepted hours per program |
Open egocentric datasets at a glance
The 14 open corpora most relevant to egocentric data are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. Only 1 of the 14 is permissively licensed for commercial use — which is the whole reason custom capture exists.
| Dataset | Size / scale | Sensors | License | Commercial use | Gap |
|---|---|---|---|---|---|
| Egocentric-10K | 10,000 h · 2,138 workers · 87 factories | Head-mounted RGB 1080p, no labels | Apache 2.0 | Yes | Raw video only — no action or pose labels, and no documented per-wearer consent chain. |
| HoloAssist | Two-person assembly/repair · HoloLens 2 | RGB + depth + eye + hand + IMU | CDLA-Permissive-style research | Conditional | Staged instructor/performer tasks; verify commercial terms before use. |
| Assembly101 | Toy take-apart/assemble · ego + exo multi-view | Multi-view RGB + 3D hand poses | Non-commercial research | No | Toy proxy, not a real facility; NC license blocks commercial training. |
| IndustReal | Assembly procedures on a construction-toy proxy | Head-mounted RGB | CC BY-SA | Conditional | Toy proxy; share-alike obligations complicate proprietary models. |
| MECCANO | Toy motorbike assembly · multimodal | RGB + depth + gaze | Research-only | No | Single toy task; research-only license. |
| ADL (UCI 2012) | 18 daily activities · chest-mounted · 2012 | Chest-mounted RGB | License unstated | Conditional | Chest-mount (not head), dated, and no clear license. |
| Charades-Ego | Scripted daily activities · paired ego/exo | RGB | Non-commercial | No | Activities are acted rather than natural; NC by default. |
| Aria Everyday Activities | ~7.3 h everyday activities · Project Aria | RGB + eye + IMU + SLAM | Non-commercial research | No | Tiny volume; NC license blocks product use. |
| Nymeria | Aria + full-body motion capture, in the wild | RGB + IMU + full-body motion | Non-commercial research | No | Motion-capture focus; NC license. |
| EGTEA Gaze+ | Cooking tasks · synchronized gaze · fine-grained actions | RGB + gaze | Research license | No | Kitchen-only; research license. |
| HOT3D | Mocap-grade 3D hand + object pose · Aria + Quest 3 | RGB + 3D hand/object ground truth | Non-commercial research | No | Gold-standard 3D ground truth you legally cannot ship in a product. |
| EgoDex | 25 joints per hand · dexterous manipulation | RGB + 25-joint hand pose | Non-commercial, no-derivatives | No | The annotation gold standard is NC-ND — untouchable commercially. |
| EgoLife | Multi-day, multi-person life-logging | RGB + audio + wearable sensors | MIT (claimed) with privacy exposure | Conditional | Card claims MIT but consent/privacy posture is unverified. |
| EgoVid-5M | 5M video-action clips for egocentric video generation | RGB + action annotations | Annotation layer open; video under source terms | Conditional | Built on source-corpus video terms; generation-focused, not real capture rights. |
Open datasets vs Truelabel custom capture
The catalog is brutal on rights. Almost every annotated egocentric corpus — EgoDex, HOT3D, Assembly101, EGTEA, the Aria family — ships under a non-commercial or no-derivatives license. The one big permissive raw corpus, Egocentric-10K, has no action labels and no documented consent chain. So 'just use the open data' quietly means 'ship a model you can't defend in a rights audit.'
Custom capture gives you spec control: your object set, your facility type, your sensor stack, your taxonomy — instead of bending a model to whatever a 2012 chest-mounted ADL set happened to record.
Consent and provenance are auditable per clip: a signed release from every wearer, documented bystander handling, and a manifest legal can actually read — the one thing no scraped or research corpus can hand you.
Exclusivity and freshness: commissioned footage isn't resold to your competitors, and you can refresh the object set as your robots, SKUs, or facilities change.
Egocentric data: by the numbers
The figures below are specific to egocentric data egocentric data and anchor the comparisons above.
- Only 1 of the 14 open egocentric corpora indexed here (Egocentric-10K) is permissively licensed for commercial use
- 26 egocentric data variants organized across the matrix
- 20,000+ collectors across 9 countries
- 10,000 hours across 2,138 workers and 87 factories in the single permissive corpus
- 829-hour annotation gold standard (EgoDex) locked behind an NC-ND license
- 5M video-action clips in the largest generation corpus (EgoVid-5M)
How custom capture works
Truelabel runs egocentric data programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to Pilot batch in days; 40–500+ accepted hours per program, delivered as H.265 clips + per-clip JSON metadata, Hugging Face-streamable, with a signed consent and provenance manifest. Go deeper via what egocentric data means, egocentric data licensing, egocentric kitchen video sourcing, egocentric warehouse video sourcing, and industrial egocentric video sourcing.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex is a large-scale egocentric video corpus built to learn dexterous manipulation from first-person human video.
arXiv ↩ - EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks
EgoLive is a large-scale egocentric dataset of real-world human tasks used as a pretraining substrate for manipulation policies.
arXiv ↩ - Physical AI with World Foundation Models | NVIDIA Cosmos
NVIDIA Cosmos world foundation models serve as backbones for world and action models post-trained on robotics and autonomous-vehicle data.
NVIDIA ↩ - AoE: Always-on Egocentric Human Video Collection for Embodied AI
AoE frames scalable, low-cost collection of egocentric human video as the answer to embodied-AI data scarcity.
arXiv ↩ - EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
EgoScale supports the claim that dexterous-manipulation performance scales with diverse egocentric human data.
arXiv ↩ - Egocentric-10K dataset card and license
The Egocentric-10K dataset card and license support the Apache-2.0 characterization and the 10,000 h / 2,138 workers / 87 factories figures.
Hugging Face - HoloAssist: an Egocentric Human Interaction Dataset
HoloAssist supports the catalog row for two-person egocentric assembly/repair captured on HoloLens 2 under a research license.
Microsoft Research - Assembly101: A Large-Scale Multi-View Video Dataset
Assembly101 supports the catalog row for a non-commercial procedural take-apart/assemble dataset with 3D hand poses.
Assembly101 project - IndustReal: industrial procedure-execution dataset
IndustReal supports the catalog row for a CC BY-SA egocentric assembly-procedure dataset on a toy proxy.
IndustReal project - MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain
MECCANO supports the catalog row for a research-only multimodal egocentric toy-assembly dataset.
University of Catania (IPLAB) - Detecting Activities of Daily Living in First-Person Camera Views (ADL dataset)
The UCI ADL dataset supports the catalog row for a 2012 chest-mounted first-person daily-living corpus with unstated license terms.
UC Irvine (Pirsiavash & Ramanan) - Charades-Ego: paired first- and third-person activity videos
Charades-Ego supports the catalog row for paired ego/exo scripted daily-activity video under a non-commercial license.
Allen Institute for AI (PRIOR) - Aria Everyday Activities (AEA)
Aria Everyday Activities supports the catalog row for ~7.3 h of everyday-activity egocentric recordings under a non-commercial research license.
Meta / Project Aria - Nymeria: egocentric full-body motion dataset
Nymeria supports the catalog row for an egocentric dataset paired with full-body motion capture under a non-commercial research license.
Meta / Project Aria - Extended GTEA Gaze+ (EGTEA Gaze+)
EGTEA Gaze+ supports the catalog row for egocentric cooking video with synchronized gaze under a research license.
Georgia Tech (First Person Vision) - HOT3D: egocentric hand and object tracking in 3D
HOT3D supports the catalog row for mocap-grade 3D hand and object pose ground truth under a non-commercial research license.
Meta Reality Labs - EgoDex: code and dataset release
EgoDex supports the catalog row for 25-joint-per-hand egocentric manipulation data under a non-commercial, no-derivatives license.
Apple - EgoLife
EgoLife supports the catalog row for multi-day, multi-person life-logging capture with an MIT-claimed card and privacy exposure.
EgoLife project - EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation
EgoVid-5M supports the catalog row for a 5M video-action egocentric generation dataset with an open annotation layer over source-corpus video.
arXiv
FAQ
Can we license your existing catalog, or is everything custom capture?
We broker custom, consented capture; we don't resell the open research corpora indexed on this page (most are non-commercial anyway). What you get is footage collected to your brief, with commercial rights and a consent chain attached.
Why not just use Ego4D or Egocentric-10K for free?
Egocentric-10K is Apache-2.0 but label-free and consent-undocumented; Ego4D and nearly every annotated set are research or non-commercial. For a product you need commercial rights plus labels plus a consent chain — which is precisely the gap custom capture fills.
Do we get exclusive rights, or will the same footage be sold to competitors?
Capture programs are commissioned for you. The footage and labels are yours; we don't resell a program's output to another buyer.
How was consent obtained from wearers and bystanders, and are faces and PII handled?
Every wearer signs a release, bystander handling follows the brief, face/PII blurring is available on request, and a per-clip consent artifact travels in the delivery manifest.
How much does egocentric video data cost?
Pricing is per accepted (QA-passed) hour. The drivers are environment access, sensor stack, annotation depth, and exclusivity — see the cost breakdown for ranges before you scope a program.
Can you match our object set or facility type, and do you capture IMU, gaze, and hand pose alongside RGB?
Yes. Object set and taxonomy are fixed in the brief; IMU and head pose are standard, and gaze, depth, and 2D/3D hand pose are optional add-ons.
Looking for egocentric data for physical AI?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request egocentric data