Industry · Egocentric data
Egocentric Data for Household Humanoid Robots
Egocentric data for humanoid robots is first-person, head-mounted video of people doing everyday household chores — the cheapest, most scalable way to pretrain a home humanoid's manipulation policies before it ever touches a real kitchen. The catch is embodiment and licensing: the open corpora that carry the retargetable hand and body motion a humanoid needs are non-commercial (Nymeria) or no-derivatives (EgoDex), so commercially-usable home-task data generally means consented custom capture matched to your robot's taxonomy.
Quick facts
- Resolution
- 1080p baseline; stereo 2160p on request for depth and 3D hand-pose retargeting
- Field of view
- ≥120° horizontal, with both hands and the manipulated object kept in frame
- Mount
- Head-mounted (glasses or cap rig), never chest or handheld — it preserves the gaze-aligned viewpoint a humanoid policy learns to act from
- Sensors
- RGB, IMU (head; optional wrist/body), optional 25-joint-per-hand 3D pose, optional full-body pose for embodiment retargeting, optional stereo depth
- Labels
- frame-aligned chore/task-step segments (verb + object); appliance and object state changes (open/closed, on/off, empty/full); optional 25-joint-per-hand pose plus full-body pose for retargeting
- Volume
- 60–300 accepted hours per home-task program (calibration pilot in days)
Key papers
Hard citations for the claims above. Each entry pairs a specific number with the paper that reports it.
EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
829 hours, 194 tasks. EgoDex pairs 829 hours of egocentric video across 194 tabletop tasks with 3D hand and finger tracking captured on Apple Vision Pro — the largest and most diverse dexterous human-manipulation dataset to date.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
3,670 hours, 74 locations. Ego4D spans 3,670 hours of daily-life first-person video from 931 camera wearers across 74 locations in 9 countries, collected under consenting-participant privacy and de-identification standards.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
1,286 hours, 740 participants. Ego-Exo4D pairs simultaneously-captured egocentric and exocentric video of skilled activity from 740 participants across 13 cities — 1,286 hours with multichannel audio, eye gaze, 3D point clouds, camera poses, and IMU.
What household humanoid pretraining needs from egocentric data
Egocentric data for humanoid robots is first-person, head-mounted video of people doing everyday household chores — the cheapest, most scalable way to pretrain a home humanoid's manipulation policies before it ever touches a real kitchen. The catch is embodiment and licensing: the open corpora that carry the retargetable hand and body motion a humanoid needs are non-commercial (Nymeria) or no-derivatives (EgoDex), so commercially-usable home-task data generally means consented custom capture matched to your robot's taxonomy.
The capture settings this covers:
- Kitchen chores a home robot inherits: loading and unloading the dishwasher, wiping counters, putting groceries away, and operating the fridge, microwave, and faucet
- Laundry end-to-end: sorting, loading the washer and dryer, folding shirts and towels, then putting clothes away in drawers and closets
- Tidying and object retrieval: picking clutter off the floor, fetching a named object from another room, clearing a table, and making a bed
- Doors, drawers, and articulated objects: opening cabinets and fridge doors, sliding drawers, turning handles and knobs — the articulated-object skills humanoids still fail on
- Deformable-object edge cases: handling bags, cloth, cables, and packaging; pouring, scooping, and two-handed carries at natural home pace
- Full-length routines, not clipped demos: uninterrupted multi-minute chore sequences so a policy sees task order, mistake recovery, and real household clutter
Why household humanoid pretraining needs first-person human video
Apple's EgoDex pretrains dexterous manipulation on large-scale egocentric human video, so a person's first-person view of household chores sits in the same training substrate a home humanoid policy learns from. [1]
EgoLive shows large-scale egocentric human demonstrations of real-world tasks lifting manipulation policies, so consented head-mounted footage of home tasks directly improves the humanoid skill it targets. [2]
AoE frames scalable, low-cost collection of egocentric human video of everyday tasks as an answer to the data scarcity a household-humanoid capture program is built to close. [3]
Capture and delivery spec
Every household humanoid pretraining capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.
| Spec | Detail |
|---|---|
| Resolution | 1080p baseline; stereo 2160p on request for depth and 3D hand-pose retargeting |
| Frame rate | 30 fps baseline; 60 fps for fast bimanual manipulation |
| Field of view | ≥120° horizontal, with both hands and the manipulated object kept in frame |
| Mount | Head-mounted (glasses or cap rig), never chest or handheld — it preserves the gaze-aligned viewpoint a humanoid policy learns to act from |
| Sensors | RGB, IMU (head; optional wrist/body), optional 25-joint-per-hand 3D pose, optional full-body pose for embodiment retargeting, optional stereo depth |
| Labels | frame-aligned chore/task-step segments (verb + object); appliance and object state changes (open/closed, on/off, empty/full); optional 25-joint-per-hand pose plus full-body pose for retargeting; per-take success, failure, and recovery flags; VLA-formatted episodes (LeRobot-compatible) |
| QA gates | hands-in-frame on manipulation frames above threshold; camera-stability and motion-blur check; household consent review with minors excluded; bystander and PII blur pass (faces, screens, documents, mail); per-clip consent artifact attached |
| Delivery | H.265 clips + per-episode JSON (steps, object states, hand and body pose, success flags); Hugging Face-streamable and LeRobot-compatible |
| Volume | 60–300 accepted hours per home-task program (calibration pilot in days) |
Open household humanoid pretraining datasets
The 5 open corpora most relevant to household humanoid pretraining are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. None of them are cleanly licensed for commercial model training — which is the whole reason custom capture exists.
| Dataset | Size / scale | Sensors | License | Commercial use | Gap |
|---|---|---|---|---|---|
| EgoDex | 829 h · 338,000 demos · 194 tasks (Apple Vision Pro) | Head-mounted RGB + 25-joint-per-hand 3D pose | CC BY-NC-ND (research) | No | Carries exactly the dexterous-hand pose a humanoid must retarget, but the no-derivatives license bars pretraining or shipping a commercial policy on it — and it is Vision Pro tabletop capture, not whole-home chores. |
| EgoMimic | Human + paired-robot bimanual demos (Georgia Tech, Project Aria) | Aria head-mount + robot teleop | Research release | No | Proves ego-to-humanoid transfer works, but it is one bimanual rig and a small task set — a method paper, not a home corpus you can pretrain on. |
| Nymeria | Large-scale · Aria glasses + full-body mocap, in the wild | RGB + IMU + full-body motion capture | Non-commercial research | No | Full-body motion is ideal for humanoid retargeting, yet labels are motion-centric with sparse object and task structure, and the NC license blocks commercial use. |
| EgoLife | Multi-day · multi-person home life-logging | Wearable RGB + audio | MIT on card (verify) | Conditional | The closest thing to real home life, but heavy bystander and PII exposure plus an unverified consent posture make it a legal risk to train a shipped product on. |
| Charades-Ego | 7,860 videos · 157 activity classes · paired ego/exo | Worn/handheld RGB | Non-commercial (research) | No | Scripted from prompts, so it lacks the improvised clutter and mistake-recovery a home humanoid must generalize to, and the NC license blocks commercial training. |
Open datasets vs Truelabel custom capture
Embodiment transfer is the whole point, and the open corpora that carry the action signal a humanoid must retarget — EgoDex's 25-joint hand pose and Nymeria's full-body mocap — are exactly the ones licensed no-derivatives or non-commercial. Custom capture is the only route to retargetable, whole-home manipulation data you can legally pretrain and ship a policy on.
Home-task taxonomy match: a household humanoid is judged on laundry, dishes, and tidying in a specific home layout, not on an academic action list. Custom capture lets you fix the chore taxonomy, the appliances and objects, and the takes-per-skill your policy actually trains on, rather than inheriting Charades-Ego's 157 scripted classes.
The household consent chain is the hard gate, not the license: filming inside real homes means consenting the household, excluding minors, and blurring bystanders and PII per clip — which is why frontier teams like Figure build proprietary home-data pipelines, such as its Brookfield real-estate partnership, instead of scraping the web.
The simulation objection is already answered by the buyers: Figure and Physical Intelligence pretrain on real human video and transfer it to robots precisely because home contact dynamics and clutter resist simulation — human-to-robot transfer is a shipped result, not a hope.
Household humanoid pretraining: by the numbers
The figures below are specific to household humanoid pretraining egocentric data and anchor the comparisons above.
- Figure's Project Go-Big (Sept 2025): internet-scale humanoid pretraining with direct human-video-to-robot transfer.
- Figure + Brookfield partnership: a frontier humanoid team building a proprietary real-world home-environment pretraining dataset rather than relying on public corpora.
- MIT Technology Review (Apr 2026): humanoid training data, not hardware, named the field's bottleneck.
- EgoDex ships 25-joint-per-hand 3D pose across 194 tasks under CC BY-NC-ND — the retargetable hand signal a humanoid needs, legally untouchable for commercial training.
- "egocentric data for humanoid robots" is the buyer conversation verbatim — Figure Go-Big, r/robotics threads, and vendor blogs frame it the same way.
How Truelabel captures household humanoid pretraining data
Truelabel runs household humanoid pretraining programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to 60–300 accepted hours per home-task program (calibration pilot in days), delivered as H.265 clips + per-episode JSON (steps, object states, hand and body pose, success flags); Hugging Face-streamable and LeRobot-compatible. Go deeper via humanoid robot training data, cross-embodiment transfer, VLA training data, imitation learning from human demonstrations, egocentric data licensing and commercial rights, and what egocentric data is.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex is a large-scale egocentric video corpus built to pretrain dexterous manipulation from first-person human video.
arXiv ↩ - EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks
EgoLive is a large-scale egocentric dataset of real-world human tasks used to lift manipulation policies.
arXiv ↩ - AoE: Always-on Egocentric Human Video Collection for Embodied AI
AoE frames scalable, low-cost collection of egocentric human video of manual tasks as an answer to embodied-AI data scarcity.
arXiv ↩ - Figure + Brookfield humanoid pretraining dataset partnership
Figure's Brookfield partnership shows a frontier humanoid team building proprietary real-world data from specific real-estate environments.
figure.ai - EgoDex: code and dataset release
The EgoDex release documents 25-joint-per-hand egocentric manipulation data across 194 tasks under a non-commercial, no-derivatives license.
Apple - EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
The EgoDex paper documents a large-scale egocentric manipulation corpus (829 hours, 338,000 demonstrations) for learning dexterous imitation from first-person human video.
arXiv - EgoMimic: Scaling Imitation Learning via Egocentric Video
EgoMimic demonstrates scaling imitation learning to a humanoid-style bimanual robot from egocentric human video paired with limited robot demonstrations.
EgoMimic (Georgia Tech) - Nymeria: egocentric full-body motion dataset
The Nymeria dataset pairs egocentric Aria video with full-body motion capture in the wild under a non-commercial research license.
Meta / Project Aria - Charades-Ego: paired first- and third-person activity videos
The Charades-Ego project page backs the 7,860-video, 157-class, scripted, paired ego/exo, non-commercial characterization.
Allen Institute for AI (PRIOR) - EgoLife
The EgoLife project page backs the multi-day multi-person home life-logging capture and the MIT-card-with-privacy-exposure caveat.
EgoLife project
FAQ
How does human egocentric video actually transfer to a humanoid's embodiment?
A head-mounted human video shares the humanoid's first-person viewpoint, and the hand and body motion in it can be retargeted onto the robot's morphology. That is what cross-embodiment transfer means in practice: the policy learns what the task looks like and how the hands move from human footage, then a smaller robot-specific set closes the gap between human and gripper. Figure's Project Go-Big and Physical Intelligence have both shown skills learned from human video transferring onto robots, which is why frontier teams pretrain on first-person human data before touching a real home.
Can you target our exact home-task taxonomy (laundry, dishes, tidying)?
Yes. We spec the chore taxonomy — your target tasks, appliances, objects, success criteria, and the number of takes per skill — around your robot, rather than inheriting a fixed academic action list. A pilot batch calibrates the taxonomy and QA gates, then accepted batches scale in whatever home layouts and demographics you need.
What does a home-capture consent chain look like (household members, minors excluded)?
Every wearer signs a consent agreement, and other household members are either consented or blurred; minors are excluded from capture. Each clip ships with a consent artifact plus a PII pass before delivery — faces, screens, documents, and mail are blurred by default — so the footage is defensible inside a commercial product.
Why not teleoperate our own humanoid instead?
Teleoperation is essential for embodiment-specific fine-tuning, but it is slow and expensive to scale — well past $50k per rig-seat — and it only produces data for the one robot in the loop. Head-mounted human capture collects far more diverse home-chore demonstrations per dollar, and the two are complementary: pretrain broadly on human video, then teleoperate a smaller set to close the embodiment gap.
Why not just use EgoDex or Nymeria for humanoid pretraining?
They carry the most useful signal — EgoDex has 25-joint-per-hand 3D pose across 194 tasks, Nymeria pairs egocentric video with full-body mocap — but EgoDex is CC BY-NC-ND and Nymeria is a non-commercial research license, so you cannot pretrain or ship a commercial home-robot policy on either. The permissive corpora at scale carry no hand or body pose at all. Custom capture is how you get retargetable, action-labeled home data you can actually train on.
Do you capture hand pose and full-body motion for retargeting?
Yes. RGB at 1080p is the baseline; head IMU, 25-joint-per-hand 3D pose, and full-body pose are add-ons, all frame-aligned to the video and delivered in a LeRobot-compatible episode format so your retargeting and behavior-cloning pipelines can ingest them directly.
Looking for egocentric data for humanoid robots?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request humanoid egocentric data