Task · Egocentric data
Hand-Object Interaction (HOI) Egocentric Datasets
A hand-object interaction (HOI) dataset records how hands reach, grasp, manipulate, and release objects from a first-person camera, usually with frame-aligned 3D hand pose and object-contact labels. The strongest HOI corpora — HOT3D, HOI4D, EgoDex, Assembly101 — are all non-commercial or access-gated, so robotics teams turn to custom capture for commercially-licensed manipulation video.
Quick facts
- Resolution
- 1080p baseline; 2160p stereo for close-range dexterity work
- Field of view
- ≥120° horizontal so both hands and the manipulated object stay in frame through the reach
- Mount
- Head-mounted (glasses or GoPro-style rig) — never chest or handheld, which lose the hands during fine manipulation
- Sensors
- RGB, IMU (head, optional wrist), Optional depth for object pose, Optional gaze to mark attention-to-object
- Labels
- Frame-aligned action segments (reach / grasp / manipulate / release); Per-hand 3D pose (MANO or 21–25 joint skeleton); Object bounding boxes and 6-DoF pose where captured
- Volume
- 60–200 accepted hours per manipulation program
Key papers
Hard citations for the claims above. Each entry pairs a specific number with the paper that reports it.
EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
829 hours, 194 tasks. EgoDex pairs 829 hours of egocentric video across 194 tabletop tasks with 3D hand and finger tracking captured on Apple Vision Pro — the largest and most diverse dexterous human-manipulation dataset to date.
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100
90K action segments on 100 hours. EPIC-KITCHENS-100 densely annotates 100 hours (20M frames) with roughly 90,000 action segments across 45 kitchens — the labeled-hour density that raw first-person corpora cannot match.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
1,286 hours, 740 participants. Ego-Exo4D pairs simultaneously-captured egocentric and exocentric video of skilled activity from 740 participants across 13 cities — 1,286 hours with multichannel audio, eye gaze, 3D point clouds, camera poses, and IMU.
What hand-object interaction (HOI) data contains
A hand-object interaction (HOI) dataset records how hands reach, grasp, manipulate, and release objects from a first-person camera, usually with frame-aligned 3D hand pose and object-contact labels. The strongest HOI corpora — HOT3D, HOI4D, EgoDex, Assembly101 — are all non-commercial or access-gated, so robotics teams turn to custom capture for commercially-licensed manipulation video.
The capture settings this covers:
- A technician's hands seating a bearing into a housing — the full reach, grasp, align, insert, release cycle in one clip.
- Bimanual coordination: one hand steadies a jar while the other breaks the lid seal and twists it off.
- Mid-task re-grasping — a hand shifts a power drill from a carry grip to a trigger grip without setting it down.
- Fine finger work: pinching, rotating, and transferring small parts or ingredients between containers.
- Pick-and-place of occluded or deformable objects, where the object briefly disappears inside the closing hand.
- Tool handovers between two people during an assembly or repair sequence.
Why robotics and AI labs need hand-object interaction (HOI) data
EgoLive-scale egocentric human manipulation video shows that first-person hand-object footage is training fuel, not just benchmark material — the clearest signal yet that these corpora feed real robot policies. [1]
EgoScale showed dexterous-manipulation policy performance scaling log-linearly with the volume and diversity of egocentric human hand-object data, which turns 'collect more HOI clips' into a measurable ROI lever. [2]
Apple's EgoDex release — large-scale egocentric video built specifically to learn dexterous manipulation — confirms that even the biggest labs now bootstrap grasp models from first-person hand-object footage rather than teleoperation alone. [3]
Capture and delivery spec
Every hand-object interaction (HOI) capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.
| Spec | Detail |
|---|---|
| Resolution | 1080p baseline; 2160p stereo for close-range dexterity work |
| Frame rate | 30 fps baseline; 60 fps for fast re-grasps and contact transitions |
| Field of view | ≥120° horizontal so both hands and the manipulated object stay in frame through the reach |
| Mount | Head-mounted (glasses or GoPro-style rig) — never chest or handheld, which lose the hands during fine manipulation |
| Sensors | RGB, IMU (head, optional wrist), Optional depth for object pose, Optional gaze to mark attention-to-object |
| Labels | Frame-aligned action segments (reach / grasp / manipulate / release); Per-hand 3D pose (MANO or 21–25 joint skeleton); Object bounding boxes and 6-DoF pose where captured; Contact state and grasp-type taxonomy; Bimanual coordination flags |
| QA gates | Both hands in frame during every contact event; No motion blur at the grasp and release moments; Manipulated object visible at approach and at contact; Stable exposure across skin tones and object surfaces |
| Delivery | H.265 clips + per-clip JSON (hand pose, object pose, contact events), Hugging Face-streamable, with a consent artifact attached to every clip |
| Volume | 60–200 accepted hours per manipulation program |
Open hand-object interaction (HOI) datasets
The 6 open corpora most relevant to hand-object interaction (HOI) are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. None of them are cleanly licensed for commercial model training — which is the whole reason custom capture exists.
| Dataset | Size / scale | Sensors | License | Commercial use | Gap |
|---|---|---|---|---|---|
| HOT3D | Egocentric sequences with per-frame 3D hand + object pose (Aria + Quest 3) | RGB + mocap-grade 3D hand and object pose ground truth | Non-commercial research license | No | Mocap GT is the field's gold standard, but the license bars any shipping product; object set is a fixed lab inventory, not your SKUs. |
| HOI4D | Millions of RGB-D frames of dexterous grasps on YCB-style objects, 3D annotated | RGB-D + 3D object/part annotations | Research-gated access | No | Category-level 3D labels are excellent, but access is gated and the object taxonomy is fixed to the paper's categories. |
| EgoDex | 829 hours of egocentric manipulation, 25 tracked joints per hand | RGB + 25-joint-per-hand pose | Non-commercial, no-derivatives | No | The largest annotated manipulation corpus here — and the NC-ND terms mean you cannot train a commercial policy on any of the 829 hours. |
| Assembly101 | Procedural take-apart / reassemble toy sequences, ego + exo, with 3D hand poses | Multi-view RGB + 3D hand poses | Non-commercial research license | No | Rich procedural structure, but the objects are toys — the contact distribution won't match a real bearing, valve, or connector. |
| HoloAssist | Two-person interactive assembly/repair tasks captured on HoloLens 2 | Egocentric RGB + head/hand signals | CDLA-Permissive-style research license (verify terms) | Conditional | The most permissive license in the set, but tasks are staged instructor-follower repairs, not autonomous single-operator manipulation. |
| MECCANO | Multimodal egocentric build of a toy motorbike (RGB + depth + gaze) | RGB + depth + gaze | Research-only | No | Useful for gaze-conditioned HOI research, but one toy-assembly domain and no commercial-use path. |
Open datasets vs Truelabel custom capture
Every gold-standard HOI corpus is off-limits commercially. HOT3D and HOI4D are non-commercial or access-gated [4], EgoDex ships hundreds of hours under a no-derivatives, non-commercial license [3], and Assembly101 is research-only [5]. You can benchmark on them; you cannot ship a product trained on them.
The permissive-ish options annotate toys. Assembly101, IndustReal, and MECCANO capture a construction set, a toy-build proxy, and a toy motorbike [5] — not the bearings, valves, connectors, or tools your robot will actually grasp. Custom capture lets you specify the exact objects, so the contact distribution matches deployment.
Open corpora freeze their own hand and object taxonomies. Custom capture lets you define the grasp-type set, the contact-event schema, and the bimanual flags your policy consumes, delivered in your episode format instead of reverse-engineered from someone else's label spec.
HOI footage shows hands and often faces, so provenance is not optional. Custom capture gives you a per-clip consent chain and the option of exclusivity, so the same manipulation footage isn't also sold to the lab training against you.
Hand-object interaction (HOI): by the numbers
The figures below are specific to hand-object interaction (HOI) egocentric data and anchor the comparisons above.
- EgoDex: 829 non-commercial hours, 25 tracked joints per hand
- HOT3D: per-frame 3D hand pose paired with 6-DoF object pose on Aria + Quest 3
- IndustReal: roughly six hours of assembly under CC BY-SA (toy-construction proxy)
- MECCANO: RGB + depth + gaze captured across a single toy-motorbike build
- Assembly101: procedural take-apart/reassemble toy sequences with 3D hand poses
How Truelabel captures hand-object interaction (HOI) data
Truelabel runs hand-object interaction (HOI) programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to 60–200 accepted hours per manipulation program, delivered as H.265 clips + per-clip JSON (hand pose, object pose, contact events), Hugging Face-streamable, with a consent artifact attached to every clip. Go deeper via egocentric data, egocentric data licensing, industrial egocentric video, kitchen manipulation footage, and warehouse egocentric video.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks
EgoLive is a large-scale egocentric dataset of real-world human tasks that shows first-person hand-object footage feeds robot manipulation policies.
arXiv ↩ - EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
EgoScale reports dexterous-manipulation performance scaling with the volume and diversity of egocentric human data.
arXiv ↩ - EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex is a large-scale egocentric video corpus built to learn dexterous hand-object manipulation.
arXiv ↩ - HOT3D: egocentric hand and object tracking in 3D
HOT3D provides mocap-grade 3D hand and object pose ground truth on Aria + Quest 3 under a non-commercial research license.
Meta Reality Labs ↩ - Assembly101: A Large-Scale Multi-View Video Dataset
Assembly101 is a procedural take-apart/reassemble toy dataset with 3D hand poses under a non-commercial research license.
Assembly101 project ↩ - EgoDex: code and dataset release
EgoDex ships 829 hours with 25 joints per hand under a non-commercial, no-derivatives license.
Apple - HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction
HOI4D is a dexterous hand-object interaction dataset of 3D-annotated grasps on YCB-style objects, with gated access.
hoi4d.github.io - HoloAssist: an Egocentric Human Interaction Dataset
HoloAssist captures two-person interactive assembly/repair tasks on HoloLens 2 under a CDLA-Permissive-style research license.
Microsoft Research - IndustReal: industrial procedure-execution dataset
IndustReal is a ~6-hour egocentric assembly dataset on a toy-construction proxy, CC BY-SA licensed.
IndustReal project - MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain
MECCANO is a multimodal (RGB, depth, gaze) research-only egocentric dataset of a toy-motorbike assembly.
University of Catania (IPLAB) - Project Go-Big: Internet-Scale Humanoid Pretraining and Direct Human-to-Robot Transfer
Figure's Project Go-Big centers egocentric pretraining for humanoid manipulation.
Figure - Humanoid data: 10 Things That Matter in AI Right Now | MIT Technology Review
MIT Technology Review documents the manipulation-data bottleneck holding back humanoid robots.
MIT Technology Review - Physical AI with World Foundation Models | NVIDIA Cosmos
NVIDIA Cosmos world/action-model work underscores the value of large egocentric video corpora for physical AI.
NVIDIA
FAQ
What is a hand-object interaction dataset?
It is a collection of first-person video in which the labels describe how hands engage objects: the reach-grasp-manipulate-release phases, per-hand 3D pose, object bounding boxes or 6-DoF pose, and contact or grasp-type state. HOI data is what grasp and dexterity policies learn from, and the same first-person corpora increasingly feed world and action models too.
Why can't I just train on HOT3D, HOI4D, or EgoDex?
Because the best HOI corpora are legally untouchable for commercial work. HOT3D is a non-commercial research license, HOI4D is access-gated, and EgoDex is non-commercial and no-derivatives. They are excellent for prototyping and benchmarking; a shipping manipulation policy needs footage you actually hold the rights to.
Do you capture 3D hand pose and object pose, or just RGB video?
Both, to spec. The baseline is head-mounted RGB with frame-aligned reach/grasp/manipulate/release segments and per-hand 21–25 joint skeletons. We add depth for object 6-DoF pose, IMU for head and wrist motion, and gaze to mark which object the wearer is attending to — the same signal stack that makes HOT3D-style contact modeling possible.
Can you match our object set and grasp types?
Yes — that is the point of custom capture. You give us the SKUs, tools, and materials your robot will handle, and we define the grasp-type and contact taxonomy around them. Open HOI datasets can't do this: Assembly101 and MECCANO are locked to toy builds, so their contact distributions don't transfer to real hardware.
How is consent handled when hands and faces are in frame?
Every clip carries a consent artifact tied to the wearer, bystander handling is defined up front, and PII is treated per your retention rules — the provenance chain the open corpora rarely document.
How much egocentric manipulation data do robotics teams actually need?
More than you'd guess, and it pays off predictably. EgoScale found dexterous-manipulation performance scaling log-linearly with egocentric human data volume and diversity, and Apple built EgoDex at large scale for exactly that reason. It is the same real-world-data bottleneck that constrains humanoids and the reason frontier teams lean on egocentric pretraining. Most programs start with a 60–200 accepted-hour pilot on a target object set, then scale once the policy responds.
Looking for hand object interaction dataset?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request HOI egocentric data