Task · Egocentric data
Hand Tracking & Pose Egocentric Datasets
A hand tracking dataset provides frame-aligned 2D/3D hand keypoints — usually a 21-point MANO or 25-joint skeleton per hand — from egocentric video. The mocap-grade open corpora (EgoDex, HOT3D, Assembly101) are non-commercial or no-derivatives, so commercially-licensed hand-pose data effectively has to be captured to order.
Quick facts
- Resolution
- 1080p baseline; 1440p–2160p for fine finger detail on small parts
- Field of view
- ≥110° horizontal so both hands stay in frame during bimanual work
- Mount
- Head-mounted (glasses or head strap) — never chest or handheld; the hands must be seen from the operator's own viewpoint
- Sensors
- RGB (primary), IMU (head, optional wrist) for monocular 3D lift, Optional second synchronized camera or native headset hand-tracking for metric 3D, Optional depth
- Labels
- Per-frame 2D keypoints (21-point MANO or 25-joint skeleton); 3D joints with per-joint confidence where captured; Left/right hand disambiguation
- Volume
- 40–150 accepted hours per pilot, scaling to 1,000+ for pretraining
Key papers
Hard citations for the claims above. Each entry pairs a specific number with the paper that reports it.
EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
829 hours, 194 tasks. EgoDex pairs 829 hours of egocentric video across 194 tabletop tasks with 3D hand and finger tracking captured on Apple Vision Pro — the largest and most diverse dexterous human-manipulation dataset to date.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
1,286 hours, 740 participants. Ego-Exo4D pairs simultaneously-captured egocentric and exocentric video of skilled activity from 740 participants across 13 cities — 1,286 hours with multichannel audio, eye gaze, 3D point clouds, camera poses, and IMU.
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100
90K action segments on 100 hours. EPIC-KITCHENS-100 densely annotates 100 hours (20M frames) with roughly 90,000 action segments across 45 kitchens — the labeled-hour density that raw first-person corpora cannot match.
What hand tracking & pose data contains
A hand tracking dataset provides frame-aligned 2D/3D hand keypoints — usually a 21-point MANO or 25-joint skeleton per hand — from egocentric video. The mocap-grade open corpora (EgoDex, HOT3D, Assembly101) are non-commercial or no-derivatives, so commercially-licensed hand-pose data effectively has to be captured to order.
The capture settings this covers:
- Bimanual assembly and disassembly where each hand articulates independently — fasteners, connectors, cable routing, snap-fits
- Fine-motor finger work: typing, buttoning, threading, and handling parts under 10 mm where individual joints matter
- Tool grips sampled across the grasp taxonomy — power grip, precision pinch, lateral/key pinch, tripod — captured mid-task rather than posed
- In-hand reorientation and regrasping: rotating or repositioning a part between the fingers without setting it down
- Reach-to-grasp trajectories where the hand enters and exits the field of view against real clutter and occlusion
- Static and dynamic gesture vocabularies — pointing, counting, sign-like handshapes — for gesture recognition and human-robot interaction
Why robotics and AI labs need hand tracking & pose data
EgoScale showed manipulation policies improving as egocentric human hand-motion demonstrations scale in volume and diversity — hand trajectories are the transferable action representation. [1]
NVIDIA's Cosmos world- and action-model program treats per-frame hand keypoints as the action channel that turns passive first-person video into trainable demonstrations, putting a premium on pose-labeled egocentric corpora. [2]
Apple's EgoDex paired 829 hours of egocentric video with 25-joint-per-hand pose captured at recording time, and reported that dexterous manipulation improves with the volume of paired hand-pose demonstrations — a scaling curve labs now want to buy into. [3]
Capture and delivery spec
Every hand tracking & pose capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.
| Spec | Detail |
|---|---|
| Resolution | 1080p baseline; 1440p–2160p for fine finger detail on small parts |
| Frame rate | 30 fps baseline, 60 fps for fine-motor and fast articulation |
| Field of view | ≥110° horizontal so both hands stay in frame during bimanual work |
| Mount | Head-mounted (glasses or head strap) — never chest or handheld; the hands must be seen from the operator's own viewpoint |
| Sensors | RGB (primary), IMU (head, optional wrist) for monocular 3D lift, Optional second synchronized camera or native headset hand-tracking for metric 3D, Optional depth |
| Labels | Per-frame 2D keypoints (21-point MANO or 25-joint skeleton); 3D joints with per-joint confidence where captured; Left/right hand disambiguation; Grasp-type tags (power, precision pinch, lateral, tripod); Frame-aligned action segments |
| QA gates | Hands-in-frame across the full clip; Motion-blur threshold on finger regions; Left/right label agreement; Temporal keypoint stability (no jitter or teleporting joints); Lighting floor for finger visibility |
| Delivery | H.265 video + per-clip JSON keypoint tracks (MANO or 25-joint), Hugging Face-streamable, with per-clip consent artifacts |
| Volume | 40–150 accepted hours per pilot, scaling to 1,000+ for pretraining |
Open hand tracking & pose datasets
The 4 open corpora most relevant to hand tracking & pose are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. None of them are cleanly licensed for commercial model training — which is the whole reason custom capture exists.
| Dataset | Size / scale | Sensors | License | Commercial use | Gap |
|---|---|---|---|---|---|
| EgoDex | 829 h · 25 joints per hand · paired at capture time | Apple Vision Pro RGB + native hand-tracking pose | Non-commercial, no-derivatives | No | The largest joint-labeled egocentric corpus, but no-derivatives means you cannot ship a model trained on it or redistribute anything built from it. |
| HOT3D | Egocentric clips with optical-mocap 3D hand + object pose | Project Aria + Quest 3, mocap-grade 3D ground truth | Non-commercial research | No | Gold-standard pose accuracy, but lab-staged tabletop tasks and a research-only license — a benchmark, not a training set you can ship. |
| Assembly101 | Procedural (dis)assembly, ego + exo views, 3D hand poses | Head-mounted RGB + multi-view static cameras | Non-commercial research | No | 3D hands on realistic procedures, but toy assemblies and non-commercial terms; no consent chain for commercial redistribution. |
| Nymeria | Large in-the-wild egocentric + full-body motion capture | Aria RGB/SLAM + body mocap suit | Non-commercial research | No | Full-body and coarse hand motion in natural settings, but motion GT — not the fine finger articulation a dexterous policy needs — and NC-only. |
Open datasets vs Truelabel custom capture
The annotation gold standard is legally untouchable. Every corpus with production-grade 3D hand pose — EgoDex, HOT3D, Assembly101 — ships under non-commercial or no-derivatives terms, so you can benchmark on them but never train a model you intend to ship.
Custom capture puts the joint convention under your control. You specify a 21-point MANO or 25-joint skeleton, left/right handedness, and per-frame confidence, instead of remapping whatever an academic release happened to publish.
Consent and provenance are auditable per clip. Fine-motor capture drags faces, screens, and documents into frame; a signed consent chain and per-clip PII handling is something a scraped or research corpus cannot hand your legal team.
Exclusivity and freshness. Open pose corpora are shared with every competitor and frozen at publication; commissioned capture is exclusive to you and targeted at your object set, grasp types, and operator demographics.
Hand tracking & pose: by the numbers
The figures below are specific to hand tracking & pose egocentric data and anchor the comparisons above.
- 25 joints per hand (EgoDex skeleton)
- 829 hours of paired hand-pose video (EgoDex)
- 21-keypoint MANO hand model
- 60 fps capture floor for fine-motor articulation
How Truelabel captures hand tracking & pose data
Truelabel runs hand tracking & pose programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to 40–150 accepted hours per pilot, scaling to 1,000+ for pretraining, delivered as H.265 video + per-clip JSON keypoint tracks (MANO or 25-joint), Hugging Face-streamable, with per-clip consent artifacts. Go deeper via why annotated hand-pose corpora are non-commercial, what egocentric data is, hand-heavy assembly and tool-use capture, fine-motor food-prep hand work, and grasp-and-pick hand motion at scale.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
EgoScale reports that dexterous-manipulation performance scales with the volume and diversity of egocentric human hand-motion demonstrations.
arXiv ↩ - Physical AI with World Foundation Models | NVIDIA Cosmos
World/action foundation models use per-frame hand keypoints as the action channel that makes first-person video trainable.
NVIDIA ↩ - EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex pairs large-scale egocentric video with per-hand pose captured at recording time; dexterous manipulation improves with paired hand-pose demonstration volume.
arXiv ↩ - EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
EgoScale: dexterous manipulation scales with diverse egocentric human hand data.
arXiv - Project Go-Big: Internet-Scale Humanoid Pretraining and Direct Human-to-Robot Transfer
Figure's Project Go-Big pretrains humanoid manipulation on large-scale egocentric human video.
Figure - Humanoid data: 10 Things That Matter in AI Right Now | MIT Technology Review
Real human demonstration data — not simulation — is the binding constraint for humanoid manipulation.
MIT Technology Review - EgoDex: code and dataset release
EgoDex release: 25 joints per hand under a non-commercial, no-derivatives license.
Apple - HOT3D: egocentric hand and object tracking in 3D
HOT3D provides mocap-grade 3D hand and object pose ground truth under a non-commercial research license.
Meta Reality Labs - Assembly101: A Large-Scale Multi-View Video Dataset
Assembly101 provides egocentric and exocentric 3D hand poses under a non-commercial research license.
Assembly101 project - Nymeria: egocentric full-body motion dataset
Nymeria pairs Aria glasses with full-body motion capture in the wild under a non-commercial research license.
Meta / Project Aria
FAQ
What's the difference between hand tracking and hand pose estimation data?
Hand tracking follows the hand's position and motion across frames; hand pose estimation resolves the articulated skeleton — a 21-point MANO or 25-joint model per hand — at each frame. Most buyers need both: a stable track for temporal continuity and per-frame joints for the action label.
Why can't I just train on EgoDex or HOT3D?
Because their licenses forbid it. EgoDex is no-derivatives and non-commercial; HOT3D and Assembly101 are non-commercial research only. They are excellent for benchmarking and pretraining experiments, but shipping a commercial model trained on them exposes you to license claims — commercially-licensed hand-pose video effectively has to be commissioned.
Do you capture 3D pose, or just 2D keypoints?
Both. Monocular RGB yields 2D keypoints directly; for metric 3D we add paired IMU and, where the task justifies it, a second synchronized camera or a headset with native hand tracking, then deliver per-joint 3D with confidence values.
Which joint convention and label format do you deliver?
Your choice — 21-point MANO or a 25-joint skeleton, per-hand left/right labels, frame-aligned to the RGB and exported as per-clip JSON that streams from Hugging Face. We match the format your training pipeline already expects rather than forcing a remap.
How do you keep pose labels clean on fast finger motion?
Machine QA rejects clips where a hand leaves the frame, motion blur on the fingers crosses threshold, or lighting drops below floor. A device floor (iPhone 11+, Galaxy S21+, Pixel 6+) holds down rolling-shutter artifacts, and we capture fine-motor tasks at 60 fps so fast articulation isn't undersampled.
Do we get exclusive rights, or is the same footage sold to competitors?
Commissioned captures are exclusive to you by default. Unlike an open pose corpus that every lab downloads, your object set, grasp taxonomy, and demographic targeting are captured once, for you, with the license terms fixed up front.
Looking for hand tracking dataset?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request hand-tracking egocentric data