Sensor · Egocentric data
Egocentric RGB-D & Depth Datasets
An egocentric RGB-D dataset pairs first-person, head-mounted video with per-frame depth — either sensor depth from LiDAR/ToF or depth estimated by a monocular network — so spatial-intelligence and manipulation models can learn 3D geometry, not just pixels. The open egocentric-depth corpora (HOI4D, THU-READ, MECCANO, Aria Digital Twin) are all research-gated or non-commercial, and head-mounted wearables rarely carry a real depth sensor, so first-person depth is far scarcer than tabletop RGB-D — which makes iPhone-Pro-LiDAR custom capture on a collector network a genuine wedge.
Quick facts
- Resolution
- 1080p RGB @ 30fps with per-frame aligned depth maps (LiDAR/ToF), 16-bit metric
- Field of view
- ≥120° horizontal so both hands, the manipulated object, and the near-field scene stay inside the depth frustum
- Mount
- Head-mounted rig on a LiDAR/ToF-capable device (iPhone Pro / Pro Max class); never chest or handheld
- Sensors
- RGB, Depth (LiDAR/ToF, metric), IMU (head), Optional gaze to mark attention-to-object
- Labels
- Per-frame aligned depth maps (metric, meters); Point clouds and partial meshes; Object masks and 6-DoF pose where captured
- Volume
- 40–150 accepted hours per depth program
Key papers
Hard citations for the claims above. Each entry pairs a specific number with the paper that reports it.
Real Time Egocentric Object Segmentation: THU-READ Labeling and Benchmarking Results
2,124 RGB-D images. This work contributes pixel-wise semantic object annotations for 2,124 images from the helmet-mounted RGB-D THU-READ egocentric dataset — one of the few head-mounted RGB-D benchmarks for egocentric object segmentation.
HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World
166 hours, 350 pairs. HoloAssist is a large-scale egocentric human-interaction dataset — 166 hours captured by 350 unique instructor-performer pairs on a mixed-reality headset with seven synchronized data streams — built for interactive AI assistants that guide people through real-world physical tasks.
EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
829 hours, 194 tasks. EgoDex pairs 829 hours of egocentric video across 194 tabletop tasks with 3D hand and finger tracking captured on Apple Vision Pro — the largest and most diverse dexterous human-manipulation dataset to date.
What depth & RGB-D data contains
An egocentric RGB-D dataset pairs first-person, head-mounted video with per-frame depth — either sensor depth from LiDAR/ToF or depth estimated by a monocular network — so spatial-intelligence and manipulation models can learn 3D geometry, not just pixels. The open egocentric-depth corpora (HOI4D, THU-READ, MECCANO, Aria Digital Twin) are all research-gated or non-commercial, and head-mounted wearables rarely carry a real depth sensor, so first-person depth is far scarcer than tabletop RGB-D — which makes iPhone-Pro-LiDAR custom capture on a collector network a genuine wedge.
The capture settings this covers:
- Close-range manipulation where the depth map resolves finger-to-object distance at the exact frame of contact, not just the RGB silhouette.
- Room-scale traversal through a cluttered home or workshop, so depth captures full scene geometry rather than only the near field in front of the hands.
- An object held and slowly rotated in-hand, where per-frame depth turns a flat RGB view into a 6-DoF object pose you can actually supervise.
- Reaching into partially-occluded shelves or bins, the case where monocular RGB alone cannot tell which of two overlapping objects is nearer.
- Reflective, transparent, or thin surfaces — glassware, wiring, bare metal tools — the exact geometry where LiDAR/ToF and estimated depth openly disagree.
- Low-light and backlit scenes where active sensor depth still returns usable geometry after the RGB frame has lost it.
Why robotics and AI labs need depth & RGB-D data
EgoScale showed dexterous-manipulation policy performance scaling log-linearly with the volume and diversity of egocentric human data — and contact-rich manipulation is precisely where the third dimension matters, because a policy has to know how far the fingers are from the object before it closes them. [1]
NVIDIA Cosmos positions world foundation models as a backbone for physical AI, and per-frame depth aligned to egocentric RGB is the geometric grounding those models consume. [2]
Apple built EgoDex at large scale specifically to learn dexterous manipulation from first-person video, and the appetite it signals — thousands of hours of hand-object geometry — is exactly the appetite that pulls buyers toward paired RGB-D rather than RGB alone. [3]
Capture and delivery spec
Every depth & RGB-D capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.
| Spec | Detail |
|---|---|
| Resolution | 1080p RGB @ 30fps with per-frame aligned depth maps (LiDAR/ToF), 16-bit metric |
| Frame rate | 30fps baseline; 60fps for fast reaches and re-grasps |
| Field of view | ≥120° horizontal so both hands, the manipulated object, and the near-field scene stay inside the depth frustum |
| Mount | Head-mounted rig on a LiDAR/ToF-capable device (iPhone Pro / Pro Max class); never chest or handheld |
| Sensors | RGB, Depth (LiDAR/ToF, metric), IMU (head), Optional gaze to mark attention-to-object |
| Labels | Per-frame aligned depth maps (metric, meters); Point clouds and partial meshes; Object masks and 6-DoF pose where captured; Reach / grasp / manipulate / release action segments |
| QA gates | Depth–RGB alignment within a pixel threshold; Depth completeness / hole-fraction check; Metric-scale sanity against a known reference object; No motion blur at the grasp and release moments; Per-clip consent artifact attached |
| Delivery | H.265 RGB + aligned 16-bit depth (PNG/EXR) + point-cloud JSON, Hugging Face-streamable, with a consent artifact attached to every clip |
| Volume | 40–150 accepted hours per depth program |
Open depth & RGB-D datasets
The 5 open corpora most relevant to depth & RGB-D are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. None of them are cleanly licensed for commercial model training — which is the whole reason custom capture exists.
| Dataset | Size / scale | Sensors | License | Commercial use | Gap |
|---|---|---|---|---|---|
| HOI4D | Egocentric RGB-D hand-object interaction with category-level 3D object and part annotations | RGB-D + 3D object/part annotations | Research-gated access | No | The richest egocentric RGB-D annotations in the set, but access is gated and the object taxonomy is fixed to the paper's categories, not your SKUs. |
| THU-READ | Helmet-mounted RGB-D egocentric daily actions (40 action classes) | Helmet-mounted RGB-D + hand/object annotation | Non-commercial research | No | One of the few helmet-mounted RGB-D action corpora, but small, lab-scripted, and non-commercial. |
| Aria Digital Twin | Indoor egocentric video with photorealistic digital-twin ground-truth depth and object/scene poses | RGB + digital-twin GT depth and poses | Non-commercial research license | No | Depth here is digital-twin ground truth, not a live sensor stream — excellent for benchmarking, unusable for a shipping product under the NC terms. |
| MECCANO | Multimodal egocentric build of a toy motorbike (RGB + depth + gaze) | RGB + depth + gaze | Research-only | No | One of the rare egocentric corpora with real synchronized depth, but a single toy-assembly domain and no commercial-use path. |
| HoloAssist | Two-person interactive assembly/repair tasks captured on HoloLens 2, with depth streams | Egocentric RGBD + head/hand and eye signals | CDLA-Permissive-style research license (verify terms) | Conditional | The most permissive license in the set, but the tasks are staged instructor-follower repairs, not autonomous single-operator manipulation. |
Open datasets vs Truelabel custom capture
Every open egocentric-depth corpus is off-limits commercially. HOI4D is research-gated, THU-READ is non-commercial research, MECCANO is research-only, and Aria Digital Twin's depth is a non-commercial digital-twin ground truth. HoloAssist is the one permissive-ish exception, and even that ships staged instructor-follower tasks rather than free-form manipulation. You can benchmark on all of them; you cannot ship a policy trained on the depth.
Egocentric depth is genuinely scarce next to tabletop RGB-D, and the reason is hardware. Static RealSense and Kinect rigs made desk-height RGB-D cheap — a RealSense D455 resolves depth across roughly a 0.6–6 m range from a fixed tripod — but head-mounted wearables mostly carry no metric depth sensor at all. Project Aria's stack, for instance, is SLAM and mono cameras plus IMU, with no dedicated depth camera, which is why first-person depth stays rare while tabletop depth is everywhere.
Estimated depth has a real ceiling. Monocular networks like Depth Anything V2 have gotten strong at relative depth, but they don't recover trustworthy metric scale from a single moving head-cam — and metric scale is exactly what a grasp policy or a spatial-intelligence model has to have. iPhone-Pro-class LiDAR gives you true sensor depth at collector scale, so you get metric geometry rather than a scale-ambiguous guess.
Custom capture lets you fix the depth format, the RGB-to-depth alignment tolerance, and the object set, and it ships a per-clip consent chain with the option of exclusivity. None of the research corpora offer that: you inherit their sensor, their scene, their license, and their taxonomy, and the same footage may already be in a competitor's training set.
Depth & RGB-D: by the numbers
The figures below are specific to depth & RGB-D egocentric data and anchor the comparisons above.
- MECCANO delivers synchronized RGB + depth + gaze across a single industrial-like assembly — one of the few egocentric corpora carrying real, not estimated, depth.
- HOI4D pairs egocentric RGB-D frames with category-level 3D object and part annotations, but distribution is research-gated.
- Aria Digital Twin's depth is photorealistic digital-twin ground truth under a non-commercial license — benchmark-only, never shippable.
- A fixed RealSense D455 resolves depth across roughly a 0.6–6 m range; head-mounted wearables like Project Aria carry no metric depth sensor at all, which is why egocentric depth stays rare.
How Truelabel captures depth & RGB-D data
Truelabel runs depth & RGB-D programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to 40–150 accepted hours per depth program, delivered as H.265 RGB + aligned 16-bit depth (PNG/EXR) + point-cloud JSON, Hugging Face-streamable, with a consent artifact attached to every clip. Go deeper via what RGB-D data is, depth data, point clouds, egocentric data, egocentric data licensing, and how to annotate 3D point clouds.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
EgoScale reports dexterous-manipulation performance scaling with the volume and diversity of egocentric human data.
arXiv ↩ - Physical AI with World Foundation Models | NVIDIA Cosmos
NVIDIA Cosmos positions world foundation models as a backbone for physical AI, grounded in real-world video.
NVIDIA ↩ - EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex is a large-scale egocentric video corpus built to learn dexterous hand-object manipulation.
arXiv ↩ - HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction
HOI4D is an egocentric RGB-D hand-object interaction dataset with category-level 3D annotations and gated access.
hoi4d.github.io - Real Time Egocentric Object Segmentation: THU-READ Labeling and Benchmarking Results
THU-READ is a helmet-mounted RGB-D egocentric action dataset released for non-commercial research.
arXiv - Aria Digital Twin (ADT)
Aria Digital Twin provides indoor egocentric video with photorealistic digital-twin ground-truth depth and poses under a non-commercial license.
Meta / Project Aria - MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain
MECCANO is a multimodal (RGB, depth, gaze) research-only egocentric dataset of a toy-motorbike assembly.
University of Catania (IPLAB) - HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World
HoloAssist captures two-person interactive assembly/repair tasks on HoloLens 2 with depth streams under a CDLA-Permissive-style research license.
arXiv - Intel RealSense Depth Camera D455
The Intel RealSense D455 is a fixed stereo depth camera with a documented usable depth range of roughly 0.6–6 m.
Intel RealSense - Project Aria Hardware Specifications
Project Aria's sensor suite is SLAM and RGB cameras plus IMU, with no dedicated metric depth camera.
Meta / Project Aria
FAQ
What is an egocentric RGB-D dataset?
It is first-person, head-mounted video where each frame is paired with a depth channel, so a model sees both color and 3D geometry from the wearer's point of view. The depth is either sensor depth (LiDAR or time-of-flight) or depth estimated by a monocular network, and the useful labels usually include aligned depth maps, point clouds, object masks, and 6-DoF object pose. It is what spatial-intelligence and manipulation models learn 3D structure from, rather than inferring it from RGB alone.
Sensor depth or estimated depth — LiDAR/ToF vs monocular nets?
Both are legitimate; they answer different needs. Estimated depth from a network like Depth Anything V2 is cheap and works on any RGB clip, but it gives you relative, scale-ambiguous depth from a moving head-cam. Sensor depth from LiDAR/ToF gives you metric geometry in meters, which is what a grasp or navigation policy actually consumes. For contact-rich manipulation we default to true sensor depth and can add a monocular-depth pass as a comparison layer.
What device floor supports true depth capture at collector scale?
iPhone Pro and Pro Max class devices carry a rear LiDAR scanner, and they are common enough on the collector network to run metric-depth programs at volume — that is the practical floor for sensor-grade egocentric depth. Below that floor you are restricted to estimated depth, which we flag explicitly in delivery so you never confuse a monocular guess with a metric reading.
What depth formats do you ship?
Per-frame aligned depth maps as 16-bit metric PNG or EXR, point clouds and partial meshes as JSON, and object masks with 6-DoF pose where the task calls for it, all time-synced to the H.265 RGB and streamable from Hugging Face. If your pipeline expects a specific convention — a particular depth encoding, alignment tolerance, or point-cloud schema — we deliver to it instead of handing you someone else's label spec.
Why is egocentric depth scarcer than tabletop RGB-D?
Two reasons. First, hardware: fixed RealSense/Kinect rigs made desk-height RGB-D trivial, while head-mounted wearables usually lack a metric depth sensor entirely — Aria, for example, ships SLAM and mono cameras with no depth camera. Second, licensing: the egocentric-depth corpora that do exist — HOI4D, THU-READ, MECCANO — are research-gated or non-commercial, so even the scarce data is legally hard to ship.
Can I train a commercial model on HOI4D, THU-READ, or MECCANO?
No. HOI4D is access-gated, THU-READ is a non-commercial research release, and MECCANO is research-only. They are ideal for prototyping and for reporting benchmark numbers, but a shipping product needs depth footage you actually hold commercial rights to — which is the whole reason to capture your own.
Looking for egocentric RGB-D dataset?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request egocentric RGB-D data