Task · Egocentric data
Gaze & Eye Tracking Egocentric Datasets
An eye tracking dataset pairs first-person video with per-frame gaze — the 2D point and 3D vector where the wearer is looking, plus fixation and saccade segmentation. Every gaze-complete open corpus today ships under an Aria, HoloLens, or eye-tracker research license that forbids commercial training, so commercially-usable gaze data effectively requires custom capture.
Quick facts
- Resolution
- 1080p RGB baseline; stereo 4K on Aria-Gen-2-class head rigs
- Field of view
- ≥110° horizontal RGB, with eye-tracking covering the central foveal field
- Mount
- Head-mounted glasses or head rig — gaze needs a fixed eye-camera geometry, never chest-mount or handheld
- Sensors
- Forward-facing head RGB camera, Binocular eye-tracking cameras, IMU for head pose, Optional depth for 3D gaze-target estimation
- Labels
- Per-frame 2D gaze point in image coordinates; 3D gaze vector plus estimated fixation target; Fixation / saccade segmentation
- Volume
- 40–150 accepted gaze-tracked hours per pilot
Key papers
Hard citations for the claims above. Each entry pairs a specific number with the paper that reports it.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
1,286 hours, 740 participants. Ego-Exo4D pairs simultaneously-captured egocentric and exocentric video of skilled activity from 740 participants across 13 cities — 1,286 hours with multichannel audio, eye gaze, 3D point clouds, camera poses, and IMU.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
3,670 hours, 74 locations. Ego4D spans 3,670 hours of daily-life first-person video from 931 camera wearers across 74 locations in 9 countries, collected under consenting-participant privacy and de-identification standards.
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100
90K action segments on 100 hours. EPIC-KITCHENS-100 densely annotates 100 hours (20M frames) with roughly 90,000 action segments across 45 kitchens — the labeled-hour density that raw first-person corpora cannot match.
What gaze & eye tracking data contains
An eye tracking dataset pairs first-person video with per-frame gaze — the 2D point and 3D vector where the wearer is looking, plus fixation and saccade segmentation. Every gaze-complete open corpus today ships under an Aria, HoloLens, or eye-tracker research license that forbids commercial training, so commercially-usable gaze data effectively requires custom capture.
The capture settings this covers:
- Assembly and repair work where a fixation lands on the next part a half-second before the hand reaches for it
- Kitchen and cooking sequences in which gaze leads the hands from ingredient to tool to hob
- Retail and warehouse picking, where the eyes lock onto a SKU on the shelf before any reach begins
- Free-viewing navigation through cluttered rooms and aisles, showing how attention is rationed across a scene
- Fine tool use — soldering, wiring, surgical-style tasks — where gaze anchors on the exact contact point
Why robotics and AI labs need gaze & eye tracking data
Labs training humanoid manipulation are mining human first-person video for intent, and gaze is the cleanest intent signal there is — where a person looks predicts what they grasp next. Apple's EgoDex shows human egocentric data transferring into robot manipulation policies, which puts a premium on attention-labeled POV footage. [1]
World- and action-model platforms scale with the volume and richness of POV video; NVIDIA's Cosmos work frames large egocentric corpora as core substrate, and a gaze channel tells a world model which pixels the human actually cared about. [2]
The AoE line of work names real-world data the bottleneck holding humanoids back. Gaze-tracked corpora are among the scarcest and most expensive of that supply, because they need calibrated eye-tracking hardware on every wearer — which is exactly why almost none of it is commercially licensed. [3]
Capture and delivery spec
Every gaze & eye tracking capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.
| Spec | Detail |
|---|---|
| Resolution | 1080p RGB baseline; stereo 4K on Aria-Gen-2-class head rigs |
| Frame rate | 30 fps video, with per-frame gaze aligned to it; native eye-tracker sampling 30–120 Hz depending on hardware |
| Field of view | ≥110° horizontal RGB, with eye-tracking covering the central foveal field |
| Mount | Head-mounted glasses or head rig — gaze needs a fixed eye-camera geometry, never chest-mount or handheld |
| Sensors | Forward-facing head RGB camera, Binocular eye-tracking cameras, IMU for head pose, Optional depth for 3D gaze-target estimation |
| Labels | Per-frame 2D gaze point in image coordinates; 3D gaze vector plus estimated fixation target; Fixation / saccade segmentation; Gaze-to-object attention labels; Frame-aligned action segments |
| QA gates | Per-wearer eye-tracker calibration and end-of-session drift check; Gaze-in-frame validity rate above threshold; Hands-and-target-in-frame; Head stability and motion-blur floor |
| Delivery | H.265 video plus per-clip JSON gaze and fixation streams, timestamp-aligned and HF-streamable, with a consent artifact per clip |
| Volume | 40–150 accepted gaze-tracked hours per pilot |
Open gaze & eye tracking datasets
The 5 open corpora most relevant to gaze & eye tracking are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. None of them are cleanly licensed for commercial model training — which is the whole reason custom capture exists.
| Dataset | Size / scale | Sensors | License | Commercial use | Gap |
|---|---|---|---|---|---|
| EGTEA Gaze+ | ~28 h · 32 subjects · 86 cooking sessions | Head RGB + synchronized eye-tracker gaze | Research-only | No | Kitchen-only; research license blocks commercial training; dated SD-era framing |
| HOT3D | ~833 min · 19 subjects · Aria + Quest 3 | Aria eye-gaze + mocap-grade 3D hand/object pose | Non-commercial research (Meta) | No | Tabletop hand-object only; no natural free-viewing gaze; NC blocks product training |
| Aria Everyday Activities (AEA) | ~7.3 h · Project Aria glasses | Aria RGB + eye-tracking + SLAM | Non-commercial research (Project Aria) | No | Tiny corpus; no task-action gaze labels; NC license |
| Nymeria | ~300 h · in-the-wild · Aria + full-body mocap | Aria gaze + full-body motion capture | Non-commercial research (Project Aria) | No | Motion-centric; gaze is secondary to body pose; NC license |
| MECCANO | Toy-motorbike assembly proxy · small | RGB + depth + gaze | Research-only | No | Toy proxy, not a real facility; small; research-only terms |
Open datasets vs Truelabel custom capture
License reality is the whole story here. EGTEA Gaze+ is research-only; HOT3D, Aria Everyday Activities, and Nymeria all ship under non-commercial Meta / Project Aria research licenses [4]. You can benchmark on them, but you cannot legally train a product you sell. Custom capture is the only path to gaze data with clean commercial rights.
Open gaze data is frozen to one hardware calibration and one activity domain — EGTEA is kitchen-only, MECCANO is a toy motorbike proxy. Custom capture lets you fix the eye-tracker calibration protocol, target your real activity and object set, and choose the gaze label format — 2D point, 3D vector, or fixation target — that your model actually consumes.
Gaze plus eye and face video is biometric-adjacent, so a documented consent chain and bystander handling isn't optional for commercial deployment. Life-logging corpora like EgoLife openly carry privacy exposure on their dataset cards [5]; a custom program bakes per-clip consent artifacts and PII handling into delivery egocentric data licensing.
The same handful of non-commercial corpora sit in every competitor's pretraining mix. Exclusive, freshly-captured gaze data — tied to your taxonomy and refreshed as your product evolves — is something no open download can give you.
Gaze & eye tracking: by the numbers
The figures below are specific to gaze & eye tracking egocentric data and anchor the comparisons above.
- EGTEA Gaze+: ~28 hours of gaze-tracked cooking across 32 subjects
- HOT3D: ~833 minutes of Project Aria eye-gaze with mocap-grade hand/object pose
- Aria Everyday Activities: 7.3 hours of eye-tracked everyday recordings
- Nymeria: ~300 hours pairing Aria gaze with in-the-wild full-body motion capture
- Zero commercially-licensed egocentric gaze corpora existed as of July 2026
- Five leading open gaze corpora — EGTEA Gaze+, HOT3D, AEA, Nymeria, MECCANO — are all research-only or non-commercial
How Truelabel captures gaze & eye tracking data
Truelabel runs gaze & eye tracking programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to 40–150 accepted gaze-tracked hours per pilot, delivered as H.265 video plus per-clip JSON gaze and fixation streams, timestamp-aligned and HF-streamable, with a consent artifact per clip. Go deeper via egocentric data licensing, what egocentric data is, egocentric kitchen capture, industrial first-person capture, and warehouse pick capture.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex shows human egocentric data transferring into robot manipulation policies, motivating attention-labeled first-person footage.
arXiv ↩ - Physical AI with World Foundation Models | NVIDIA Cosmos
NVIDIA's Cosmos world-foundation-model platform frames large egocentric/POV video corpora as core substrate for world and action models.
NVIDIA ↩ - AoE: Always-on Egocentric Human Video Collection for Embodied AI
AoE frames scalable, low-cost collection of egocentric human video as the answer to the real-world data bottleneck holding back humanoid robots.
arXiv ↩ - HOT3D: egocentric hand and object tracking in 3D
HOT3D provides Project Aria eye-gaze alongside mocap-grade 3D hand and object pose under a non-commercial research license.
Meta Reality Labs ↩ - EgoLife
EgoLife is a multi-day Aria life-logging capture whose dataset card carries notable privacy and consent exposure.
EgoLife project ↩ - Extended GTEA Gaze+ (EGTEA Gaze+)
EGTEA Gaze+ is an egocentric cooking dataset with synchronized gaze tracking and fine-grained action labels, distributed under a research license.
Georgia Tech (First Person Vision) - Aria Everyday Activities (AEA)
Aria Everyday Activities is ~7.3 hours of eye-tracked everyday egocentric recordings from Project Aria glasses under a non-commercial research license.
Meta / Project Aria - Nymeria: egocentric full-body motion dataset
Nymeria pairs Project Aria gaze with in-the-wild full-body motion capture under a non-commercial research license.
Meta / Project Aria - MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain
MECCANO is a multimodal (RGB, depth, gaze) egocentric dataset of a toy motorbike assembly, released research-only.
University of Catania (IPLAB) - AoE: Always-on Egocentric Human Video Collection for Embodied AI
AoE (CVPR 2026 Workshops) collects egocentric human video at scale for embodied AI, an attention-sensitive setting for gaze data.
arXiv
FAQ
Why not just use EGTEA Gaze+ or HOT3D gaze for free?
You can, for research and benchmarking — but not for a commercial model. EGTEA Gaze+ is a research-license cooking corpus, and HOT3D's mocap-grade gaze ships under a non-commercial Meta license. Training a product you sell on either is outside their terms. That legal wall, not data quality, is why commercial teams commission gaze capture.
Do you capture gaze alongside RGB, hand pose, and IMU?
Yes. The head rig carries a forward RGB camera plus binocular eye-tracking cameras and an IMU, so every frame of video has a time-aligned 2D gaze point and 3D gaze vector, and we can co-register hand pose and head pose. That multimodal alignment is what makes gaze useful for attention and intent models — bare gaze, without the scene it points at, trains nothing.
How accurate is the gaze signal, and how do you keep it that way?
Every session starts with a per-wearer eye-tracker calibration and ends with a drift check; clips that fail calibration or fall below a gaze-in-frame validity threshold are rejected before delivery. We also gate on head stability and motion blur, because a gaze point is only as good as the frame it sits on. QA numbers ship with the batch.
What does gaze data actually train?
Gaze-conditioned attention and foveation, next-action and intent prediction, and attention signals for imitation and world models — recent work such as AoE learns robot actions from egocentric human video where attention matters. If you're mapping how egocentric data feeds these models, our egocentric data primer covers the taxonomy.
Can we license your existing catalog, or is everything custom capture?
For gaze specifically it's custom capture by necessity — there is no rights-cleared, commercially-licensable egocentric gaze catalog to license, ours or anyone's. We scope a pilot to your activity, object set, and gaze label format, then scale the accepted-hours band from there.
How is consent handled given eye and face video are sensitive?
Each wearer signs a consent artifact covering gaze, eye, and face capture and downstream commercial use, and bystanders are handled under a documented policy with PII blurring where required. This is the gap open life-logging data leaves wide open — EgoLife, for instance, flags privacy exposure on its own card.
Looking for eye tracking dataset?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request gaze egocentric data