Industry · Egocentric data
Egocentric Video Data for Smart Glasses & AR Assistants
A smart glasses dataset is first-person, head-mounted video — plus the IMU, eye-gaze, and audio streams a real pair of glasses carries — matched to the sensor stack an AR assistant actually ships on. The open glasses corpora are almost entirely non-commercial Aria research data, so consented custom capture (on real glasses hardware or phone-on-head rigs) is the practical route to commercially-usable AR assistant training data.
Quick facts
- Resolution
- 1080p @ 30fps baseline; higher-res stills for text/label reads
- Field of view
- ≥110° horizontal to match a glasses form factor, not a chest GoPro
- Mount
- Smart-glasses hardware where available, or a calibrated phone-on-head rig that mimics eye-line and FOV
- Sensors
- RGB (baseline), IMU (head), Eye-gaze (optional, the differentiator for attention modelling), Audio (optional, for assistant dialog and translation)
- Labels
- Assist-intent and query segments (when the wearer would ask the assistant for help); Gaze-target annotations (which object or text the eyes fixate); Task-step boundaries for hands-free guidance flows
- Volume
- 80–300 accepted hours per AR-assistant program
Key papers
Hard citations for the claims above. Each entry pairs a specific number with the paper that reports it.
HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World
166 hours, 350 pairs. HoloAssist is a large-scale egocentric human-interaction dataset — 166 hours captured by 350 unique instructor-performer pairs on a mixed-reality headset with seven synchronized data streams — built for interactive AI assistants that guide people through real-world physical tasks.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
1,286 hours, 740 participants. Ego-Exo4D pairs simultaneously-captured egocentric and exocentric video of skilled activity from 740 participants across 13 cities — 1,286 hours with multichannel audio, eye gaze, 3D point clouds, camera poses, and IMU.
EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
829 hours, 194 tasks. EgoDex pairs 829 hours of egocentric video across 194 tabletop tasks with 3D hand and finger tracking captured on Apple Vision Pro — the largest and most diverse dexterous human-manipulation dataset to date.
What smart glasses needs from egocentric data
A smart glasses dataset is first-person, head-mounted video — plus the IMU, eye-gaze, and audio streams a real pair of glasses carries — matched to the sensor stack an AR assistant actually ships on. The open glasses corpora are almost entirely non-commercial Aria research data, so consented custom capture (on real glasses hardware or phone-on-head rigs) is the practical route to commercially-usable AR assistant training data.
The capture settings this covers:
- Wayfinding through a transit hub — the wearer scans signage and platform numbers, and the assistant has to know what the eyes actually landed on, not just what was in frame.
- Hands-free step guidance: following a repair or recipe while glancing between the manual and the object, so the model can track which step the wearer is really on.
- Everyday errands in a store aisle — reading labels, price tags, and shelf edges with bystanders drifting through the background of an always-on stream.
- Contextual recall moments: looking at a business card, a whiteboard, or a parking-level sign, the raw material for 'where did I leave my keys' memory queries.
- A face-to-face conversation the assistant captions or translates live, where non-consenting third parties sit squarely in the wearer's view.
- The hard cases glasses cameras hit in the wild — walking outdoors at dusk, fast head turns, glare, and the rolling-shutter smear a lab capture never sees.
Why smart glasses needs first-person human video
Apple's EgoDex pretrains dexterous manipulation on large-scale egocentric human video, the kind of first-person hand-and-object footage smart-glasses assistants must understand to be useful. [1]
EgoScale shows dexterous-manipulation performance scaling with the volume and diversity of egocentric human data, the same scaling argument behind capturing a large glasses-worn egocentric corpus. [2]
AoE frames scalable, low-cost collection of egocentric human video of everyday tasks as an answer to the data scarcity a smart-glasses assistant corpus is built to close. [3]
Capture and delivery spec
Every smart glasses capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.
| Spec | Detail |
|---|---|
| Resolution | 1080p @ 30fps baseline; higher-res stills for text/label reads |
| Frame rate | 30fps baseline; 60fps for fast head motion and outdoor walking |
| Field of view | ≥110° horizontal to match a glasses form factor, not a chest GoPro |
| Mount | Smart-glasses hardware where available, or a calibrated phone-on-head rig that mimics eye-line and FOV |
| Sensors | RGB (baseline), IMU (head), Eye-gaze (optional, the differentiator for attention modelling), Audio (optional, for assistant dialog and translation) |
| Labels | Assist-intent and query segments (when the wearer would ask the assistant for help); Gaze-target annotations (which object or text the eyes fixate); Task-step boundaries for hands-free guidance flows; Bystander / face regions flagged for privacy handling |
| QA gates | Target or text-of-interest in frame above threshold; Bystander-privacy handling verified on every always-on clip; FOV and horizon check (eye-line, not chest-line); Per-clip consent artifact attached |
| Delivery | H.265 clips + per-clip JSON metadata (intent, gaze targets, task steps), Hugging Face-streamable, with a consent artifact on every clip |
| Volume | 80–300 accepted hours per AR-assistant program |
Open smart glasses datasets
The 5 open corpora most relevant to smart glasses are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. None of them are cleanly licensed for commercial model training — which is the whole reason custom capture exists.
| Dataset | Size / scale | Sensors | License | Commercial use | Gap |
|---|---|---|---|---|---|
| Aria Everyday Activities (AEA) | ~7.3 h of everyday-activity recordings from Project Aria glasses | RGB + IMU + eye-gaze + audio | Aria research license (non-commercial) | No | The closest thing to an open 'everyday glasses' corpus, and it is barely 7 hours — smaller than a single pilot batch, and non-commercial on top of that. |
| Aria Digital Twin (ADT) | Indoor glasses capture with photorealistic digital-twin object/scene ground truth | RGB + IMU + gaze + GT object/scene poses | Aria research license (non-commercial) | No | Beautiful ground truth for benchmarking spatial models, but staged instrumented rooms — not your users' kitchens, offices, or streets — and no commercial-use path. |
| Ego-Exo4D | ~1,286 h across 740 participants in 13 cities, paired Aria + exocentric views | Aria RGB + IMU + gaze + audio, plus external cameras | Ego-Exo4D signed license | Conditional | By far the largest Aria glasses capture, but the signed license and its fixed skilled-activity task set are a poor fit for an always-on consumer assistant. |
| HoloAssist | ~166 h of two-person interactive assistant guidance on HoloLens 2 | RGB + eye/hand tracking + dialog | CDLA-Permissive-style research license (verify terms) | Conditional | The one near-permissive corpus in the glasses set, but the tasks are staged instructor-follower repairs, not the messy always-on stream a shipping assistant sees. |
| EgoLife | Multi-day, multi-person glasses life-logging behind the EgoGPT/EgoRAG assistants | RGB + audio | Dataset card claims MIT; privacy-exposed (verify) | Conditional | The most assistant-shaped open corpus, and also the most consent-fraught — days of a household's private life with a permissive card but no defensible consent chain. |
Open datasets vs Truelabel custom capture
The open glasses corpora are almost all non-commercial Aria research data. Aria Everyday Activities is roughly 7.3 hours under a research license, Aria Digital Twin is a staged, instrumented indoor set under the same terms, and Ego-Exo4D — the biggest of them — ships under a signed license. You can benchmark on all three; you cannot legally train a shipping assistant on any of them.
HoloAssist is the lone near-permissive exception, and even it is staged instructor-follower repairs rather than the always-on stream a consumer assistant actually sees. Custom capture lets you specify the real use cases — errands, wayfinding, hands-free guidance — instead of borrowing someone else's task list.
Research glasses data is captured on hardware you don't ship, with a consent posture built for a lab, not a consumer product. Aria Gen 2 carries eye tracking, a PPG sensor, and a contact mic; your glasses carry a different set. Custom capture matches the exact RGB, IMU, gaze, and audio configuration your device records so the training distribution isn't off from day one.
Bystander privacy is the product risk on always-on glasses, not a footnote. EgoLife's own card is permissive but the footage is days of private household life with no documented consent chain. Custom capture bakes in face handling and a per-clip consent artifact, so the same footage that trains your assistant also survives a privacy review.
Smart glasses: by the numbers
The figures below are specific to smart glasses egocentric data and anchor the comparisons above.
- Aria Everyday Activities is roughly 7.3 hours — the entire open 'everyday glasses' corpus is smaller than a single pilot batch
- HoloAssist's ~166 hours are the only near-permissive (CDLA-style) footage in the glasses set
- Aria Gen 2 adds eye tracking, a PPG heart-rate sensor, and a contact microphone on top of the RGB + IMU baseline
- "augmented reality dataset" draws about 10 searches/mo and "smart glasses dataset" returns an academic-only SERP (DataForSEO + Serper, 2026-07-06)
How Truelabel captures smart glasses data
Truelabel runs smart glasses programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to 80–300 accepted hours per AR-assistant program, delivered as H.265 clips + per-clip JSON metadata (intent, gaze targets, task steps), Hugging Face-streamable, with a consent artifact on every clip. Go deeper via what egocentric data is, egocentric data licensing, the per-clip consent artifact, provenance manifest, custom egocentric video sourcing, and the physical AI data marketplace.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex is a large-scale egocentric video corpus built to pretrain dexterous manipulation from first-person human video.
arXiv ↩ - EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
EgoScale supports the claim that dexterous-manipulation performance scales with the volume and diversity of egocentric human data.
arXiv ↩ - AoE: Always-on Egocentric Human Video Collection for Embodied AI
AoE frames scalable, low-cost collection of egocentric human video of manual tasks as an answer to embodied-AI data scarcity.
arXiv ↩ - Aria Everyday Activities (AEA)
Aria Everyday Activities is about 7.3 hours of everyday-activity egocentric recordings from Project Aria glasses under a non-commercial research license.
Meta / Project Aria - Aria Digital Twin (ADT)
Aria Digital Twin is an indoor egocentric dataset with photorealistic digital-twin ground truth under a non-commercial research license.
Meta / Project Aria - Ego-Exo4D project site
Ego-Exo4D pairs first-person Aria and third-person views of skilled activities and is distributed under a signed dataset license.
ego-exo4d-data.org - EgoLife
EgoLife is a multi-day, multi-person egocentric life-logging capture whose card claims MIT but carries unresolved privacy exposure.
EgoLife project - Project Aria Hardware Specifications
Project Aria hardware specifications document the glasses sensor suite (RGB, IMU, eye tracking, microphones) used as the smart-glasses capture reference.
Meta / Project Aria - Apple Vision Pro - Technical Specifications - Apple
Apple Vision Pro technical specs anchor the headset end of the smart-glasses capture-hardware spectrum.
apple.com - GoPro HERO13 Black
GoPro HERO13 Black is the reference head-mount action camera for phone-free egocentric capture rigs.
gopro.com
FAQ
What is a smart glasses dataset?
It is first-person video recorded from a head-mounted glasses viewpoint, ideally with the same side-channels a real pair of glasses records — IMU for head motion, eye-gaze for attention, and audio for dialog. The open reference points are the Project Aria corpora (Aria Everyday Activities, Aria Digital Twin) and Ego-Exo4D, but those are research-licensed, so a commercially-usable smart glasses dataset almost always means consented custom capture.
Can you capture on actual smart-glasses hardware, or only phone-on-head rigs?
Both, depending on what your program needs. Where you have glasses hardware to deploy — anything from a Project Aria-class sensor rig to a Vision Pro-style headset — we collect on-device so the sensor geometry matches your product exactly. Where you need scale fast, a calibrated phone-on-head or GoPro head-mount rig reproduces the eye-line and field of view closely enough to pretrain on, and it runs on the iPhone 11-and-up floor our collector network already carries, so you get thousands of hours without waiting on a hardware rollout.
What sensor set do AR assistant teams actually need?
For most assistant work the useful stack is RGB plus IMU plus eye-gaze plus audio. RGB and IMU are the baseline; gaze is the differentiator, because attention — what the wearer is looking at — is what turns a video feed into an assistant that answers the right question. Aria Gen 2's push into eye tracking and richer sensing is the reason we treat gaze as a first-class capture channel rather than an add-on.
How do you handle bystander privacy for always-on glasses footage?
Every clip carries a consent artifact tied to the wearer, faces and other bystander regions are flagged at QA for blurring or removal per your retention rules, and the whole chain is auditable. This is exactly the gap the open corpora leave open: EgoLife, for instance, is days of private household life under a permissive card with no documented consent posture, which is unusable the moment a privacy review looks at it.
Why not just wait for Aria Gen 2 open data?
Because when it arrives it will almost certainly land under the same non-commercial research license the entire Aria family already uses — Aria Everyday Activities, Aria Digital Twin, and Nymeria are all research-only. Great for a benchmark, useless for a product you intend to ship. Waiting also cedes the head start; the labs collecting their own glasses data now are the ones who will have shipping-grade models first.
Why not use Ego-Exo4D or HoloAssist for free?
You can, for research. Ego-Exo4D is ~1,286 hours but sits under a signed license and a fixed skilled-activity task set, and HoloAssist's ~166 hours are staged two-person repairs rather than an always-on assistant stream. Neither matches a consumer AR-assistant distribution, and neither gives you the commercial rights or the exclusivity to keep the footage away from a competitor.
Looking for smart glasses dataset?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request smart glasses data