Environment · Egocentric data
Egocentric Video Data for Driving
Egocentric driving data is head-mounted, driver-point-of-view video — what the person behind the wheel actually sees and looks at — not vehicle-mounted dashcam footage or a fixed in-cabin driver-monitoring (DMS) camera feed. No commercially-licensed head-mounted driver-POV corpus exists today, so labs that need it commission custom capture.
Quick facts
- Resolution
- 1080p baseline; 2160p stereo option where depth or gaze geometry matters
- Field of view
- ≥120° horizontal, head-mounted at the driver's eyeline — not a fixed cabin-mounted cone
- Mount
- Head-mounted glasses/cap rig on the driver — explicitly NOT a vehicle-mounted dashcam or a fixed in-cabin DMS camera
- Sensors
- RGB (mono or stereo), IMU (head motion), Eye-gaze stream (fixation/saccade), Optional depth, GPS + speed telemetry, timestamp-synced to video
- Labels
- Frame-aligned glance targets (road, mirror, cluster, phone, controls); Gaze fixation/saccade streams; Hands-on-wheel and hands-on-controls state
- Volume
- 40–200 accepted driver-hours per pilot; scalable across geographies and vehicle types
Key papers
Hard citations for the claims above. Each entry pairs a specific number with the paper that reports it.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
3,670 hours, 74 locations. Ego4D spans 3,670 hours of daily-life first-person video from 931 camera wearers across 74 locations in 9 countries, collected under consenting-participant privacy and de-identification standards.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
1,286 hours, 740 participants. Ego-Exo4D pairs simultaneously-captured egocentric and exocentric video of skilled activity from 740 participants across 13 cities — 1,286 hours with multichannel audio, eye gaze, 3D point clouds, camera poses, and IMU.
EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
829 hours, 194 tasks. EgoDex pairs 829 hours of egocentric video across 194 tabletop tasks with 3D hand and finger tracking captured on Apple Vision Pro — the largest and most diverse dexterous human-manipulation dataset to date.
What driving & in-cabin egocentric data captures
Egocentric driving data is head-mounted, driver-point-of-view video — what the person behind the wheel actually sees and looks at — not vehicle-mounted dashcam footage or a fixed in-cabin driver-monitoring (DMS) camera feed. No commercially-licensed head-mounted driver-POV corpus exists today, so labs that need it commission custom capture.
The capture settings this covers:
- Highway and dense-city driving shot from the driver's eyeline — hands moving between wheel, gear selector, indicator stalk, and infotainment while the road scene shifts.
- In-cabin secondary tasks that DMS teams care about: reaching for a phone, adjusting climate and audio controls, handling a coffee, glancing at mirrors and the instrument cluster.
- Ride-hail and last-mile delivery workflows: reading the dispatch app at the curb, pulling over, retrieving and handing off packages, then re-entering traffic.
- Pre-drive and ingress/egress routines: buckling, adjusting the seat and mirrors, starting the vehicle, and the mirror-check sequence before pulling out.
- Adverse conditions from the wearer's viewpoint — night driving, rain and wiper glare, low sun — where gaze and glance behavior change and cabin cameras struggle.
- Passenger and rear-seat POV for occupant-monitoring work: who is in the cabin, seatbelt state, and attention during hand-offs in semi-autonomous modes.
Why robotics and AI labs need driving & in-cabin data
World foundation models for physical AI are trained on large-scale point-of-view video, and NVIDIA's Cosmos platform explicitly names autonomous-vehicle development as a target — first-person driving footage feeds that pipeline directly. [1]
Apple's EgoDex showed that egocentric human video meaningfully improves skill transfer to robot policies, which extends the same argument to driver-POV: the wearer's viewpoint is a usable training substrate, not just a monitoring feed. [2]
The bottleneck for embodied and in-cabin AI is real-world first-person data, not model architecture — the AoE line of work frames scalable egocentric capture as the fix — and driver-POV with synced gaze is one of the scarcest slices of it. [3]
Capture and delivery spec
Every driving & in-cabin capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.
| Spec | Detail |
|---|---|
| Resolution | 1080p baseline; 2160p stereo option where depth or gaze geometry matters |
| Frame rate | 30 fps baseline; 60 fps for glance and saccade detection during fast head/eye motion |
| Field of view | ≥120° horizontal, head-mounted at the driver's eyeline — not a fixed cabin-mounted cone |
| Mount | Head-mounted glasses/cap rig on the driver — explicitly NOT a vehicle-mounted dashcam or a fixed in-cabin DMS camera |
| Sensors | RGB (mono or stereo), IMU (head motion), Eye-gaze stream (fixation/saccade), Optional depth, GPS + speed telemetry, timestamp-synced to video |
| Labels | Frame-aligned glance targets (road, mirror, cluster, phone, controls); Gaze fixation/saccade streams; Hands-on-wheel and hands-on-controls state; Secondary-task and distraction event segments; Object states (phone, cup, buttons) and optional drowsiness/attention tags |
| QA gates | Head-alignment and stability check (footage tracks the eyeline, not the cabin); ≥120° FOV verification; Hands-and-controls in frame; Face and license-plate PII redaction pass; Gaze/IMU-to-RGB timestamp sync verification |
| Delivery | H.265 clips + per-clip JSON (gaze, IMU, GPS/speed, glance labels), Hugging-Face-streamable, with a consent artifact attached to every clip |
| Volume | 40–200 accepted driver-hours per pilot; scalable across geographies and vehicle types |
Open driving & in-cabin datasets
The 4 open corpora most relevant to driving & in-cabin are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. Only 1 of the 4 is permissively licensed for commercial use — which is the whole reason custom capture exists.
| Dataset | Size / scale | Sensors | License | Commercial use | Gap |
|---|---|---|---|---|---|
| DMD (Driver Monitoring Dataset) | Multi-driver, multi-camera in-cabin recordings | Three synchronized driver-facing views (face, body, hands) — cabin-mounted, not head-worn | Research license | No | The closest thing to a driving-cabin corpus, but it is a fixed DMS rig filming the driver — not head-mounted driver-POV — and terms are research-only. |
| Aria Everyday Activities (AEA) | ~7.3 hours of head-mounted egocentric recording | Project Aria glasses (RGB + IMU + eye-gaze) | Non-commercial research license | No | Genuine head-mounted egocentric, but everyday-life activities with no driving focus, tiny volume, and a license that blocks any commercial training use. |
| Nymeria | Large egocentric + full-body motion-capture corpus, in the wild | Aria glasses paired with full-body mocap | Non-commercial research license | No | Head-mounted and outdoor, but built for full-body motion, with no in-cabin or driving scenarios, and non-commercial only. |
| Egocentric-10K | 10,000 hours · 2,138 workers · 87 factories | Head-mounted RGB | Apache-2.0 | Yes | The one large permissively-licensed head-mounted corpus — but 100% factory-floor manufacturing work, with zero in-cabin or driving footage and no labels. |
Open datasets vs Truelabel custom capture
Dashcam and DMS corpora are the wrong modality. A dashcam is vehicle-mounted and films the forward road scene (exocentric); a DMS camera is fixed and films the driver (third-person of the driver). Neither captures what the driver's eyes actually fix on. Head-mounted driver-POV with a synced gaze stream is a different signal, and open corpora effectively do not carry it.
Every candidate open corpus fails on license OR domain. DMD is research-only and cabin-mounted; the true head-mounted corpora (Aria Everyday Activities, Nymeria) are non-commercial and contain no driving; Egocentric-10K is Apache-2.0 but 100% factory floor. There is no commercially-licensed head-mounted driver-POV dataset to license off the shelf.
In-cabin consent is genuinely hard — driver, passengers, bystanders, faces, and plates all appear. Custom capture gives you a documented, per-clip consent record plus a PII redaction pass you can actually put in front of counsel; scraped or repurposed footage cannot.
You set the terms: vehicle types, geographies, the glance taxonomy, the sensor stack, and the edge cases (night, rain, glare) — and you can hold the footage exclusively. A frozen academic corpus gives you none of that.
Driving & in-cabin: by the numbers
The figures below are specific to driving & in-cabin egocentric data and anchor the comparisons above.
- DMD ships three synchronized driver-facing camera views (face, body, hands) — none of them head-mounted driver-POV
- Zero commercially-licensed head-mounted driver-POV hours exist in open corpora today
- 60 fps capture for in-cabin glance and saccade detection
- ≥120° driver-eyeline FOV versus a fixed cabin DMS camera's narrow driver-facing cone
- 40–200 accepted driver-hours per pilot batch, across vehicle types and geographies
How Truelabel captures driving & in-cabin data
Truelabel runs driving & in-cabin programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to 40–200 accepted driver-hours per pilot; scalable across geographies and vehicle types, delivered as H.265 clips + per-clip JSON (gaze, IMU, GPS/speed, glance labels), Hugging-Face-streamable, with a consent artifact attached to every clip. Go deeper via what egocentric data actually means, how we license first-person video, our industrial egocentric capture spec, warehouse first-person video sourcing, and kitchen first-person video sourcing.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- Physical AI with World Foundation Models | NVIDIA Cosmos
NVIDIA Cosmos world foundation models for physical AI are trained on large-scale POV/egocentric video and name autonomous-vehicle development as a target.
NVIDIA ↩ - EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex is a large-scale egocentric human video corpus that improves human-to-robot skill transfer for manipulation policies.
arXiv ↩ - AoE: Always-on Egocentric Human Video Collection for Embodied AI
AoE frames scalable, low-cost collection of egocentric first-person video as the answer to the real-world data bottleneck for embodied AI.
arXiv ↩ - DMD: Driver Monitoring Dataset
DMD is a multi-camera in-cabin driver-monitoring dataset (face/body/hands) released under a research license.
Vicomtech - Aria Everyday Activities (AEA)
Aria Everyday Activities is ~7.3 h of head-mounted egocentric recording under a non-commercial research license.
Meta / Project Aria - Nymeria: egocentric full-body motion dataset
Nymeria pairs Aria-glasses egocentric video with full-body motion capture under a non-commercial research license.
Meta / Project Aria - Egocentric-10K
Egocentric-10K is a large head-mounted egocentric corpus of factory-floor work.
Hugging Face - Egocentric-10K dataset card and license
Egocentric-10K is released under Apache-2.0 (10,000 h / 2,138 workers / 87 factories).
Hugging Face - EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
EgoScale shows dexterous-manipulation performance scaling with diverse egocentric human data.
arXiv - EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation
EgoVid-5M is a large video-action dataset built for egocentric video generation and world models.
arXiv
FAQ
Is dashcam footage the same as egocentric driving data?
No. A dashcam is vehicle-mounted and points at the road ahead — it is exocentric footage of the scene. Egocentric driving data is head-mounted on the driver and captures the driver's own viewpoint, including where their eyes go. If you are training driver attention, glance, or in-cabin behavior models, dashcam data is the wrong modality.
Can I use the Driver Monitoring Dataset (DMD) commercially?
DMD is released under a research license, so it is not a commercial-training asset, and it is a fixed in-cabin (DMS) rig filming the driver rather than head-mounted driver-POV. It is useful for benchmarking driver-monitoring models, but it does not give you the head-aligned, gaze-synced footage most in-cabin buyers actually need — and you cannot ship a product trained on it without negotiating terms.
Is there any open dataset with head-mounted driver-POV video I can license?
Not today. The head-mounted egocentric corpora that exist (Aria Everyday Activities, Nymeria) are non-commercial and contain no driving; the one permissively-licensed head-mounted corpus, Egocentric-10K, is entirely factory-floor work. This slice is effectively whitespace, which is exactly why teams commission custom capture for it.
Do you capture in-cabin gaze and IMU alongside the RGB video?
Yes. The default driving spec is head-mounted RGB at 30 fps (60 fps for glance/saccade work) with a synchronized eye-gaze stream, head IMU, and GPS/speed telemetry, all timestamp-aligned per clip. That gaze-plus-motion package is the signal DMS and attention models need and the thing dashcam data can never provide.
How do you handle consent and PII for the driver, passengers, and bystanders?
Every clip carries a consent artifact, and delivery includes a face and license-plate redaction pass. Because capture is commissioned rather than scraped, the consent chain covers the wearer plus any passengers, and bystander handling is documented — the auditable record you need before any in-cabin footage reaches a training set.
Do we get exclusive rights, or will the same footage be sold to competitors?
Exclusivity is a term you choose. Custom driver-POV batches can be held exclusively to you, unlike a shared academic corpus that every competitor also trains on. You also fix the vehicle types, geographies, and glance taxonomy up front, so the dataset matches your product rather than someone else's benchmark.
Looking for egocentric driving dataset?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request driving egocentric data