Sensor · Egocentric data
Egocentric Audio & Audio-Visual Datasets
An egocentric audio dataset pairs first-person, head-mounted video with synchronized on-device audio and time-aligned sound-event labels — contact sounds, machine states, speech presence — so a model learns what a task sounds like, not only how it looks. The open corpora are kitchen-bound and locked down: EPIC-SOUNDS and HD-EPIC are non-commercial, Ego4D and Ego-Exo4D require signed licenses, and EgoLife's continuous audio is a bystander-consent problem. That leaves consented custom capture, with a sound-event taxonomy you define, as the commercial route.
Quick facts
- Resolution
- 1080p @ 30fps video baseline; stereo 2160p @ 60fps for close manipulation with tight contact-sound timing
- Field of view
- ≥120° horizontal — the reach envelope and both hands stay in frame while audio runs
- Mount
- Head-mounted rig with an on-device mic array; optional lavalier for clean wearer speech; never a fixed room mic — the microphone must travel with the wearer
- Sensors
- RGB (baseline), On-device mic array (multichannel), Lavalier mic (optional, wearer speech), Head IMU (optional, for motion-correlated sound)
- Labels
- Time-aligned sound-event segments (contact, machine state, ambient event); Speech-presence flags with speaker role (wearer vs bystander); Audio-visual action labels aligned to the video frames
- Volume
- 40–200 accepted hours per audio program; calibration batch in days
Key papers
Hard citations for the claims above. Each entry pairs a specific number with the paper that reports it.
EPIC-SOUNDS: A Large-Scale Dataset of Actions That Sound
78.4k segments, 44 classes. EPIC-SOUNDS is a large-scale audio-annotation layer over EPIC-KITCHENS egocentric video with 78.4k categorised audible-event segments across 44 sound classes — the reference taxonomy for actions that can be recognised purely from sound.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
3,670 hours, 74 locations. Ego4D spans 3,670 hours of daily-life first-person video from 931 camera wearers across 74 locations in 9 countries, collected under consenting-participant privacy and de-identification standards.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
1,286 hours, 740 participants. Ego-Exo4D pairs simultaneously-captured egocentric and exocentric video of skilled activity from 740 participants across 13 cities — 1,286 hours with multichannel audio, eye gaze, 3D point clouds, camera poses, and IMU.
What audio-annotated data contains
An egocentric audio dataset pairs first-person, head-mounted video with synchronized on-device audio and time-aligned sound-event labels — contact sounds, machine states, speech presence — so a model learns what a task sounds like, not only how it looks. The open corpora are kitchen-bound and locked down: EPIC-SOUNDS and HD-EPIC are non-commercial, Ego4D and Ego-Exo4D require signed licenses, and EgoLife's continuous audio is a bystander-consent problem. That leaves consented custom capture, with a sound-event taxonomy you define, as the commercial route.
The capture settings this covers:
- Contact and impact sounds: objects set down, snapped, poured, torn or clicked into place, where the audio marks the moment of contact the video only implies
- Machine and appliance operation: motors, presses, blenders, power tools and HVAC changing state, where a running / idle / fault sound is a label in its own right
- Speech-in-task: the wearer's instructions, confirmations and think-aloud narration, plus bystander conversation that must be consent-gated and role-tagged
- Ambient and event cues: doorbells, alarms, timers, footsteps and background chatter that place the activity in its environment and segment it in time
- Tool-and-material acoustics: drilling, sanding, cutting, stirring and typing, where the sound encodes which material is being worked and how hard
- Kitchen and food-prep audio (the EPIC-SOUNDS domain): sizzling, chopping, running water, cupboard and drawer actions that look alike on camera but sound distinct
Why robotics and AI labs need audio-annotated data
Apple's EgoDex pretrains dexterous manipulation on large-scale egocentric human video, and audio-annotated first-person footage extends that substrate with the contact and action sounds a multimodal policy can key on. [1]
EgoScale shows dexterous-manipulation performance scaling with the volume and diversity of egocentric human data, which is the case for capturing a large audio-plus-video egocentric corpus rather than reusing a small benchmark. [2]
AoE frames scalable, low-cost collection of egocentric human video of manual tasks as an answer to the data scarcity a synchronized audio-annotated corpus is built to close. [3]
Capture and delivery spec
Every audio-annotated capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.
| Spec | Detail |
|---|---|
| Resolution | 1080p @ 30fps video baseline; stereo 2160p @ 60fps for close manipulation with tight contact-sound timing |
| Frame rate | 30fps video; 48 kHz audio sampling, frame-accurate to the video clock |
| Field of view | ≥120° horizontal — the reach envelope and both hands stay in frame while audio runs |
| Mount | Head-mounted rig with an on-device mic array; optional lavalier for clean wearer speech; never a fixed room mic — the microphone must travel with the wearer |
| Sensors | RGB (baseline), On-device mic array (multichannel), Lavalier mic (optional, wearer speech), Head IMU (optional, for motion-correlated sound) |
| Labels | Time-aligned sound-event segments (contact, machine state, ambient event); Speech-presence flags with speaker role (wearer vs bystander); Audio-visual action labels aligned to the video frames; Per-segment consent status for any captured speech |
| QA gates | Audio-video sync within tolerance (frame-accurate onset alignment); Audio SNR and clipping check on the action segments; Bystander-speech consent and PII redaction pass; Per-clip consent + capture-release artifact attached |
| Delivery | H.265 video + multichannel WAV/FLAC audio + frame-aligned sound-event JSON; consent artifacts shipped per clip, Hugging Face-streamable |
| Volume | 40–200 accepted hours per audio program; calibration batch in days |
Open audio-annotated datasets
The 5 open corpora most relevant to audio-annotated are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. None of them are cleanly licensed for commercial model training — which is the whole reason custom capture exists.
| Dataset | Size / scale | Sensors | License | Commercial use | Gap |
|---|---|---|---|---|---|
| EPIC-SOUNDS | ~78.4k audio segments · 44 sound classes on EPIC-KITCHENS-100 | On-device kitchen audio + sound-event/action labels | Non-commercial research | No | Kitchen-only, and an annotation layer over existing EPIC video — no rights to ship a model on it, and no soundscape beyond food prep. |
| Ego4D (audio) | 3,670 h video; a subset carries multi-channel audio + AV benchmarks | RGB + multi-channel audio | Ego4D signed data-use license | Conditional | Signed-license friction; audio coverage varies by clip and the AV benchmarks are diarisation/social, not contact-sound labels. |
| Ego-Exo4D (audio) | ~1,286 h · 740 participants · time-synced ego+exo with audio | RGB + audio + gaze (multi-view) | Ego-Exo4D signed license | Conditional | Signed license; the audio is ambient capture, not a labeled sound-event taxonomy you can train a detector on directly. |
| HD-EPIC | Highly-detailed egocentric kitchen video incl. synchronized audio | RGB + audio + dense annotations | Non-commercial research | No | Kitchen domain, research-only terms; built for annotation richness, not commercial audio training. |
| EgoLife | Multi-day egocentric life-logging with continuous audio | RGB + long-form audio | Research terms (verify) | Conditional | Continuous home audio is a bystander-consent minefield; terms are research-oriented and the PII exposure is the highest in the set. |
Open datasets vs Truelabel custom capture
Licensing reality: of the corpora above, not one gives you clean commercial rights to sound-labeled first-person audio. EPIC-SOUNDS and HD-EPIC are non-commercial, Ego4D and Ego-Exo4D require signed data-use agreements, and EgoLife's terms are research-oriented. Custom capture ships a signed wearer consent and capture-release chain on every clip, so the audio is commercially usable the day it lands.
Domain gap: every open egocentric audio corpus is a kitchen (EPIC-SOUNDS, HD-EPIC) or a home lifelog (EgoLife). None of them is your factory line, your machine acoustics or your retail floor, and the acoustics that matter to your model — a specific press cycle, an alarm, a pour into your product — simply aren't in the data. A detector trained on cupboard-and-drawer sounds does not transfer to a shop floor. Custom capture records the exact soundscape your model will run in, not someone else's food prep.
Consent gap unique to audio: bystander speech is personally identifiable in a way a blurred face is not. Scraped or continuously-logged audio carries un-consented voices you cannot lawfully train on. We handle speech-presence flagging, consent gating and redaction at capture, so the audio you receive is already clean.
Taxonomy control: open corpora freeze someone else's sound ontology — EPIC-SOUNDS' 44 classes, tuned to a kitchen. Custom capture lets you define the contact, machine-state and speech-presence labels your model actually predicts, aligned frame-for-frame to the video.
Audio-annotated: by the numbers
The figures below are specific to audio-annotated egocentric data and anchor the comparisons above.
- EPIC-SOUNDS: ~78.4k categorised audio segments across 44 sound classes on EPIC-KITCHENS-100 — the reference egocentric sound-event taxonomy
- Ego-Exo4D: ~1,286 h across 740 participants with time-synchronized audio, under a signed data-use license
- "audio visual dataset" measures 110/mo — the largest measured secondary keyword in the whole egocentric matrix family (DataForSEO, 2026-07-06)
- The "egocentric audio dataset" SERP was 100% academic as of 2026-07-06 (CVPR ego-av-loc, Meta Research, UT Austin, Ego4D, HD-EPIC) — zero commercial pages
- TrueLabel audio programs: 48 kHz multichannel audio, frame-accurate to 30fps video, 40–200 accepted hours per program
How Truelabel captures audio-annotated data
Truelabel runs audio-annotated programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to 40–200 accepted hours per audio program; calibration batch in days, delivered as H.265 video + multichannel WAV/FLAC audio + frame-aligned sound-event JSON; consent artifacts shipped per clip, Hugging Face-streamable. Go deeper via what egocentric data is, egocentric data licensing, the physical AI data marketplace, VLA training data, egocentric kitchen video sourcing, how datasets are delivered, and kitchen manipulation data.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex is a large-scale egocentric video corpus built to pretrain dexterous manipulation from first-person human video.
arXiv ↩ - EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
EgoScale supports the claim that dexterous-manipulation performance scales with the volume and diversity of egocentric human data.
arXiv ↩ - AoE: Always-on Egocentric Human Video Collection for Embodied AI
AoE frames scalable, low-cost collection of egocentric human video of manual tasks as an answer to embodied-AI data scarcity.
arXiv ↩ - Ego-Exo4D project site
Ego-Exo4D provides time-synchronized multi-view egocentric video with audio streams under a signed data-use license.
ego-exo4d-data.org - HD-EPIC: A Highly-Detailed Egocentric Video Dataset
HD-EPIC is a highly-detailed egocentric kitchen dataset that includes synchronized audio alongside dense annotations, released under non-commercial research terms.
arXiv - EgoLife
EgoLife is a multi-day egocentric life-logging dataset that records continuous audio alongside video, carrying high bystander-privacy exposure.
EgoLife project
FAQ
What audio channels ship, and at what sync tolerance to the video?
The baseline is a head-mounted multichannel mic array at 48 kHz, frame-accurate to 30fps video, with an optional lavalier for clean wearer speech. We QA every clip for onset alignment so a contact sound lands on the same frame as the contact you see — that frame-level sync is exactly what an audio-visual model needs and what loosely-muxed footage lacks.
Do you label sound events, or just deliver raw audio?
Both are on the table. The default deliverable is time-aligned sound-event segments — contact, machine state, ambient event — plus speech-presence flags with speaker role, so the audio arrives as training-ready labels rather than a waveform you have to annotate yourself. If you only want raw multichannel WAV/FLAC to run your own detector, we ship that too.
How is bystander speech handled for consent and PII?
Speech is treated as PII. Every segment gets a speech-presence flag and a speaker role (wearer vs bystander); bystander utterances without consent are redacted, and each clip ships with a consent artifact covering the audio. That auditable chain is precisely what a scraped or continuously-logged corpus cannot give you.
Why does a robot or model need audio at all?
Because sound resolves what video can't. EPIC-SOUNDS exists because a pan set down and a drawer shut look the same on a first-person camera and sound completely different, and ObjectFolder shows contact acoustics encode an object's material and state. Audio also flags machine faults before they're visible, confirms an action completed even when the hands occlude the object, and disambiguates fast manipulation the frame rate blurs. For a robot policy, the click of a latch or the change in a motor's pitch is a reliable success signal; for an assistant model, hearing the task is often the difference between guessing and knowing. That is signal a silent RGB stream simply doesn't carry.
Why not just train on EPIC-SOUNDS — it already has sound labels?
You can, as a research base. But it is non-commercial, so you can't ship a model trained on it, and it is entirely kitchen food prep — none of your machines, your floor or your task vocabulary. It's also an annotation layer over existing EPIC video, so you inherit that domain and can't extend it. Custom capture gives you commercial rights and your own soundscape.
Can you match our specific soundscape — our machines and environment?
Yes. We brief collectors on the environment, the equipment and the sound events you care about, then QA each clip against that taxonomy. If your model has to recognise a specific press cycle, appliance or alarm, that becomes a labeled class rather than an unlabeled noise floor.
Looking for egocentric audio dataset?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request egocentric audio data