Task · Egocentric data
Action Recognition Egocentric Datasets
An action recognition dataset is egocentric video segmented into labeled activity classes — the training signal for models that name what a person is doing from their own point of view. The strongest annotated corpora (Charades-Ego, EGTEA Gaze+, Assembly101) ship under non-commercial research licenses, so labs that need commercially-usable, taxonomy-matched action data commission custom capture instead.
Quick facts
- Resolution
- 1080p baseline; 2160p or stereo for fine-grained manipulation verbs
- Field of view
- ≥120° horizontal, head-mounted
- Mount
- Head-mounted only — chest and handheld rigs drop hands and manipulated objects out of frame
- Sensors
- RGB, +IMU for motion-based boundary segmentation, +gaze (optional, disambiguates attended object), +hand pose (optional, for verb grounding)
- Labels
- Frame-aligned verb-noun action segments; Temporal start/end boundaries per action; Per-clip activity class label
- Volume
- 40–200 accepted hours per taxonomy, balanced per action class
Key papers
Hard citations for the claims above. Each entry pairs a specific number with the paper that reports it.
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100
90K action segments on 100 hours. EPIC-KITCHENS-100 densely annotates 100 hours (20M frames) with roughly 90,000 action segments across 45 kitchens — the labeled-hour density that raw first-person corpora cannot match.
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
55 hours, 39.6K action segments. The original EPIC-KITCHENS captured 55 hours of unscripted head-mounted kitchen video (11.5M frames) with 39.6K action segments, establishing the egocentric action-recognition benchmark.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
3,670 hours, 74 locations. Ego4D spans 3,670 hours of daily-life first-person video from 931 camera wearers across 74 locations in 9 countries, collected under consenting-participant privacy and de-identification standards.
What activity recognition data contains
An action recognition dataset is egocentric video segmented into labeled activity classes — the training signal for models that name what a person is doing from their own point of view. The strongest annotated corpora (Charades-Ego, EGTEA Gaze+, Assembly101) ship under non-commercial research licenses, so labs that need commercially-usable, taxonomy-matched action data commission custom capture instead.
The capture settings this covers:
- Kitchen meal-prep runs sliced into fine-grained verbs — pour, stir, cut, open, place — for cooking-action classifiers.
- Assembly and repair workflows where one action (align, screw, snap) flows into the next, stress-testing temporal action segmentation.
- Whole-home activities-of-daily-living scripts — brushing teeth, making a bed, folding laundry — for eldercare and ambient-assisted-living recognition.
- Multi-actor social activities captured from paired first- and third-person cameras, so a model sees the same action from two viewpoints.
- Long-horizon procedural tasks where the label is not a single verb but an ordered activity graph.
Why robotics and AI labs need activity recognition data
Apple's EgoDex learns dexterous manipulation from large-scale egocentric human video, evidence that recognizing a human action is the first step to reproducing it on a robot [1].
The EgoScale study found dexterous-manipulation performance scaling log-linearly with the volume and diversity of egocentric human demonstrations, so labeled action variety — not just raw hours — is now the binding constraint [2].
The AoE line of work names the shortage of real-world action data as the main bottleneck holding back humanoid robots, which is why labs are commissioning task-specific egocentric corpora rather than reusing research benchmarks [3].
EgoLive-scale corpora of real-world human tasks anchor humanoid pretraining, and NVIDIA's Cosmos world- and action-model work treats large POV corpora as first-class inputs — both signal enterprise demand for action-labeled first-person data [4] [5].
Capture and delivery spec
Every activity recognition capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.
| Spec | Detail |
|---|---|
| Resolution | 1080p baseline; 2160p or stereo for fine-grained manipulation verbs |
| Frame rate | 30 fps baseline; 60 fps for fast actions (pour, flip, snap) |
| Field of view | ≥120° horizontal, head-mounted |
| Mount | Head-mounted only — chest and handheld rigs drop hands and manipulated objects out of frame |
| Sensors | RGB, +IMU for motion-based boundary segmentation, +gaze (optional, disambiguates attended object), +hand pose (optional, for verb grounding) |
| Labels | Frame-aligned verb-noun action segments; Temporal start/end boundaries per action; Per-clip activity class label; Optional ordered action-graph / procedure sequence |
| QA gates | Hands-and-manipulated-object-in-frame check; Action-boundary agreement across annotators; Taxonomy-conformance audit against the buyer's verb set; Stability and motion-blur floor |
| Delivery | H.265 clips + per-clip JSON action segments (verb-noun, timestamps), Hugging-Face-streamable; consent artifact attached per clip |
| Volume | 40–200 accepted hours per taxonomy, balanced per action class |
Open activity recognition datasets
The 6 open corpora most relevant to activity recognition are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. None of them are cleanly licensed for commercial model training — which is the whole reason custom capture exists.
| Dataset | Size / scale | Sensors | License | Commercial use | Gap |
|---|---|---|---|---|---|
| Charades-Ego | 157 activity classes · paired first-/third-person clips | Head-mounted RGB | Non-commercial research (default) | No | Scripted, acted activities rather than spontaneous behavior; commercial use requires negotiation |
| EGTEA Gaze+ | ~28 h cooking · 106 fine-grained action classes · synchronized gaze | Head-mounted RGB + gaze | Research only | No | Single kitchen domain; research license blocks any shipped model |
| Assembly101 | 513 h · 4,321 videos · 1,380 fine-grained actions · 3D hand poses | 4 ego + 8 exo RGB | Non-commercial research | No | Toy take-apart/assemble proxy, not a real production line; NC-licensed |
| HoloAssist | ~166 h interactive assembly/repair · instruction-following sessions | HoloLens 2 RGB + hand + eye | CDLA-Permissive-style research | Conditional | Permissive-leaning but verify terms; staged two-person tasks, narrow object set |
| ADL (UCI 2012) | ~10 h · 20 people · 18 daily-living activities | Chest-mounted RGB | Not clearly stated | No | 2012 chest-mount, dated resolution; unstated license means treat as non-commercial |
| Aria Everyday Activities | ~7.3 h everyday-activity recordings | Aria glasses RGB + SLAM + gaze + IMU | Non-commercial research | No | Tiny at 7.3 h; sensor-rich but NC and single-wearer scale |
Open datasets vs Truelabel custom capture
The annotation is the product, and the best action annotations are legally locked: Assembly101's fine-grained actions and EGTEA's gaze-aligned verbs are exactly the label quality labs want, and both are non-commercial, so you cannot ship a model trained on them [6].
Egocentric-10K is the one large permissively-licensed corpus (Apache-2.0), but it is thousands of hours of raw, unlabeled factory POV with zero action segments — it pretrains representations, not classifiers.
Every open action corpus freezes someone else's taxonomy; custom capture lets you define the verb-noun label set your policy actually consumes, then collect balanced examples per class instead of the long-tailed distributions academic sets ship with.
Consent and provenance travel with each clip — a documented wearer-and-bystander consent chain and a per-clip commercial license make the footage exclusive to you, the opposite of a shared academic benchmark.
Activity recognition: by the numbers
The figures below are specific to activity recognition egocentric data and anchor the comparisons above.
- 1,380 fine-grained action classes across 513 hours and 4,321 videos in Assembly101
- 157 activity classes in Charades-Ego (paired ego/exo)
- 106 fine-grained action classes in EGTEA Gaze+ over ~28 cooking hours
- 10,000 raw unlabeled hours in Egocentric-10K with zero action segments
- ~7.3 hours in Aria Everyday Activities under a non-commercial license
- 18 daily-living activities across 20 people in the 2012 ADL/UCI set
How Truelabel captures activity recognition data
Truelabel runs activity recognition programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to 40–200 accepted hours per taxonomy, balanced per action class, delivered as H.265 clips + per-clip JSON action segments (verb-noun, timestamps), Hugging-Face-streamable; consent artifact attached per clip. Go deeper via egocentric data, egocentric data licensing, egocentric kitchen video for cooking actions, industrial egocentric video for assembly actions, and warehouse egocentric video.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex learns dexterous manipulation from large-scale egocentric human video, evidence that recognized human actions transfer to robots.
arXiv ↩ - EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
EgoScale finding that dexterous-manipulation performance scales with the volume and diversity of egocentric human demonstrations.
arXiv ↩ - AoE: Always-on Egocentric Human Video Collection for Embodied AI
AoE frames scalable, low-cost egocentric human-video collection as the answer to the scarcity of real-world action data.
arXiv ↩ - EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks
EgoLive is a large-scale egocentric dataset of real-world human tasks used to anchor humanoid pretraining.
arXiv ↩ - Physical AI with World Foundation Models | NVIDIA Cosmos
NVIDIA Cosmos world- and action-model platform treating large egocentric/POV corpora as first-class training inputs.
NVIDIA ↩ - Assembly101: A Large-Scale Multi-View Video Dataset
Assembly101 is a procedural take-apart/assemble dataset with fine-grained action labels and 3D hand poses under a non-commercial research license.
Assembly101 project ↩ - Charades-Ego: paired first- and third-person activity videos
Charades-Ego paired first-/third-person activity videos with 157 activity classes under a non-commercial default license.
Allen Institute for AI (PRIOR) - Extended GTEA Gaze+ (EGTEA Gaze+)
EGTEA Gaze+ egocentric cooking dataset with 106 fine-grained action classes and synchronized gaze under a research license.
Georgia Tech (First Person Vision) - HoloAssist: an Egocentric Human Interaction Dataset
HoloAssist interactive assembly/repair egocentric dataset (~166 h) under a CDLA-Permissive-style research license.
Microsoft Research - MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain
MECCANO multimodal (RGB/depth/gaze) egocentric industrial-like action dataset under a research-only license.
University of Catania (IPLAB) - IndustReal: industrial procedure-execution dataset
IndustReal egocentric procedure-execution dataset on a toy-construction proxy under CC BY-SA.
IndustReal project - Detecting Activities of Daily Living in First-Person Camera Views (ADL dataset)
2012 UCI ADL chest-mounted first-person dataset (20 people, 18 daily-living activities) with license terms not clearly stated.
UC Irvine (Pirsiavash & Ramanan) - Aria Everyday Activities (AEA)
Aria Everyday Activities: ~7.3 h of sensor-rich everyday-activity egocentric recordings under a non-commercial research license.
Meta / Project Aria - EgoLife
EgoLife multi-day life-logging capture whose dataset card claims MIT but carries consent/privacy exposure.
EgoLife project - Egocentric-10K
Egocentric-10K as a large raw egocentric corpus with no action-segment labels.
Hugging Face - Egocentric-10K dataset card and license
Egocentric-10K dataset card for its Apache-2.0 license characterization and the 10,000 h / 2,138 workers / 87 factories figures.
Hugging Face
FAQ
Why not just train on the big open action datasets for free?
Because the good ones are off-limits commercially. Charades-Ego defaults to a non-commercial license, EGTEA Gaze+ is research-only, and Assembly101 is non-commercial — none can legally sit under a model you ship. Egocentric-10K is permissively licensed but has no action labels at all. Custom capture is how you get annotated action data you can actually deploy.
What label format do you deliver for action recognition?
Frame-aligned verb-noun action segments with explicit start/end timestamps, a per-clip activity class, and optionally an ordered action-graph for procedural tasks — delivered as per-clip JSON alongside H.265 video, Hugging-Face-streamable.
Can you match our action taxonomy and verb set?
Yes. You freeze the label schema, and we collect balanced examples per action class against it. That is the main reason to commission capture instead of reusing an academic set whose taxonomy and class distribution were designed for a different benchmark.
Do you capture gaze, IMU, or hand pose alongside RGB?
Yes, as optional add-ons. IMU helps segment action boundaries from motion, gaze disambiguates which object is being acted on, and hand pose grounds the verb. The RGB-plus-IMU baseline covers most human-activity-recognition training needs.
Do we get exclusive rights, or is the same footage sold to competitors?
Custom-captured action data is delivered exclusive to you under a per-clip commercial license. Unlike a shared academic benchmark, no competitor is training on the same clips.
How is consent handled for wearers and bystanders?
Every clip carries a documented wearer consent record, bystander handling per the capture location's rules, and optional face/PII blurring — the provenance chain that research corpora like ADL or EgoLife do not guarantee.
Looking for action recognition dataset?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request action-recognition egocentric data