Physical-world AI data
Embodied AI Datasets
An embodied AI dataset contains observations, actions, scenes, tasks, or demonstrations that help models perceive and act in physical environments. Egocentric and first-person video can be part of embodied AI data when the viewpoint helps represent interaction, manipulation, navigation, or task context.
Quick facts
- Ego4D embodied benchmark context
- Ego4D contributes about 3,670 hours of egocentric video across 74 locations and 9 countries for benchmark and task-language planning.
- Ego-Exo4D skilled-activity context
- Ego-Exo4D contributes about 1,286 hours from 740 participants or camera wearers across 13 cities and 123 sites for paired-view skilled activities.
- EPIC-KITCHENS-100 action context
- EPIC-KITCHENS-100 contributes 90K action segments, 97 verb classes, and 300 noun classes for kitchen-activity framing under non-commercial public terms.
Comparison
| Perspective | Modality | Task | Domain |
|---|---|---|---|
| Egocentric | RGB, audio, IMU, gaze | Action anticipation | Human activity and tools |
| Robot-mounted | RGB-D, proprioception, actions | Manipulation | Warehouse, lab, home |
| Ego-exo | First- and third-person views | Skilled activity analysis | Sports, craft, repair |
How embodied AI data differs from generic video data
Generic video data may show scenes passively. Embodied AI datasets must map observations to actions, tasks, objects, environments, and sometimes robot state. Egocentric video is valuable when the actor viewpoint reveals task flow that an external camera may miss [1].
Modalities and annotations
Depending on the model, embodied AI data may require RGB video, depth, pose, action labels, narration, object state, robot proprioception, trajectories, success/failure labels, and source documentation. Ego-Exo4D shows why perspective can be part of the dataset design, not just a visual style [2].
Use cases in robotics and embodied AI
Embodied AI datasets can support manipulation, navigation, human demonstration learning, hand-object interaction analysis, tool-use reasoning, success/failure evaluation, and environment-specific model testing. The useful data shape changes by task: manipulation may need object state and action labels, while navigation may need scene context, trajectory, and success criteria.
Public datasets may not match deployment needs
Public datasets help benchmark model behavior and define terms. They may not match the buyer's target embodiment, environment, object set, rights posture, consent basis, or QA rubric. Those gaps belong in a custom collection plan rather than in unsupported performance claims.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- Ego4D: Around the World in 3,000 Hours of Egocentric Video
The Ego4D paper is the source-backed reference for first-person daily-life activity video and benchmark design.
arXiv ↩ - Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
The Ego-Exo4D paper describes skilled human activity from first- and third-person perspectives.
arXiv ↩ - Egocentric video remains useful but incomplete for robot data buyers
Ego4D is an official public reference for egocentric video dataset scope, access, and dataset documentation.
ego4d-data.org - Ego-Exo4D project site
Ego-Exo4D is the official project source for paired first-person and third-person skilled-activity capture.
ego-exo4d-data.org - Ego-Exo4D annotations documentation
Ego-Exo4D annotation documentation supports dataset-structure and skilled-activity-label discussion.
docs.ego-exo4d-data.org - EPIC-KITCHENS project site
EPIC-KITCHENS is an official project reference for egocentric kitchen-activity data.
epic-kitchens.github.io - Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100
The EPIC-KITCHENS-100 paper supports public kitchen-activity benchmark facts and caveats.
arXiv - EPIC-KITCHENS-100 annotations license
The EPIC-KITCHENS-100 annotation license is a visible source for non-commercial licensing caveats.
GitHub
More glossary terms
FAQ
What is an embodied AI dataset?
It is data describing agents acting in physical environments, often including observations, actions, tasks, scenes, and annotations.
How is egocentric video useful for embodied AI?
It can show task execution from the actor viewpoint, including hands, objects, occlusion, tool use, and sequential context.
What data does a robotics model need?
It depends on the task, but common requirements include observations, actions, task labels, object state, metadata, and quality checks tied to the deployment context.
What is the difference between embodied AI data and generic video data?
Embodied AI data is connected to action and physical-world tasks; generic video may lack action labels, embodiment context, rights details, and task-specific metadata.
Find datasets covering embodied AI datasets
Truelabel surfaces vetted datasets and capture partners working with embodied AI datasets. Send the modality, scale, and rights you need and we route you to the closest match.
Map your robotics data requirement to capture, consent, and QA constraints