Environment · Egocentric data
Egocentric Cooking Video Datasets
A cooking video dataset for egocentric AI is first-person, head-mounted footage of recipe and kitchen actions — chopping, pouring, searing, plating — carrying recipe-step and verb-noun action labels. The richest open cooking corpora (EPIC-KITCHENS-100, HD-EPIC, EGTEA Gaze+) ship under non-commercial research licenses, and the largest (YouCook2, roughly 2,000 videos) is third-person YouTube footage, so consented custom capture is the only clean route to commercially-usable, first-person cooking-action data.
Quick facts
- Resolution
- 1080p @ 30fps baseline; 2160p or 60fps for fast knife work and pan transitions
- Field of view
- ≥120° horizontal — both hands, the board and the pan stay in frame through each action
- Mount
- Head-mounted glasses or cap rig — never chest-mount or tripod, which lose the hands during fine prep and plating
- Sensors
- RGB (baseline), IMU (head, optional wrist), Gaze (optional — attention on the attended ingredient or tool), Depth (optional — for object and ingredient pose)
- Labels
- Recipe/step segments (mise en place, prep, cook, plate, clean); Frame-aligned verb-noun action segments (chop, pour, stir, sear, flip, plate); Ingredient and cookware state transitions (raw → chopped → cooked; empty → full)
- Volume
- 40–150 accepted hours per recipe/cuisine program; pilot batch in days
Key papers
Hard citations for the claims above. Each entry pairs a specific number with the paper that reports it.
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100
90K action segments on 100 hours. EPIC-KITCHENS-100 densely annotates 100 hours (20M frames) with roughly 90,000 action segments across 45 kitchens — the labeled-hour density that raw first-person corpora cannot match.
HD-EPIC: A Highly-Detailed Egocentric Video Dataset
41 hours, 69 recipes. HD-EPIC is 41 hours of unscripted egocentric kitchen video across 9 kitchens, densely annotated with recipe steps, 59K fine-grained actions, 51K audio events and 20K object movements, all grounded in 3D digital twins of 413 kitchen fixtures.
My View is the Best View: Procedure Learning from Egocentric Videos
62 hours, 16 tasks. EgoProceL is an egocentric procedure-learning dataset of 62 hours of first-person video from 130 subjects across 16 tasks, motivated by the finding that first-person video gives an unobstructed, unoccluded view of the manipulated object that third-person procedure-learning video lacks.
What cooking egocentric data captures
A cooking video dataset for egocentric AI is first-person, head-mounted footage of recipe and kitchen actions — chopping, pouring, searing, plating — carrying recipe-step and verb-noun action labels. The richest open cooking corpora (EPIC-KITCHENS-100, HD-EPIC, EGTEA Gaze+) ship under non-commercial research licenses, and the largest (YouCook2, roughly 2,000 videos) is third-person YouTube footage, so consented custom capture is the only clean route to commercially-usable, first-person cooking-action data.
The capture settings this covers:
- Knife work in close-up: peeling, dicing, julienning and deboning, with both hands and the cutting board held in frame through every cut.
- Measuring and combining — scooping and levelling, pouring liquids to a line, whisking, folding a batter, and kneading dough by hand.
- Heat management at the stove: searing, flipping in a pan, stirring a pot, adjusting a burner, and checking doneness by eye and touch.
- Plating and assembly — portioning, spooning and streaking a sauce, garnishing, and stacking or layering a finished dish.
- Appliance operation: loading an oven, running a stand mixer, blender or pressure cooker, and setting and reading timers mid-task.
- Between-step transitions — re-grasping and swapping utensils, wiping hands, and moving between prep bench, stove and sink without losing the task thread.
Why robotics and AI labs need cooking data
Apple built EgoDex to learn dexterous manipulation from large-scale egocentric human video, and cooking is one of the densest manipulation domains there is — chopping, pouring, flipping and plating in a single clip — which makes first-person cooking footage prime pretraining fuel. [1]
EgoScale reported dexterous-manipulation performance scaling log-linearly with the volume and diversity of egocentric human data, so a recipe- and cuisine-diverse cooking corpus is a measurable ROI lever rather than a nice-to-have. [2]
AoE frames scalable, low-cost collection of egocentric human video of manual tasks as the answer to real-world data scarcity — and cooking's long-horizon procedural structure is exactly the kind of data teleoperation rigs cannot scale cheaply. [3]
Capture and delivery spec
Every cooking capture program runs to an explicit spec so the footage is training-ready on delivery rather than after a re-shoot. The baseline below is tuned per program; sensors, labels, and volume scale with the buyer's model.
| Spec | Detail |
|---|---|
| Resolution | 1080p @ 30fps baseline; 2160p or 60fps for fast knife work and pan transitions |
| Frame rate | 30fps baseline; 60fps for fast chopping, flipping and pouring motions |
| Field of view | ≥120° horizontal — both hands, the board and the pan stay in frame through each action |
| Mount | Head-mounted glasses or cap rig — never chest-mount or tripod, which lose the hands during fine prep and plating |
| Sensors | RGB (baseline), IMU (head, optional wrist), Gaze (optional — attention on the attended ingredient or tool), Depth (optional — for object and ingredient pose) |
| Labels | Recipe/step segments (mise en place, prep, cook, plate, clean); Frame-aligned verb-noun action segments (chop, pour, stir, sear, flip, plate); Ingredient and cookware state transitions (raw → chopped → cooked; empty → full); Optional 21–25 joint 3D hand pose per hand; Optional gaze fixation on the attended ingredient or utensil |
| QA gates | Both hands and the work surface in frame during every action; No motion blur on fast knife work and pan motions; FOV and horizon check; No identifiable bystander faces without a signed release; Per-clip consent artifact attached |
| Delivery | H.265 clips + per-clip JSON metadata (recipe steps, verb-noun actions, ingredient states), Hugging Face-streamable; consent artifacts shipped per clip |
| Volume | 40–150 accepted hours per recipe/cuisine program; pilot batch in days |
Open cooking datasets
The 6 open corpora most relevant to cooking are compared below on scale, sensors, license, commercial use, and the gap each leaves for a buyer. None of them are cleanly licensed for commercial model training — which is the whole reason custom capture exists.
| Dataset | Size / scale | Sensors | License | Commercial use | Gap |
|---|---|---|---|---|---|
| EPIC-KITCHENS-100 | 100 h · ~90K action segments · 45 kitchens · 97 verb / 300 noun classes | Head-mounted RGB; dense action + narration labels (no depth) | CC BY-NC 4.0 | No | The de-facto egocentric cooking-action benchmark, but non-commercial — you can benchmark on it, you cannot ship a model trained on it, and the kitchens and recipes are the authors', not yours. |
| HD-EPIC | Highly-detailed egocentric kitchen video with dense recipe-step, action and 3D annotations | RGB + audio + multi-layer dense annotations | CC BY-NC 4.0 (research) | No | The richest annotation layer in the family, but the same non-commercial EPIC license — and the annotation schema is fixed to the paper's recipe set. |
| EGTEA Gaze+ | ~28 h egocentric cooking · 106 fine-grained action classes · synchronized gaze | Head-mounted RGB + gaze | Research only | No | Gaze-aligned cooking verbs are exactly the label quality labs want, but it is a single research kitchen under a research license — no commercial path, no taxonomy control. |
| EgoProceL | ~62 h of procedural-task video (cooking among 16 tasks) with procedure-step labels | Head-mounted RGB + procedure/key-step annotations | Research-only | No | Great for procedure/step-learning research, but it is a procedure-general aggregate — cooking is a fraction of it — and research-only terms exclude commercial use. |
| EgoExoLearn | ~120 h of paired ego + exo procedural task-following (cooking and lab procedures) | Paired ego + exo RGB with demonstration-following annotations | Research license | No | Useful for demonstration-following and ego-exo transfer, but the cooking share is mixed with lab procedures and the whole set is research-licensed. |
| YouCook2 | ~2,000 instructional cooking videos across 89 recipes (third-person YouTube) | Third-person YouTube video + temporal recipe-step segments | Research (YouTube-sourced) | No | The largest cooking corpus by video count, but it is third-person YouTube footage under creator rights — the wrong viewpoint for a robot that cooks from its own head, and not a clean commercial license. |
Open datasets vs Truelabel custom capture
Every strong cooking corpus is legally off-limits commercially. EPIC-KITCHENS-100 and HD-EPIC are CC BY-NC 4.0, EGTEA Gaze+ and EgoProceL are research-only, and EgoExoLearn is research-licensed. You can benchmark a cooking model on all of them; you cannot ship one trained on any of them.
The biggest cooking dataset is the wrong point of view. YouCook2 has roughly 2,000 instructional videos, but it is third-person YouTube footage — a camera watching a cook, not the cook's own eyes. A robot that chops and plates sees the board and the pan from its own head, so first-person is the requirement, and it is exactly what the open cooking sets ship least of under a commercial license.
Open corpora freeze someone else's recipes and taxonomy. EPIC-KITCHENS-100's 97 verbs and 300 nouns are the de-facto cooking-action schema, but they are the authors' kitchens and dishes. Custom capture lets you specify the cuisines, recipes, ingredient sets, and the exact step and verb-noun taxonomy your policy consumes, delivered in your episode format.
Home kitchens are private spaces, so provenance is not optional. Cooking footage shows hands, faces, and someone's actual kitchen — custom capture attaches a per-clip consent artifact and can grant exclusivity, so the same recipe footage is not also feeding the lab training against you.
Cooking: by the numbers
The figures below are specific to cooking egocentric data and anchor the comparisons above.
- EPIC-KITCHENS-100: 100 hours, ~90K action segments across 45 kitchens — the de-facto egocentric cooking benchmark, CC BY-NC 4.0
- EPIC-KITCHENS-100 taxonomy: 97 verb classes and 300 noun classes, non-commercial
- YouCook2: ~2,000 instructional cooking videos across 89 recipes — third-person YouTube, not egocentric
- EgoProceL: ~62 hours of procedural-task video (cooking among 16 tasks), research-only
- EgoExoLearn: ~120 hours of paired ego/exo procedural following, research license
- "cooking video dataset" primary is clean — 0 hits in the 360-page corpus (2026-07-06)
How Truelabel captures cooking data
Truelabel runs cooking programs on a network of 20,000+ consented collectors across nine countries, capturing to your brief on a head-mounted rig. Every clip passes per-clip machine QA — head-mount stability, field of view, and hands-in-frame coverage — and ships with a signed wearer consent artifact and provenance manifest. A calibration pilot returns its first batch in days, then accepted batches scale to 40–150 accepted hours per recipe/cuisine program; pilot batch in days, delivered as H.265 clips + per-clip JSON metadata (recipe steps, verb-noun actions, ingredient states), Hugging Face-streamable; consent artifacts shipped per clip. Go deeper via egocentric kitchen video sourcing spec, kitchen manipulation training data, what egocentric data is, egocentric data licensing, VLA training data, and the physical AI data marketplace.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
EgoDex is a large-scale egocentric video corpus built to learn dexterous hand-object manipulation, making first-person manual-task footage the base layer of the training stack.
arXiv ↩ - EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
EgoScale reports dexterous-manipulation performance scaling log-linearly with the volume and diversity of egocentric human data.
arXiv ↩ - AoE: Always-on Egocentric Human Video Collection for Embodied AI
AoE frames scalable, low-cost collection of egocentric human video of manual tasks as an answer to real-world embodied-AI data scarcity.
arXiv ↩ - EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks
EgoLive is a large-scale egocentric dataset of real-world human tasks used to lift manipulation policies, evidence that first-person task footage feeds robot skill.
arXiv - EPIC-KITCHENS-100 dataset page
The EPIC-KITCHENS-100 dataset page is the source for 100 hours, ~90K action segments, 45 kitchens, 97 verb classes and 300 noun classes.
epic-kitchens.github.io - EPIC-KITCHENS-100 annotations license
The EPIC-KITCHENS-100 annotations license file states the material is CC BY-NC 4.0 and may not be used for commercial purposes.
GitHub - HD-EPIC: A Highly-Detailed Egocentric Video Dataset
HD-EPIC is a highly-detailed egocentric kitchen video dataset with dense multi-layer annotations, released under the non-commercial EPIC license family.
arXiv - Extended GTEA Gaze+ (EGTEA Gaze+)
EGTEA Gaze+ is an egocentric cooking dataset with roughly 28 hours, 106 fine-grained action classes and synchronized gaze under a research license.
Georgia Tech (First Person Vision) - My View is the Best View: Procedure Learning from Egocentric Videos
EgoProceL is an egocentric procedure-learning benchmark spanning multiple procedural tasks (cooking among them) under a research-only license.
arXiv - EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World
EgoExoLearn pairs asynchronous egocentric and exocentric video of procedural activities, including cooking and lab procedures, under a research license.
arXiv - YouCook2: instructional cooking video dataset
YouCook2 is an instructional cooking video dataset of roughly 2,000 third-person YouTube videos with temporal recipe-step segments, not egocentric footage.
University of Michigan - Physical AI with World Foundation Models | NVIDIA Cosmos
NVIDIA Cosmos world- and action-model work treats large egocentric/POV corpora as first-class training inputs for physical AI.
NVIDIA - Humanoid data: 10 Things That Matter in AI Right Now | MIT Technology Review
MIT Technology Review documents the real-world manipulation-data bottleneck that makes consented first-person capture urgent.
MIT Technology Review
FAQ
Is your cooking data egocentric or third-person?
Egocentric — first-person, head-mounted, from the cook's own point of view. Most cooking video buyers can find (YouCook2, recipe channels) is third-person YouTube footage of someone being filmed. A robot cooks from its own head, so it needs the first-person view, and that is exactly what open cooking data ships least of under a clean commercial license.
Why can't I just use EPIC-KITCHENS or YouCook2?
Because neither is a clean, first-person, commercial-license source. EPIC-KITCHENS-100 is CC BY-NC 4.0 — excellent for benchmarking, non-commercial for training — and YouCook2 is third-person YouTube footage under creator rights. They are the right references and the wrong rights. Custom capture is how you get first-person cooking data you can actually deploy.
Can you match our recipe/step taxonomy and cuisines?
Yes — that is the whole point of commissioning capture. You give us the cuisines, recipes, ingredient sets, and the step and verb-noun schema your model predicts, and we brief collectors and QA every clip against it. Open sets can't do this: EPIC-KITCHENS-100 freezes its own 97-verb/300-noun taxonomy and EGTEA is one research kitchen.
What labels ship — steps, actions, ingredient states, hand pose?
All of them, to spec. The baseline is recipe/step segments (mise en place, prep, cook, plate, clean) and frame-aligned verb-noun action segments, plus ingredient and cookware state transitions (raw to chopped to cooked). We add 3D hand pose and gaze as optional streams so the data is VLA- and manipulation-ready, delivered as per-clip JSON alongside H.265 video.
How is consent handled when filming in home kitchens?
A home kitchen is a private space, so consent is the real gate, not license. Every wearer signs a capture consent, household members and visitors in frame either sign a release or are blurred or excluded by framing, and PII is handled per your retention rules. Each clip ships with its consent artifact — the provenance chain the research corpora rarely document per clip.
How much egocentric cooking data do teams actually need?
Most programs start with a 40–150 accepted-hour pilot on a target recipe or cuisine set, then scale once the policy responds. The payoff is predictable: EgoScale found manipulation performance scaling log-linearly with the volume and diversity of egocentric human data, and Apple built EgoDex at scale for exactly that reason. Recipe and cuisine diversity, not just raw hours, is the lever.
Looking for cooking video dataset?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request cooking egocentric data