Robot foundation model data guide
How VLAs, World Models, and Egocentric Data Fit Together
Egocentric data — first-person video shot from the actor's viewpoint — earns its place in a robot training plan because it densely records the things a manipulation policy has to get right: hands approaching objects, the moment of contact, tools changing an object's state, and the near-field occlusions a robot will actually see. It is a scale lever, because humans generate far more of this behavior than any teleoperation rig can. It is not a shortcut to a deployable policy. Raw first-person video carries no robot action vectors, no proprioception, no target-embodiment alignment, and no commercial-use clearance. In every serious result to date, egocentric video is a pretraining and affordance layer that still gets paired with action labels, sensorized human demonstrations, or a smaller amount of robot-native data before a policy controls anything. So the useful framing is not "VLA versus world model" or "egocentric versus teleop." It's a stack. Web-scale vision-language data supplies semantics. Egocentric human video supplies actor-view dynamics and affordances. Sensorized human demonstrations add action-relevant structure. Robot demonstrations and teleoperation align everything to a specific embodiment. Simulation and world-model rollouts add cheap variation. Held-out physical evaluation tells you whether any of it worked. This guide walks that stack, says what each layer proves, and — because the whole point of this cluster is evidence discipline — flags what each layer does not prove. A note on where this argument came from. The public conversation that pushed "video-first robotics" into founder decks is Jim Fan's Sequoia talk. We treat that talk as editorial context and a discovery seed only — a prompt to go find primary sources. Every technical and numeric claim below is cited to a paper, official project page, model card, or format doc, not to the talk.
The data stack, one layer at a time
Think of robot-foundation-model data as six layers, each teaching something the layer above it can't.
Web text / images / video → semantics: object names, language, goals, scene priors Egocentric human video (first-person) → actor viewpoint, hand-object timing, contact, near-field state change Sensorized human demonstrations (gloves, wrist cams, pose) → hand/wrist pose, contact events, robot-relevant action structure Robot demonstrations / teleoperation → embodiment-specific action labels, proprioception, control alignment Simulation / world scans / world-model rollouts → cheap variation, perturbations, imagined futures for planning Held-out physical evaluation → the only layer that tells you the policy actually works
Each arrow is a claim, and each claim has a source. Here is the same stack as a table you can hand to a data lead.
The single most useful thing to internalize: the layers are complements, not substitutes. A team that buys only the cheap, scalable layer (egocentric video) and skips the expensive, aligned layer (robot demos, evaluation) does not have a cheaper policy — it has an unvalidated one.
| Layer | What it teaches | Missing by default | Representative primary source |
|---|---|---|---|
| Web vision-language | Object semantics, language grounding, goals | Any sensorimotor mapping | RT-2 — https://proceedings.mlr.press/v229/zitkovich23a.html |
| Egocentric human video | Actor viewpoint, hand-object timing, contact, object-state change | Robot actions, proprioception, rights | Ego4D — https://ego4d-data.org/, Ego-Exo4D — https://ego-exo4d-data.org/ |
| Sensorized human demos | Hand/wrist pose, contact, retargetable action structure | Direct robot-embodiment fit | UMI — https://umi-gripper.github.io/ |
| Robot demos / teleop | Embodiment-aligned actions, proprioception, real failures | Cheap scale, task breadth | DROID — https://droid-dataset.github.io/, BridgeData V2 — https://rail-berkeley.github.io/bridgedata/ |
| Simulation / world models | Variation, rollouts, action-conditioned prediction | Real physics, provenance | Deep Visual Foresight — https://arxiv.org/abs/1610.00696, DreamZero — https://dreamzero0.github.io/ |
| Held-out physical eval | Deployment-relevant truth | Everything above assumes it | (project-specific eval splits) |
VLA vs world model: two different learning signals
A vision-language-action model (VLA) maps what the robot sees plus a language instruction to an action. The recipe that named the category is RT-2, which co-fine-tunes a vision-language model on both web vision-language tasks and robot trajectories, expressing actions as tokens. RT-2 reported roughly 6,000 robot evaluation trials and, more interestingly, emergent semantic behavior — following commands that referenced objects and concepts never seen in the robot data (RT-2 paper[1]). That is the strength of the VLA bet: it imports the internet's knowledge of nouns.
OpenVLA is the open, reproducible version of that bet — a 7B-parameter model trained on 970,000 robot episodes drawn from Open X-Embodiment. It's worth being precise here, because the web is full of a specific error: OpenVLA predicts discretized action tokens, not a diffusion action head. The paper reports that in its own evaluation this 7B model outperformed the far larger 55B RT-2-X by 16.5% absolute task success rate (OpenVLA paper[2]). Read that as source-specific evidence that a well-chosen data mix can outweigh raw parameter count in a particular evaluation — not a universal law that data diversity always beats scale. The caveat is on OpenVLA's own model card: zero-shot use is tied to the embodiments and domains in the training mix (model card[3]). A VLA is only "general" over the robots it has actually seen.
A world model learns a different thing: given the current state and an action, what happens next? The idea is older than the current branding. Deep Visual Foresight combined action-conditioned video prediction with model-predictive control using unlabeled robot interaction, and could push novel objects around without calibrated cameras, depth sensors, or explicit 3D object models (Deep Visual Foresight[4]). The modern version — NVIDIA's DreamZero and the broader "world action model" (WAM) framing — jointly predicts future world states and the actions to reach them. DreamZero reports more than 2× improvement over VLA baselines on new tasks and environments, and 7Hz closed-loop control after optimization (DreamZero[5], paper[6]). Promising — and lab-reported from a single team, not independently replicated at scale. Frame it that way.
Here's the honest state of play, and it is not the slogan version. VLAs are strong on semantics and weaker, by their critics' account, on physical dynamics. World models attack the dynamics directly. NVIDIA's own technical framing is that the winner may be neither a pure VLA nor a pure world model, but a hybrid: policies that start from a video/world backbone and get aligned with action fine-tuning (NVIDIA WAM blog[7]). "VLAs are dead" is not a claim the evidence supports. "The two bets are converging" is.
Why the first-person viewpoint changes the signal
Third-person web video shows you that a task happened. First-person video shows you the causal chain the way a policy has to consume it: hand enters frame → approaches → contact → object changes state → task progresses. That structure is exactly what an action-conditioned model needs, and it's why egocentric datasets foreground it.
Four things the actor's viewpoint gives you that an outside camera usually doesn't:
• Hand-object timing and contact. The camera sits where the manipulation happens. Ego4D's 3,670 hours from 923 participants across 74 locations and 9 countries is built around daily hand-object activity, with benchmarks for hand-object interaction and forecasting (Ego4D[8]). That's near-field contact data at a scale robots can't self-generate. • Affordances, not just object labels. Hands reveal which parts of an object are actionable — the handle, the lid, the button, the pour lip. HRP turns exactly this into supervision: it extracts hand, object, and contact affordance labels from human video, pretrains robot visual representations on them, and reports over 15% improvement across 5 real-world tasks, 3,000+ robot trials, and 3 robot morphologies (HRP[9]). That is human video being converted into a robot-relevant training signal, with a caveat: the gains are measured in that setup, not promised everywhere. • A reusable motor prior. Human wrist and finger trajectories are a manipulation prior you can retarget. EgoScale pretrains a VLA on 20,854 hours of action-labeled egocentric video using wrist motion and retargeted hand actions, then does aligned human-robot mid-training; it reports a near-perfect log-linear scaling law between data hours and validation loss (R²=0.9983) and a 54% average success-rate improvement over no-pretraining on a 22-DoF dexterous hand (EgoScale[10], paper[11]). Read the recipe, not just the headline: the scaling win still sits on top of aligned robot data and robot evaluation. • Blind-spot coverage when you pair it. First-person view occludes body pose and scene geometry. Ego-Exo4D captures 1,286.3 hours of skilled activity from 740 camera wearers across 13 cities, with synchronized first-person Aria glasses and 4–5 exocentric GoPros plus gaze, IMU, point clouds, and expert commentary (Ego-Exo4D[12]). The pairing exists precisely because ego-only has blind spots.
What egocentric data does not contain by default
This is the section most "egocentric data for robots" content skips, and it's where a data plan lives or dies.
• No robot action vectors or proprioception. A video of a hand does not contain joint commands, gripper states, or force readings. EgoScale doesn't train on raw video — it uses action-labeled ego video and retargeted hand actions. Methods that instead infer latent actions from raw video add a layer of uncertainty you should price in. • An embodiment gap that has to be closed. A human hand can do things a robot hand can't, and vice versa. UMI's own paper is blunt that unstructured human video carries a large embodiment gap, which is why UMI designs a handheld gripper and action interface rather than training on passive clips (UMI[13]). • Ego-motion is mixed into the signal. The camera moves with the wearer's head, so separating "the object moved" from "I turned my head" depends on logging a motion channel. Capture teams should record IMU or head-pose alongside the video — as paired datasets like Ego-Exo4D do, with synchronized IMU and camera pose (Ego-Exo4D[12]) — so reviewers can distinguish camera motion from object motion rather than infer it later. • Fine-grained interaction is still hard. EgoNCE++ shows current egocentric video-language models skew toward recognizing objects rather than temporal dynamics, and degrade on fine-grained hand-object interactions (EgoNCE++[14]). "More ego video" does not automatically fix action understanding. • No rights by default. Public egocentric datasets come with license and consent terms. Ego4D and Ego-Exo4D require license agreements; a dataset's existence is not permission to train a commercial model on it. Rights review is a step, not an assumption. • Plausible video ≠ physical success. A world model can render a convincing future and still be physically wrong. The only correction for that is held-out robot evaluation.
If you take one procurement rule from this page: the value of a first-person clip for training is proportional to the labels and alignment attached to it, not to the raw hour count.
The "ChatGPT moment for robotics" analogy — stated carefully
Here's the argument, made as a qualified claim rather than a slogan.
The LLM playbook was: pretrain on a massive, cheap, self-supervised corpus, then align with a smaller amount of supervised and preference data. The robotics analogy is real at the pretraining layer — egocentric human video is the closest thing robotics has to a web-scale corpus of physical behavior, and EgoScale's clean scaling law is genuine evidence that more diverse human data buys lower loss (EgoScale[11]). That is the part of the thesis the evidence supports.
The part that does not transfer cleanly is the alignment layer. Text alignment is cheap; robot alignment is not. Robot demonstrations require hardware, operators, and time, and they remain necessary for embodiment-specific actions and for evaluation (DROID[15], BridgeData V2[16]). World models can imagine futures to plan against, but they can also hallucinate physics, so real-world evaluation stays the source of truth (Deep Visual Foresight[4]). So the defensible version of the thesis is narrow and, we'd argue, more useful. One useful analogy is that robotics needs a scalable pretraining corpus plus a harder physical-alignment step — the same two-part shape as the LLM playbook, not a promise that the playbook transfers wholesale. The safer conclusion is that data quality, labels, embodiment alignment, and rights are major constraints alongside model architecture — not that any single one is the binding constraint. Not "video solves robotics." Not "teleop is dead." A data-mix problem with a rights problem attached.
TrueLabel's stake in this is deliberately unglamorous: we don't claim model outcomes. What we do is normalize this evidence into rows a buyer can act on, translate it into a capture-and-annotation spec, and keep the rights question in the room from the start. That's the work below.
Scale vs alignment: the tradeoff in one table
Every data source trades scalability against how directly it aligns to your robot. This is the table to argue over in a planning meeting.
The evidence matrix guide puts a primary source behind every one of these rows.
| Data source | Scalability | Embodiment alignment | Action-label strength | Best use | Common failure mode |
|---|---|---|---|---|---|
| Passive egocentric human video | High | Low–medium | Low unless annotated | Representation & affordance pretraining | No robot action labels |
| Sensorized human demonstrations | Medium–high | Medium | Medium–high | Dexterity priors, action bridge | Human→robot retargeting gaps |
| Teleoperation | Low–medium | High | High | Embodiment-specific fine-tuning | Operator throughput, robot uptime |
| Robot demonstrations | Medium | High | High | Policy training & evaluation | Narrow hardware/task coverage |
| Exocentric video | High | Low | Low–medium | Context, multi-view supervision | Weak actor-view intent signal |
| Simulation / world scans | Very high after setup | Variable | Synthetic | Variation, RL rollouts | Sim-to-real / model-reality gap |
Minimum useful data spec
Passive first-person video becomes training-ready only when it carries the fields below. This is the difference between "we have footage" and "we have data a policy can learn from."
TrueLabel delivers in RLDS, LeRobot, MCAP, and custom schemas with per-trajectory provenance and consent artifacts, and sends a sample packet with QA evidence before any scale commitment. Nothing on this page implies a delivery timeline, a model gain, or legal clearance — those are scoped per project.
| Spec dimension | Minimum field | Why it matters |
|---|---|---|
| Viewpoint | First-person position, field of view, frame rate, stabilization state | Prevents vague "egocentric" claims |
| Temporal sync | Timestamps across video, pose, audio, sensors, labels | Required for any action/world-model sequence claim |
| Task language | Instruction or dense narration aligned to segments | Supports VLA language-action alignment |
| Hand/wrist pose | 2D/3D keypoints or sensor stream when available | Supports dexterity and retargeting |
| Contact / object state | Contact events, object identity, state transitions | Supports manipulation and world-state learning |
| Rights | Consent, license, location category, commercial-use terms | The difference between a demo and a shippable asset |
| QA | Blur/occlusion thresholds, sensor dropout, annotation agreement, rejection reasons | Buyer-verifiable reliability |
| Delivery | RLDS, LeRobot-compatible structure, MCAP, custom JSON, a data card | Makes the spec operational |
Claims to avoid (and what to say instead)
We keep this table visible on purpose. It's the fastest way to tell careful robotics-data content from hype.
| Tempting claim | What the evidence actually supports |
|---|---|
| "VLAs are dead." | VLA and world-model approaches are converging; tie any claim to a specific architecture and evaluation (NVIDIA WAM — https://developer.nvidia.com/blog/pretrained-to-imagine-fine-tuned-to-act-the-rise-of-world-action-models/). |
| "World models replace VLAs." | World models add action-conditioned prediction; policy grounding and action alignment still matter. |
| "Egocentric video alone trains deployable robots." | It's useful for pretraining and affordances; deployment needs labels, retargeting, robot data, and evaluation (UMI — https://arxiv.org/abs/2402.10329). |
| "Video models learn physics automatically." | Some systems learn predictive priors in measured settings; physical control still needs validation (Deep Visual Foresight — https://arxiv.org/abs/1610.00696). |
| "Teleoperation is dead." | Teleop is high-alignment but hard to scale; human data reduces, not eliminates, robot-in-the-loop collection. |
| "Public datasets are commercially usable by default." | They require license and intended-use review before commercial training. |
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Primary or official source cited by the authored page
Proceedings of Machine Learning Research ↩ - OpenVLA: An Open-Source Vision-Language-Action Model
Primary or official source cited by the authored page
arXiv ↩ - OpenVLA 7B model card
Primary or official source cited by the authored page
OpenVLA on Hugging Face ↩ - Deep Visual Foresight for Planning Robot Motion
Primary or official source cited by the authored page
arXiv ↩ - DreamZero: World Action Models are Zero-shot Policies
Primary or official source cited by the authored page
NVIDIA Research ↩ - World Action Models are Zero-shot Policies
Primary or official source cited by the authored page
arXiv ↩ - Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models
Primary or official source cited by the authored page
NVIDIA Technical Blog ↩ - Ego4D
Primary or official source cited by the authored page
Ego4D Consortium ↩ - HRP: Human Affordances for Robotic Pre-Training
Primary or official source cited by the authored page
arXiv ↩ - EgoScale: Scaling Human Video to Unlock Dexterous Robot Intelligence
Primary or official source cited by the authored page
NVIDIA Research GEAR Lab ↩ - EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data
Primary or official source cited by the authored page
arXiv ↩ - Ego-Exo4D project site
Primary or official source cited by the authored page
ego-exo4d-data.org ↩ - Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots
Primary or official source cited by the authored page
arXiv ↩ - Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?
Primary or official source cited by the authored page
arXiv ↩ - Project site
Primary or official source cited by the authored page
droid-dataset.github.io ↩ - Project site
Primary or official source cited by the authored page
rail-berkeley.github.io ↩ - Egocentric Data for Imitation Learning
Authored internal route
truelabel.ai - Physical AI data-spec generator
Authored internal route
truelabel.ai - Post a data specification
Authored internal route
truelabel.ai - Robotics' End Game: NVIDIA's Jim Fan
Narrative context and discovery seed only; not technical proof
Sequoia Capital
FAQ
What is egocentric data for robotics?
Egocentric data is first-person sensor data recorded from the acting human's or robot's viewpoint. It's valuable for robot foundation models because the camera sees the same hands, tools, occlusions, and object-state changes that a policy must predict or control. It becomes training-ready when paired with labels — task language, temporal boundaries, hand/wrist pose, contact events — and, for deployment, a smaller amount of robot-aligned data (Ego4D).
Why do VLAs need egocentric data?
VLAs benefit from egocentric data when the target behavior depends on manipulation, tool use, or dexterity. Web images and language teach object semantics; egocentric demonstrations add the temporal sequence of reach, contact, grasp, and state change an action policy has to learn. It's a complement to robot data, not a replacement (RT-2, EgoScale).
Is egocentric video enough to train a robot policy?
Usually not on its own. It's a scalable source of human action priors, but robot demonstrations, teleoperation, or sensorized human data are still needed to align those priors to a robot's embodiment and action space, and evaluation is needed to confirm control (UMI, DROID).
How is egocentric different from teleoperation data?
Egocentric data scales human behavior capture cheaply but carries no robot action labels by default. Teleoperation gives robot-specific action labels and proprioception but is constrained by operator throughput and robot uptime. A sound plan uses egocentric pretraining plus a smaller, embodiment-aligned teleop/robot set — see the imitation-learning rubric.
Does video prediction mean the model learned physics?
No. A world model can predict plausible future frames and still be physically wrong. Action-conditioned prediction supports planning in specific settings, but plausible video is not the same as reliable physical success, which is why held-out robot evaluation stays the source of truth (Deep Visual Foresight).
Can I train a commercial model on public egocentric datasets?
It depends on each dataset's license and intended-use terms. Public datasets like Ego4D and Ego-Exo4D are strong evidence and benchmark sources, but commercial training needs a license and intended-use review — treat them as evidence, not procurement supply (Ego-Exo4D).
What should I actually collect?
Scope the layers your task needs, then specify fields per the minimum-useful spec above: viewpoint, timestamp sync, task language, hand/wrist pose, contact/object state, rights, QA, and delivery format. Draft it with the data-spec generator or post a spec to scope a rights-cleared sample packet.
Planning a VLA or world-model dataset?
Turn the data stack on this page into a request: draft the per-layer capture-and-annotation requirements with the physical AI data-spec generator, or post a spec to scope a rights-cleared sample packet covering the specific layers your architecture needs — before you commit to scale. TrueLabel is a physical AI data marketplace: you post a spec, vetted suppliers return samples. No delivery-time, model-performance, or legal-clearance promises — those get scoped against your task.
Post a spec