Physical AI Implementation Guide
World Model Training Data: What Physical AI Teams Actually Need to Source
A world model learns to predict how a scene changes when an agent acts, so its training data has to preserve the full causal chain—observation, action, and the next observation that resulted—rather than isolated frames. Teams combine two regimes: web-scale video for temporal and physical priors, and action-labeled robot trajectories that ground prediction in real cause and effect. Open X-Embodiment is the public scale reference—its project page reports 1M+ trajectories across 22 embodiments, while its abstract headlines 527 skills across 160,266 tasks rather than a single trajectory count, so anchor budgets on that project-page figure knowingly; production teams close the coverage gap with custom teleoperation and egocentric capture carrying per-step actions, failure recoveries, and provenance.
Quick facts
- Topic
- World Model Training Data
- Audience
- Robotics ML engineers, procurement leads, research teams
- Deliverable
- Sourcing playbook with capture spec + accept-rule criteria for action-conditioned data
What a world model learns—and why a static dataset starves it
A world model predicts the next environment state from the current observation and a proposed action. That one sentence dictates the entire data requirement: the model is learning a transition function, so the training data has to contain transitions. An observation paired with the action taken and the observation that resulted—not a frame in isolation. A dataset of beautifully labeled images teaches appearance and semantics; it says nothing about what happens when a gripper closes half a centimeter early, or when a mug tips past its center of mass. World models live on exactly that information.
This is the structural reason image-annotation pipelines do not produce world-model data. Bounding boxes, segmentation masks, and captions describe a moment. A world model needs the arrow between moments: cause, then effect, with the action that connected them recorded at the rate the controller runs. Ha and Schmidhuber's original formulation made the point early—they compressed observations into a latent space and then trained a separate model whose only job was to predict the next latent state, because prediction is the thing the agent uses to plan[1].
The practical test for any candidate dataset is simple. Can you reconstruct, for a given timestep, (1) what the agent saw, (2) what it did, and (3) what it saw next? If any of the three is missing or unsynchronized, the data trains a video generator or a classifier, not a world model.
The two data regimes: web video for priors, robot trajectories for grounding
Modern world-model stacks draw from two very different pools, and most of the confusion in sourcing comes from treating them as interchangeable.
The first is large-scale video without action labels. This is what gives a model its temporal and physical priors—object permanence, how liquids pour, how a shadow moves as a hand crosses a light. NVIDIA's Cosmos world foundation models are pretrained on video precisely to hand robotics teams this kind of general physical prior before any robot data enters the loop[2]. Web and egocentric video is abundant and cheap to accumulate, but it is action-free: you can see the hand move, but the joint commands that produced the motion are not in the file.
The second pool is action-labeled robot trajectories. Here every observation carries the action the policy or teleoperator issued, so the model can learn action-conditioned dynamics rather than passive prediction. NVIDIA's GR00T N1 is explicit about needing both: it is trained on a heterogeneous mixture of real-robot trajectories, human videos, and synthetically generated data, with a vision-language module feeding a diffusion-transformer action module[3]. The video teaches the model what the world looks like and how it tends to move; the trajectories teach it what its own actions do to that world.
The budgeting mistake is spending everything on one pool. Video-only models predict plausible futures that ignore the agent; trajectory-only models are sample-starved on visual and physical diversity. The sourcing job is to buy the scarce half—action-labeled, embodiment-matched trajectories in your target environment—and let cheap video carry the priors.
Open X-Embodiment: the public scale reference (and its ceiling)
When teams ask "how much data is enough," the honest answer starts with the largest public reference point. Open X-Embodiment assembles data from 22 different robots collected across 21 institutions, demonstrating 527 skills spanning 160,266 tasks[4]. The project reports 1M+ real robot trajectories covering those 22 embodiments, from single arms to bi-manual robots and quadrupeds[5]. That is the number to anchor against—and to correct a common misattribution: the 22-embodiment, million-trajectory figure belongs to Open X-Embodiment, not to any single VLA trained on top of it.
Use Open X-Embodiment for what it is: a broad, standardized aggregation that proves cross-embodiment transfer is real and gives you a pretraining base. Its ceiling is just as important. It is a fixed collection of what a set of labs happened to record. Your robot's morphology, your gripper, your camera placement, your factory lighting, and your task's failure modes are almost certainly underrepresented or absent. A world model trained only on it will predict confidently inside that distribution and hallucinate outside it. Closing the gap is a targeted-capture problem, not a bigger-download problem—you commission the transitions the public aggregation is missing.
Why 'General Agents Contain World Models' raises the data bar
There is now a formal argument for why this data matters at all. The 2025 result General Agents Contain World Models shows that any agent capable of generalizing to multi-step, goal-directed tasks must have learned a predictive model of its environment—and that this model can be extracted from the agent's policy[6]. The consequence for data is direct: the paper argues that increasing an agent's performance, or the complexity of the goals it can reach, requires learning an increasingly accurate world model[6].
Read that as a data-coverage statement. Accuracy in a learned dynamics model is bounded by the transitions the model has seen. If you want an agent that reliably chains ten steps in a cluttered scene, you cannot get there on trajectories that only ever show clean, single-step successes—the model has no examples of the intermediate and off-nominal states those long horizons pass through. Harder goals do not just need more data; they need transitions that cover the states the harder goals traverse. That is the difference between a dataset that scales a number and one that scales capability.
What to specify when you source world-model data
A world-model capture spec is more demanding than a perception-labeling spec because it constrains the recording rig, not just the annotation. The fields that actually change model quality:
Modalities and synchronization. RGB at minimum, usually RGB-D, plus proprioception (joint angles, end-effector pose, gripper state). Every stream needs a shared clock; skew above roughly a control period pairs the wrong action with the wrong frame and quietly poisons the transition.
Actions at controller rate. Record the action issued each timestep at the frequency the controller runs, in a documented space (joint velocity, end-effector delta, or torque). Down-sampled or interpolated actions break the causal link the model is trying to learn.
Temporal windows and episode boundaries. World models are trained on sequences; clean `is_first`/`is_last` markers and consistent episode lengths matter as much as the content inside them.
Failure and recovery coverage. Deliberately include episodes where something goes wrong and the operator recovers. These off-nominal transitions are the ones a success-only dataset lacks and the ones long-horizon planning depends on.
Scene, object, and lighting diversity. Vary the things your deployment will vary. Distribution breadth in the data caps distribution breadth in the model.
Provenance and consent. For any commercial world model, per-trajectory provenance and contributor consent are procurement blockers if they arrive after delivery. Specify them up front. Truelabel ships per-trajectory provenance and consent artifacts with every batch, which is the difference between data you can train on and data legal will not clear.
Delivery: the fields a world model actually consumes
World-model data ships in the same episodic formats as the rest of the robot-learning stack, and the schema is not optional decoration—it is the interface the training loop reads. RLDS structures each episode as a sequence of steps, where every step carries an observation dictionary, an action vector, and boolean flags for episode boundaries[7]. LeRobot stores the same synchronized observation, action, and metadata streams in a PyTorch-native layout[8]. MCAP handles the multi-sensor time-series case when you are aligning cameras, depth, and telemetry from heterogeneous sources.
For a world model specifically, confirm the delivered episodes expose the transition triple cleanly: observation at t, action at t, and observation at t+1 reachable without guesswork, plus camera intrinsics and timestamps so you can verify synchronization yourself. Truelabel delivers in RLDS, LeRobot, MCAP, or a custom schema to S3, GCS, or Azure, and the 750+ datasets profiled in the research catalog are indexed by exactly these fields so you can check format fit before sourcing custom capture. DROID—76,000 teleoperation trajectories across 564 scenes—is a good public example of the density and format a manipulation world model expects[9].
Common failure modes in world-model datasets
Four defects account for most world-model data that trains poorly.
No action labels. The most common one. A vendor delivers gorgeous video and calls it robot data; without the per-step action, it trains passive video prediction, not an action-conditioned model. Large egocentric datasets fall into this trap for buyers—useful for hand and object priors, but they rarely include robot proprioception or the consent artifacts a commercial model needs[10].
Success-only trajectories. If every episode ends in success, the model never learns the dynamics of things going wrong, and its predictions collapse the moment a real rollout drifts off-nominal. Recovery episodes are a feature to buy, not noise to filter.
Temporal skew. Unsynchronized streams are the silent killer. The data looks complete; the transitions are subtly wrong because the action logged at t actually caused the frame at t+2. This is invisible without checking timestamps, which is why you demand them in delivery.
Distribution collapse. Ten thousand trajectories in one kitchen under one light is one environment sampled ten thousand times. The model overfits the backdrop and fails on yours. Diversity is the spec line that most affects generalization and the easiest one to let slide.
Sourcing world-model data with Truelabel
The hard part of world-model data is not annotation—it is capturing action-conditioned transitions in the specific embodiments and environments your model will face, with provenance clean enough to deploy. That is a marketplace problem. On Truelabel, a buyer posts a spec, vetted capture partners return sample batches, and the buyer scales the batches that clear QA. The network runs around 10,000 consented collectors across 100 countries—homes, factories, and streets—so a spec matches to the right environment and embodiment instead of to whatever footage already exists.
Every batch is gated against your acceptance rubric before scale-up, and each trajectory carries per-session consent and per-trajectory provenance, with delivery in RLDS, LeRobot, MCAP, or a custom schema. Sample packets and QA evidence come before you commit to scale. If you are training a vision-language-action model alongside the world model, the same capture programs feed both. Post the spec—embodiment, modalities, action space, failure coverage, and diversity targets—and request world-model training data to get a first calibration batch back for review.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- World Models
Ha and Schmidhuber learn compressed spatial and temporal representations that predict future states for policy learning
worldmodels.github.io ↩ - NVIDIA Cosmos World Foundation Models
NVIDIA Cosmos world foundation models are pretrained on video to generate and reason over physical AI scenarios
NVIDIA Developer ↩ - NVIDIA GR00T N1 technical report
GR00T N1 uses a dual-system architecture (vision-language module plus diffusion transformer) trained on a heterogeneous mixture of real-robot trajectories, human videos, and synthetic data
arXiv ↩ - Open X-Embodiment: Robotic Learning Datasets and RT-X Models
Open X-Embodiment assembles data from 22 robots across 21 institutions demonstrating 527 skills and 160,266 tasks
arXiv ↩ - Project site
Open X-Embodiment contains 1M+ real robot trajectories spanning 22 robot embodiments from single arms to bi-manual robots and quadrupeds
robotics-transformer-x.github.io ↩ - General agents need world models
Any agent that generalizes to multi-step goal-directed tasks must have learned a predictive model of its environment, and harder goals require more accurate world models
arXiv ↩ - RLDS GitHub repository
RLDS defines a standardized episodic schema of observations, actions, and metadata
GitHub ↩ - LeRobot GitHub repository
LeRobot stores synchronized observation, action, and metadata streams for robot learning
GitHub ↩ - DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
DROID provides 76,000 teleoperation trajectories, 350 interaction hours, and 564 scenes
arXiv ↩ - Egocentric video remains useful but incomplete for robot data buyers
Large egocentric datasets help models reason about hands and objects but rarely include robot proprioception or buyer-specific consent artifacts
ego4d-data.org ↩
FAQ
Do world models need action labels, or is video enough?
Both, for different jobs. Action-free video—web clips, egocentric footage—gives a world model its temporal and physical priors and is what pretraining efforts like NVIDIA Cosmos rely on. But an action-conditioned world model, the kind used for planning, needs trajectories where every observation is paired with the action that was taken, recorded at controller rate. Without actions, you train a video predictor that ignores the agent. The scarce, high-value half of the data budget is the action-labeled trajectories in your target embodiment; cheap video carries the priors.
Is Open X-Embodiment enough to train a production world model?
It is the best public starting point and a strong pretraining base—1M+ real robot trajectories across 22 embodiments, 527 skills, and 160,266 tasks. It is not sufficient on its own for a production system, because it is a fixed aggregation of what a set of labs recorded. Your robot morphology, camera placement, environment, and failure modes are usually underrepresented or absent. A model trained only on it predicts confidently inside that distribution and degrades outside it. Production teams pretrain on it, then commission targeted capture to cover the transitions it is missing.
How much world-model training data do I actually need?
Anchor on public reference points rather than a single magic number. Open X-Embodiment aggregates over a million trajectories across 22 embodiments; DROID contributes 76,000 teleoperation trajectories across 564 scenes for manipulation. The more useful framing is coverage, not count: the 2025 result 'General Agents Contain World Models' shows that reaching harder, longer-horizon goals requires an increasingly accurate learned model, which in data terms means transitions that span the states those goals traverse—including off-nominal and recovery states. Diversity and failure coverage move capability more than raw trajectory totals do.
What is the difference between a world model and a simulator?
A simulator is hand-coded physics: it computes environment behavior from known equations, is exact within its modeled scope, and fails on phenomena it does not model. A world model is a neural network that learns dynamics from data, generalizes to phenomena a simulator would not capture, but accumulates prediction error over long rollouts. The data implication is the whole point: a simulator needs no training data, while a world model is only as good as the transitions it has seen—which is why sourcing action-conditioned, diverse, real-world trajectories is the gating problem.
Which formats should world-model training data ship in?
The episodic robot-learning formats: RLDS, LeRobot, or MCAP, or a custom schema mapped to those. RLDS and LeRobot both store per-step observations, actions, and metadata with clean episode boundaries; MCAP handles multi-sensor time-series alignment. For a world model specifically, verify the delivered episodes expose the observation-action-next-observation triple cleanly and include camera intrinsics and timestamps so you can independently confirm synchronization. Truelabel delivers in all of these to S3, GCS, or Azure.
Can egocentric human video train a world model?
It contributes priors, not the full model. Egocentric datasets are strong for learning hand-object interaction, affordances, and task structure from a first-person view, and they scale cheaply. Their limits for world-model training are that they are action-free (no robot proprioception or issued commands) and often lack the consent artifacts a commercial deployment requires. Use them to pretrain visual and physical priors, then ground the model in action-labeled robot trajectories—teleoperation or in-embodiment capture—that carry the actions and provenance the video does not.
Looking for world model training data?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request world-model training data