VLA data requirements
VLA Training Data Requirements: The Checklist to Run Before You Buy
A vision-language-action (VLA) dataset is training-ready only when each episode carries a language instruction, time-synchronized visual observations, the robot's proprioceptive state, and the exact action the policy will output — all aligned to the same clock and delivered in a schema the trainer can read. Volume alone is not the requirement: OpenVLA reached generalist behavior on 970,000 real-world robot demonstrations, but the same 970,000 episodes are worthless to a policy if the action space, control frequency, or timestamp alignment don't match your robot. Buy against a written requirements spec, review a sample packet before scale, and reject anything that fails the seven checks below.
The short answer: a VLA episode is four aligned streams, not a video
The word that trips people up is "data." A VLA model does not learn from footage. It learns from episodes, and an episode is a bundle of streams that have to agree with each other at every instant: what the robot was told to do (a language instruction), what it saw (camera frames, sometimes depth), how it was configured (proprioceptive state — joint positions, end-effector pose, gripper width), and what it did next (the action). Take away any one of those and you don't have a cheaper dataset, you have a different and mostly useless one.
The clock is the part that decides whether the whole thing is trainable. If the camera timestamps and the proprioception timestamps drift by even a few frames, the model learns to predict actions from observations that came slightly after them, and it quietly bakes that lag into the policy. This is why so many VLA runs "look fine" offline and then stutter or overshoot on hardware. The requirements checklist that follows is really one requirement repeated seven ways: every stream must be present, defined, and synchronized to the same timeline before the data is worth paying for.
Volume is a floor, not the spec
Everyone fixates on episode count, and the reference points are genuinely large. OpenVLA reached generalist manipulation on 970,000 real-world robot demonstrations. DROID collected 76,000 trajectories across 564 scenes and 86 tasks. Open X-Embodiment pooled data from 22 robot embodiments across 21 institutions and showed that heterogeneous cross-robot data can transfer. Those numbers are real, and they anchor what "a lot of data" means for a foundation-scale model.
Here is the part that gets skipped: none of that volume helps if the episodes don't match your robot. A million demonstrations recorded at 5 Hz absolute end-effector control cannot train a policy that outputs 50 Hz joint-velocity deltas without resampling that throws away information and introduces artifacts. Count is the easiest number to negotiate and the least predictive of success. If you are fine-tuning an existing VLA for a narrow task, a few hundred well-specified, on-distribution demonstrations usually move the policy more than tens of thousands of mismatched ones. Ask for the spec first and the count second.
The requirement people forget: action representation
The action stream is where good-looking datasets go to die, because two datasets can describe the same physical motion in incompatible ways. Before you accept a single episode, pin down exactly what the numbers in the action field mean. Is the action an absolute target or a delta from the current state? Is it expressed in end-effector (Cartesian) space or joint space? What is the rotation convention — quaternion, axis-angle, Euler, and in which frame? How is the gripper encoded: continuous width, normalized 0-1, or a binary open/close? What is the control frequency, and does it match the observation rate?
There is also a modeling choice hiding in the data: many strong manipulation policies predict a short sequence of future actions at once rather than one step at a time. That idea, action chunking as introduced by ACT, changes what you need from the data — you want smooth, temporally consistent action sequences, not just correct single-step labels, and you want them at a frequency dense enough to chunk. A vendor who can answer these questions crisply has actually thought about training. One who says "it's standard robot data" has not.
| Dimension | The question to ask | Why a mismatch breaks training |
|---|---|---|
| Reference | Absolute target or delta from current state? | Deltas and absolutes are not interconvertible without the full state trajectory |
| Space | End-effector (Cartesian) or joint space? | Retargeting across spaces needs the robot's kinematics and loses fidelity |
| Rotation | Quaternion, axis-angle, or Euler, and in which frame? | Silent convention mismatches flip or wrap orientations |
| Gripper | Continuous width, normalized, or binary? | A binary label can't teach force-sensitive or partial grasps |
| Frequency | What control rate, and does it match the images? | Resampling to fit your rate discards or fabricates motion |
Coverage: the long tail is the requirement, not the happy path
A dataset that only contains clean, first-try successes teaches a policy that has never seen recovery. Real deployments fail on the variants: a different object finish, a shifted layout, a distractor in frame, a grasp that slips and has to be retried. Coverage requirements should name those explicitly — the object and pose distribution, the environment and lighting variation, and whether the set includes recoveries and near-misses rather than only textbook executions.
This is measurable, and the benchmarks tell you what to measure against. THE COLOSSEUM evaluates manipulation generalization by systematically perturbing lighting, distractors, object color, and camera pose — a useful template for the perturbation axes your own spec should cover. And when you evaluate, evaluate on real hardware under out-of-distribution conditions: ManipArena tests generalist manipulation with 10,812 expert trajectories and 13.5M frames across 20 real-world tasks precisely because offline metrics on in-distribution data flatter a policy that will still fail in the room. Write coverage as a distribution requirement, not a volume one.
- Object/pose distribution: which objects, orientations, and initial states must appear, and in what proportion.
- Environment variation: lighting, backgrounds, distractors, surface and camera-pose changes.
- Recovery behavior: retries, regrasps, and corrections, not only clean successes.
- Negative and edge cases: what the operator was told to do when the task couldn't be completed.
Rights and provenance: the requirement that surfaces after delivery
The failure here is slow. A dataset trains well, ships in a product, and then a customer or partner asks who consented to the footage and under what license — and the answer is a shrug. For VLA data that includes humans, hands, homes, or faces, rights are a hard requirement, not paperwork you retrofit. Each episode should carry a documented consent artifact, a location release where the setting calls for one, and per-trajectory provenance: which operator, which robot, when, where, and under what usage scope.
The reason this belongs in the data spec and not the contract appendix is that it can't be reconstructed later. You cannot go back and get consent for footage already captured by an operator you can't identify. If the provenance chain isn't captured at collection time, it doesn't exist. Treat missing or vague rights documentation as a reject condition on the same footing as a broken action space — because in a commercial deployment, it is the one that gets the whole dataset pulled.
Delivery format and acceptance QA: how to make the spec enforceable
A requirements list only works if a supplier can be measured against it, which means the format has to be one your trainer already reads and the acceptance step has to happen while the collection is still small. Two formats dominate real robot-learning pipelines: RLDS, which stores each episode as a sequence of steps carrying observations, actions, rewards, discounts, and metadata, and LeRobotDataset, the documented format distributed with the LeRobot library. Asking for delivery in one of these is not a preference; it is what keeps you out of a bespoke ETL project for every batch. The generalist policies themselves are built this way — π0 combines large-scale multi-task, multi-robot data into one model, which only works because the underlying episodes share a consistent structure.
The workflow that makes all of it enforceable is boring and it works: write the acceptance rules as pass/fail checks, get a small sample packet, run those checks on it before approving scale, and feed failures back while the correction is cheap. That is exactly the model Truelabel is built around — a physical-AI data marketplace where you post a VLA spec and vetted capture partners return sample packets first, with per-trajectory consent artifacts and provenance, delivery in RLDS, LeRobot, or MCAP, and QA evidence for buyer review. Around 10,000 collectors across 100 countries and 100+ vetted capture partners exist so that "collect data that matches this action space, in these environments, with these rights" is a spec you can send, not an operation you have to build.
- 01
Encode the spec as checks
Turn each requirement into a machine- or reviewer-checkable rule: instruction present and accurate, timestamps aligned within tolerance, action fields in the agreed space and units, metadata and consent complete.
- 02
Review a sample packet, not a pitch
Pull actual trajectories in your target format and run the checks. Play back a handful on your own loader to confirm the action stream reproduces the motion in the video.
- 03
Reject and re-scope before scale
Fail the batch on any hard check — broken action space, drifting clocks, missing rights — and tighten the spec before the supplier collects thousands more of the same mistake.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
FAQ
What data does a VLA model need to train on?
Each training episode needs four aligned streams: a language instruction describing the task, visual observations (camera frames, sometimes depth) with known camera parameters, the robot's proprioceptive state (joint positions, end-effector pose, gripper), and the action the policy should output — all sampled against the same clock. It also needs per-trajectory metadata (robot, operator, environment, timestamps) and, for footage involving people or private spaces, consent and provenance records. Missing or unsynchronized streams make the episode untrainable, not just lower quality.
How much VLA training data is enough?
There is no single number, and volume is a floor rather than the spec. Foundation-scale references are large — OpenVLA trained on 970,000 real-world demonstrations and DROID collected 76,000 trajectories across 564 scenes — but those matter for building generalist models from scratch. If you are fine-tuning an existing VLA for a narrow task, a few hundred well-specified, on-distribution demonstrations often move the policy more than tens of thousands of episodes recorded with a mismatched action space or control frequency. Define the spec first; the count is secondary.
Why does timestamp synchronization matter so much for VLA data?
Because a VLA policy learns to map observations to the action that follows them, and if the camera and proprioception clocks drift, the model is trained to predict actions from state that arrived slightly late. It bakes that lag into the policy, which then overshoots or stutters on real hardware even though offline metrics looked fine. Synchronization is why two datasets with identical content can produce very different policies — one aligns the streams to a single clock and the other does not.
What format should VLA training data be delivered in?
Ask for a format your training stack already reads so you avoid rebuilding ETL for every batch. The two common choices in robot learning are RLDS, which stores each episode as a sequence of steps carrying observations, actions, rewards, discounts, and metadata, and LeRobotDataset, the documented format shipped with the LeRobot library. MCAP is also used for raw multimodal logs. The format matters because a consistent episode structure is what lets policies like π0 combine multi-robot data at all.
What is a VLA data acceptance checklist?
It's the set of pass/fail rules you run on a sample packet before approving a full collection: instruction present and accurate; observations with known camera parameters and stable mounting; proprioception synchronized to the images; action stream in the agreed space, units, and control frequency; coverage of the specified object, environment, and recovery cases; and complete rights, consent, and provenance per trajectory. Reject the batch on any hard failure — a broken action space or missing rights is grounds to stop before the supplier scales the same error.
Can egocentric human video be used to train a VLA?
It can contribute, but it doesn't satisfy the full requirement on its own. First-person human video provides rich manipulation observations and language grounding, and teams use it for pretraining and representation learning. What it lacks is the robot's action and proprioceptive streams in your action space, so it can't directly teach the control mapping a VLA emits. Human video is a useful ingredient for the observation and instruction side; you still need robot demonstrations, or a retargeting pipeline, to supply actions.
Looking for VLA training data requirements?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Post a VLA data spec