VLA (Vision-Language-Action Model)

Quick facts

Term

VLA (Vision-Language-Action Model)

Domain

Robotics and physical AI

Last reviewed

2025-06-15

Architecture: Vision Encoder, Language Backbone, Action Head

VLA models decompose into three stages. The vision encoder converts RGB images (often 224×224 or 320×256 resolution) into a grid of visual tokens; OpenVLA uses a SigLIP ViT-SO400M encoder producing 729 tokens per frame^[1]. The language backbone—typically a decoder-only transformer like Llama-2 7B or PaLM 8B—processes the concatenated sequence of visual tokens, instruction tokens, and proprioceptive state tokens. The action head maps the backbone's final hidden states to continuous control outputs: 7-DoF end-effector deltas (x, y, z, roll, pitch, yaw, gripper) at 10–50 Hz.

Action tokenization strategies vary. RT-1 discretizes each action dimension into 256 bins and treats actions as language tokens, enabling direct next-token prediction^[2]. RT-2 extends this by fine-tuning PaLI-X (a 55B vision-language model) on robot trajectories, inheriting web-scale visual priors^[3]. OpenVLA takes a different path: it keeps actions continuous and adds a lightweight MLP head, preserving the language model's generative structure while avoiding quantization error.

The key architectural insight is co-training: the language backbone never freezes. During robot fine-tuning, gradients flow back through the entire vision-language stack, allowing the model to adapt its semantic representations ("grasp," "place," "avoid") to the statistics of physical interaction. This differs from older imitation learning pipelines that froze a pretrained vision encoder and trained only a behavior-cloning head.

Training Data Requirements: Trajectories, Diversity, Scale

VLA training demands two dataset types. Pretraining uses internet-scale vision-language pairs—billions of image-caption examples (LAION-5B, DataComp) or video-text pairs (WebVid, HowTo100M)—to build a visual-semantic prior. Fine-tuning requires robot demonstration trajectories: sequences of (observation, action, language_instruction) tuples collected via teleoperation, scripted policies, or human video.

Open X-Embodiment aggregates 970,000 trajectories across 22 robot embodiments, spanning 160,000 tasks^[4]. RT-X models trained on this dataset achieve 50% higher success rates on unseen tasks than single-robot policies. DROID contributes 76,000 trajectories from 564 scenes and 86 buildings, emphasizing long-horizon mobile manipulation^[5]. BridgeData V2 provides 60,000 trajectories in kitchen environments with fine-grained language annotations^[6].

Diversity matters more than raw scale. A VLA trained on 100,000 trajectories spanning 50 object categories and 20 task families outperforms a model trained on 500,000 trajectories of a single pick-place task. Embodiment diversity is equally critical: RoboNet showed that multi-robot pretraining (7 platforms, 113 environments) improves few-shot adaptation by 40% over single-robot baselines^[7].

Data format standards remain fragmented. RLDS (Reinforcement Learning Datasets) wraps trajectories in TensorFlow Episodes; LeRobot uses Parquet + MP4 for Hugging Face distribution^[8]. MCAP is emerging for ROS2 ecosystems. Buyers must budget conversion effort or specify format requirements upfront.

Pretraining Transfer: Why Internet Data Accelerates Robot Learning

The central hypothesis of VLA research is that vision-language pretraining provides a compressed world model. A model pretrained on LAION-2B has seen millions of images of kitchens, tools, and human hands; it already encodes priors about object affordances, spatial layouts, and action semantics. Fine-tuning on 10,000 robot trajectories then grounds these priors in motor commands, rather than learning visual representations from scratch.

Empirical evidence supports this. RT-2 fine-tuned PaLI-X (pretrained on 10B image-text pairs) and achieved 62% success on unseen objects versus 32% for RT-1 (trained only on robot data)^[3]. OpenVLA fine-tuned Llama-2 7B on 970,000 Open X-Embodiment trajectories and matched or exceeded RT-2-X performance at 1/8 the parameter count^[1].

The transfer mechanism is semantic grounding. When a VLA sees a novel object (e.g., a translucent bottle), the vision encoder maps it to a region of embedding space near "bottle," "container," "transparent"—concepts learned from internet images. The language model then retrieves manipulation strategies associated with those concepts ("grasp cylindrical objects with parallel gripper") without requiring explicit bottle-specific demonstrations.

Limitations exist. Pretraining on static images does not teach dynamics, contact physics, or force control. Scale AI's Physical AI initiative emphasizes video pretraining (ego4d, Kinetics-700) to capture temporal structure^[9]. NVIDIA's Cosmos models pretrain on 20M hours of driving and manipulation video, learning predictive world models before action fine-tuning^[10].

Action Tokenization vs. Continuous Regression

VLA architectures diverge on how to represent actions. Discrete tokenization (RT-1, RT-2) bins each action dimension into N buckets (typically 256) and treats actions as vocabulary tokens. The language model predicts action sequences via standard next-token sampling. Benefits: unified training objective (cross-entropy loss), easy integration with pretrained LLMs, natural handling of multi-modal action distributions. Drawbacks: quantization error (±0.5 cm positional noise), difficulty representing high-frequency control (50 Hz × 7 DoF = 350 tokens/sec).

Continuous regression (OpenVLA, Octo) adds a lightweight MLP head that maps the language model's hidden states to real-valued action vectors. Benefits: no quantization loss, lower token budget, simpler deployment (direct servo commands). Drawbacks: requires custom loss functions (MSE, Huber), cannot leverage LLM's generative priors as directly.

Hybrid approaches are emerging. LeRobot's Diffusion Policy uses a diffusion model as the action head, iteratively denoising a Gaussian prior into a smooth action trajectory^[11]. This combines the expressiveness of continuous actions with the multi-modality of generative models. Training cost is higher (50–100 denoising steps per forward pass) but success rates improve 10–15% on contact-rich tasks.

The choice depends on task horizon and control frequency. For high-level planning ("navigate to the kitchen, then pick up the red mug"), discrete tokens suffice. For dexterous manipulation ("insert USB cable"), continuous regression or diffusion is necessary.

Embodiment Generalization: Single Model, Multiple Robots

A core VLA promise is embodiment transfer: train once on diverse robots, deploy anywhere. Open X-Embodiment demonstrated this by training RT-X on 22 robot types (UR5, Franka, Fetch, Stretch) and achieving 30–50% success on 6 held-out embodiments^[4]. The model learns embodiment-agnostic representations of tasks ("pick up object") while the action head adapts to each robot's kinematics.

Embodiment conditioning is critical. RT-X prepends a learnable embodiment token to the input sequence; the language model uses this token to modulate action predictions. OpenVLA extends this with embodiment-specific LoRA adapters: 8M-parameter low-rank matrices that specialize the 7B backbone for each robot without full fine-tuning^[1].

Limitations remain. Cross-embodiment transfer works best when robots share similar action spaces (all 7-DoF arms with parallel grippers). Transferring from a 2-finger gripper to a 5-finger dexterous hand requires additional data. DROID addresses this by collecting trajectories with consistent end-effector parameterizations (SE(3) pose + 1D gripper) across platforms^[5].

The economic implication: a single VLA trained on 1M diverse trajectories can replace 50 task-specific policies, amortizing data collection cost. Truelabel's marketplace enables buyers to pool procurement budgets and share embodiment-diverse datasets^[12].

Language Conditioning: Instructions, Corrections, Hierarchical Goals

VLAs accept natural language instructions at multiple granularities. Task-level instructions ("pick up the red block and place it in the bin") specify the goal. Step-level corrections ("no, the other bin") provide real-time feedback. Hierarchical goals ("clean the table: first remove trash, then wipe surface") decompose long-horizon tasks.

RT-2 showed that language conditioning enables zero-shot generalization to novel object attributes^[3]. A model trained on "pick up the red cup" succeeds on "pick up the blue cup" without blue-cup demonstrations, because the vision-language backbone already understands color semantics. Success rate: 78% on unseen colors vs. 12% for non-language baselines.

Language also enables chain-of-thought planning. RT-2 can be prompted with "To clean the table, I should first pick up the trash, then wipe the surface. Step 1:" and will execute the implied sequence. This emergent capability arises from pretraining on internet text containing procedural instructions (WikiHow, recipe sites).

Data annotation requirements are high. Each trajectory needs a natural language description; BridgeData V2 employs human annotators to write free-form instructions for 60,000 trajectories^[6]. Cheaper alternatives: template-based generation ("pick up the {color} {object}"), automatic speech recognition of teleoperator narration, or LLM-based captioning of robot videos.

Sim-to-Real Transfer: Synthetic Data for VLA Pretraining

Simulation offers infinite data at zero marginal cost, but sim-to-real gap remains a challenge. VLAs pretrained purely on synthetic data (Isaac Sim, MuJoCo, PyBullet) achieve 20–40% real-world success without domain randomization. Adding photorealistic rendering (ray tracing, PBR materials) improves this to 50–60%.

NVIDIA Cosmos introduces world foundation models: diffusion transformers pretrained on 20M hours of real video, then fine-tuned to generate synthetic robot trajectories^[10]. The model learns physical priors (gravity, contact, occlusion) from real data, then applies them to simulated scenes. VLAs trained on Cosmos-generated data achieve 85% real-world success—approaching real-data performance.

Domain randomization remains essential. Randomizing lighting, textures, camera poses, and object shapes forces the VLA to learn invariant features. Tobin et al. (2017) showed that extreme randomization (1000+ texture variations per object) enables zero-shot sim-to-real transfer for vision-based grasping^[13].

The data procurement implication: buyers can bootstrap VLA training with 100,000 synthetic trajectories (cost: $5K–$10K for simulation engineering), then fine-tune on 5,000–10,000 real trajectories (cost: $50K–$100K for teleoperation). This 10:1 synthetic:real ratio is becoming standard practice.

Teleoperation Data Collection: Quality, Diversity, Cost

Teleoperation is the dominant method for collecting high-quality robot demonstrations. A human operator controls the robot via a leader device (3D mouse, VR controller, or replica arm) while cameras record observations and the robot logs actions. DROID collected 76,000 trajectories using a custom bilateral teleoperation rig with force feedback^[5].

Quality metrics matter. Success rate (did the trajectory achieve the goal?) should exceed 90%; failed demonstrations teach the VLA incorrect behaviors. Smoothness (low jerk, no oscillations) improves policy stability. Diversity (varied object poses, lighting, backgrounds) prevents overfitting. BridgeData V2 enforces diversity by randomizing 15 scene variables per trajectory^[6].

Cost structure: $20–$50 per trajectory for simple pick-place tasks (5–10 seconds, single object), $100–$200 for long-horizon tasks (60+ seconds, multi-step), $500+ for dexterous manipulation (in-hand reorientation, cable routing). Truelabel's collector network offers per-trajectory pricing with quality guarantees: 95% success rate, <5% retake rate, delivery in 2–4 weeks^[12].

Scaling teleoperation requires infrastructure. Scale AI's data engine operates 50+ teleoperation stations with trained operators, delivering 10,000 trajectories/month^[9]. For buyers without in-house robotics labs, outsourcing to specialized vendors is the only viable path to 100K+ trajectory datasets.

Evaluation Benchmarks: Success Rate, Generalization, Robustness

VLA evaluation lacks standardization. Most papers report success rate on held-out test tasks (50–100 episodes per task, binary success/failure). RT-X reports 67% average success across 6 unseen embodiments^[4]. OpenVLA achieves 73% on the same benchmark^[1].

Generalization axes include: unseen objects (novel shapes, textures), unseen scenes (new backgrounds, lighting), unseen instructions (paraphrased or compositional language), unseen embodiments (different robot morphologies). A robust VLA should maintain >60% success across all axes. Few published models meet this bar.

Robustness metrics are emerging. THE COLOSSEUM benchmark measures performance under adversarial perturbations: random object displacements, camera occlusions, dynamic obstacles^[14]. Top VLAs degrade from 80% success (clean) to 45% (adversarial). ManipArena evaluates long-horizon reasoning with 100-step tasks requiring tool use and multi-object coordination^[15].

Buyers should demand benchmark results on tasks similar to their deployment scenario. A VLA with 90% success on tabletop pick-place may achieve only 30% on warehouse bin-picking due to clutter, occlusion, and scale differences.

Data Licensing and Provenance for VLA Training

VLA training datasets carry complex licensing. Open X-Embodiment aggregates 60+ datasets, each with different terms: some allow commercial use (MIT, Apache 2.0), others restrict to research (CC BY-NC)^[4]. A VLA trained on mixed-license data inherits the most restrictive terms—often prohibiting commercial deployment.

Provenance tracking is critical for compliance^[16]. Every trajectory must link to: collector identity, collection date, robot platform, scene metadata, and license grant. Truelabel's marketplace enforces structured provenance via PROV-O metadata and cryptographic signatures^[12].

EU AI Act Article 53 requires "sufficiently detailed" training data documentation for high-risk AI systems (which includes autonomous robots)^[17]. Buyers must maintain: dataset composition reports, bias audits, data-sheet-style documentation. Gebru et al.'s Datasheets for Datasets framework is becoming the de facto standard^[18].

The procurement implication: budget 10–15% of dataset cost for legal review and provenance tooling. A $200K trajectory purchase may require $20K–$30K in licensing diligence, metadata engineering, and audit trail infrastructure.

VLA vs. Diffusion Policy vs. Behavior Transformers

VLAs compete with alternative policy architectures. Diffusion policies (Chi et al., 2023) model actions as iterative denoising processes, achieving state-of-the-art performance on contact-rich tasks (peg insertion, cable routing). LeRobot's implementation shows 15% higher success than VLAs on dexterous manipulation^[11]. Drawback: 10× slower inference (100ms vs. 10ms per action).

Behavior Transformers (Shafiullah et al., 2022) use GPT-style autoregressive models over discretized action sequences, without vision-language pretraining. They excel at long-horizon tasks (100+ steps) but require 10× more robot data than VLAs to reach comparable performance.

Implicit policies (Florence et al., 2022) learn energy-based models that score action candidates rather than directly predicting actions. They handle multi-modal action distributions (e.g., grasp from left or right) better than VLAs but require expensive sampling at inference.

The choice depends on task requirements. For semantic generalization ("pick up the red object" → "pick up the blue object"), VLAs dominate. For contact-rich precision (0.1mm insertion tolerance), diffusion policies win. For long-horizon planning (20-step assembly), behavior transformers are strongest. Many practitioners train all three and ensemble predictions.

Future Directions: Multimodal Sensing, Tactile, Proprioception

Current VLAs rely almost exclusively on RGB cameras. Next-generation models will integrate depth (structured light, stereo, LiDAR), tactile (GelSight, BioTac), force-torque, and proprioceptive (joint angles, velocities) streams. DROID includes wrist-mounted depth cameras and demonstrates 20% higher success on transparent-object tasks^[5].

Tactile sensing is critical for contact-rich manipulation. A VLA trained on RGB alone cannot distinguish "gripper closed on object" from "gripper closed on air." Adding binary contact sensors improves grasping success from 60% to 85%. High-resolution tactile arrays (16×16 taxels) enable in-hand manipulation and slip detection.

Proprioceptive history (last 10 timesteps of joint states) provides implicit force feedback. When the robot encounters unexpected resistance (e.g., drawer is stuck), joint velocities deviate from commanded values. A VLA conditioned on proprioceptive history learns to detect and recover from such failures.

Data format implications: multimodal trajectories require synchronized streams. MCAP is emerging as the standard container format, supporting arbitrary message schemas and nanosecond timestamps^[19]. LeRobot is adding MCAP export to enable cross-platform multimodal datasets^[8].

Commercial VLA Deployments: Warehouses, Kitchens, Assembly

VLAs are moving from research to production. Covariant (now part of Amazon) deployed VLAs in 20+ warehouses for bin-picking and package sorting, processing 1M+ picks/day. Physical Intelligence raised $400M in 2024 to build general-purpose VLAs for household tasks; their π₀ model handles laundry folding, dishwasher loading, and table clearing.

Figure AI partnered with Brookfield Asset Management to collect 10M+ hours of humanoid teleoperation data for VLA pretraining^[20]. Target deployment: manufacturing assembly lines (automotive, electronics). NVIDIA GR00T is a VLA foundation model for humanoid robots, pretrained on 1M+ trajectories and fine-tunable for customer-specific tasks^[21].

Deployment challenges include: safety certification (VLAs are black-box models, hard to verify), failure recovery (VLAs rarely signal uncertainty), data drift (performance degrades as real-world distribution shifts from training data). Most commercial deployments use VLAs for high-level planning and switch to classical controllers for safety-critical motions.

The market implication: demand for VLA training data is growing 300% year-over-year. Truelabel's marketplace saw 12,000 trajectory purchases in 2024, up from 3,000 in 2023^[12].

Data Procurement Strategy: Build, Buy, or Partner

Organizations face three paths to VLA training data. Build: deploy internal teleoperation infrastructure, hire operators, collect 50K–100K trajectories over 12–18 months. Cost: $500K–$2M (hardware, labor, facilities). Advantage: full control over task distribution and data rights. Disadvantage: slow time-to-data, high fixed cost.

Buy: purchase existing datasets from vendors or marketplaces. Truelabel offers 800+ robotics datasets (12M+ trajectories) with per-trajectory pricing and commercial licenses^[12]. Scale AI provides custom data collection (minimum $100K order)^[9]. Cost: $10–$50 per trajectory for standard tasks, $100–$500 for specialized tasks. Advantage: fast (2–4 week delivery), pay-as-you-go. Disadvantage: less control over task distribution.

Partner: join a data consortium (Open X-Embodiment, LeRobot community) and contribute + access pooled datasets. Cost: engineering time to standardize data formats and contribute trajectories. Advantage: access to 1M+ trajectories at near-zero marginal cost. Disadvantage: data is typically research-only licensed, limiting commercial use.

Hybrid strategies are common: bootstrap with 100K purchased trajectories, fine-tune on 10K custom-collected trajectories for deployment-specific tasks. This balances speed, cost, and control.

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

External references and source context

1. OpenVLA: An Open-Source Vision-Language-Action Model

   OpenVLA achieves 73% success on Open X-Embodiment benchmark with embodiment-specific LoRA adapters

   arXiv ↩
2. RT-1: Robotics Transformer for Real-World Control at Scale

   RT-1 architecture and action tokenization strategy

   arXiv ↩
3. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

   RT-2 achieves 62% success on unseen objects vs 32% for RT-1; 78% success on unseen colors with language conditioning

   arXiv ↩
4. Open X-Embodiment: Robotic Learning Datasets and RT-X Models

   RT-X models achieve 50% higher success on unseen tasks; 67% average success across 6 unseen embodiments; mixed licensing across 60+ datasets

   arXiv ↩
5. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

   DROID uses bilateral teleoperation with force feedback; consistent SE(3) end-effector parameterization; wrist depth cameras improve transparent-object handling by 20%

   arXiv ↩
6. BridgeData V2: A Dataset for Robot Learning at Scale

   BridgeData V2 randomizes 15 scene variables per trajectory; human annotators write free-form instructions

   arXiv ↩
7. RoboNet: Large-Scale Multi-Robot Learning

   RoboNet demonstrates multi-robot learning benefits

   arXiv ↩
8. LeRobot documentation

   LeRobot documentation on dataset formats and MCAP export plans

   Hugging Face ↩
9. Scale AI: Expanding Our Data Engine for Physical AI

   Scale AI emphasizes video pretraining for temporal structure; operates 50+ teleoperation stations delivering 10,000 trajectories/month

   scale.com ↩
10. NVIDIA Cosmos World Foundation Models

    NVIDIA Cosmos world foundation models pretrain on 20M hours of video; VLAs trained on Cosmos-generated data achieve 85% real-world success

    NVIDIA Developer ↩
11. LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch

    LeRobot's Diffusion Policy shows 15% higher success than VLAs on dexterous manipulation

    arXiv ↩
12. truelabel physical AI data marketplace bounty intake

    Truelabel marketplace offers 800+ robotics datasets with 12M+ trajectories; per-trajectory pricing; 95% success rate guarantee; 12,000 trajectory purchases in 2024 vs 3,000 in 2023

    truelabel.ai ↩
13. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

    Extreme randomization (1000+ texture variations) enables zero-shot sim-to-real transfer for vision-based grasping

    arXiv ↩
14. THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation

    Top VLAs degrade from 80% success (clean) to 45% (adversarial) on COLOSSEUM

    arXiv ↩
15. ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation

    ManipArena benchmark for tool use and multi-object coordination

    arXiv ↩
16. truelabel data provenance glossary

    Truelabel provenance tracking enforces structured metadata via PROV-O and cryptographic signatures

    truelabel.ai ↩
17. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence

    EU AI Act Article 53 requires sufficiently detailed training data documentation for high-risk AI systems

    EUR-Lex ↩
18. Datasheets for Datasets

    Datasheets for Datasets is becoming the de facto standard for dataset documentation

    arXiv ↩
19. MCAP guides

    MCAP supports arbitrary message schemas and nanosecond timestamps for synchronized multimodal streams

    MCAP ↩
20. Figure + Brookfield humanoid pretraining dataset partnership

    Figure AI partnered with Brookfield to collect 10M+ hours of humanoid teleoperation data

    figure.ai ↩
21. NVIDIA GR00T N1 technical report

    NVIDIA GR00T is a VLA foundation model for humanoid robots pretrained on 1M+ trajectories

    arXiv ↩

More glossary terms

FAQ