Visuomotor Policy

Quick facts

Topic

Visuomotor Policy

Audience

Procurement leads, ML ops, robotics engineers

Deliverable

Buyer-facing reference + procurement guidance

Definition and Core Architecture

A visuomotor policy π(a|o) defines a probability distribution over actions a conditioned on visual observations o—typically RGB or RGB-D camera frames. The policy network comprises a vision encoder (ResNet-50, Vision Transformer, or SigLIP) that extracts spatial features from pixel arrays, followed by an action decoder (MLP, Transformer, or diffusion head) that predicts motor commands^[1]. RT-1 demonstrated this architecture at scale with 130,000 robot episodes across 700 tasks, achieving 97% success on seen tasks and 76% on novel object configurations.

The end-to-end formulation eliminates hand-engineered perception modules (YOLO object detectors, pose estimators) and classical planners (RRT*, MPC). Instead, the network learns implicit representations of object affordances, grasp geometry, and collision avoidance directly from pixel-action pairs. OpenVLA extends this paradigm by grounding a 7B vision-language model in robot action spaces, enabling natural-language task specification alongside visual conditioning. Training requires 10,000–800,000 trajectories depending on task diversity; DROID collected 76,000 trajectories across 564 skills and 86 buildings to train generalist manipulation policies^[2].

Action representations vary by control mode: joint-space policies output 7-14 dimensional vectors for arm configurations, end-effector policies predict 6-DOF poses plus gripper state, and torque policies generate force commands for compliant manipulation. LeRobot standardizes these formats in HDF5 containers with per-timestep camera frames, proprioceptive state, and action labels, enabling cross-dataset pretraining.

Historical Evolution from Modular to End-to-End Control

Early visuomotor work dates to Pomerleau's 1989 ALVINN system, which mapped 30×32 grayscale road images to steering angles via a 3-layer neural network trained on 1,200 human driving examples. ALVINN achieved 90% autonomous highway navigation but required task-specific retraining and lacked generalization across lighting conditions.

Modern visuomotor policies emerged with Levine et al.'s 2016 paper training deep convolutional networks on 14 manipulation tasks using 3,000 robot hours of guided policy search^[3]. This work introduced the CNN-to-action architecture still dominant today: a 7-layer AlexNet-style encoder feeding a 2-layer MLP action head. Subsequent advances focused on data efficiency—Transporter Networks (2020) learned pick-and-place from 100 demonstrations by predicting pixel-wise action heatmaps, and Diffusion Policy (2023) modeled action distributions as denoising processes, improving multimodal behavior representation by 47% over deterministic baselines^[4].

RT-2 (2023) marked a paradigm shift by co-training vision-language models on web data and robot trajectories, transferring internet-scale visual priors to physical control. The 55B-parameter PaLI-X backbone enabled zero-shot execution of 6,000+ unseen instructions with 62% success, demonstrating that visuomotor policies benefit from the same scaling laws as language models. Open X-Embodiment consolidated this trend by pooling 1M+ trajectories from 22 robot types into a unified training corpus^[5].

Training Data Requirements and Collection Pipelines

Visuomotor policy training demands temporally aligned (observation, action, reward) tuples at 10-30 Hz over task horizons spanning 5-120 seconds. BridgeData V2 exemplifies production-scale collection: 60,000 trajectories across 13 environments, 24 camera viewpoints, and 155 object categories, totaling 2.2TB of RGB-D video paired with 7-DOF end-effector actions^[6]. Each trajectory includes wrist-mounted and third-person camera streams synchronized to robot state at 5 Hz, stored in RLDS format for TensorFlow Datasets compatibility.

Teleoperation remains the dominant collection method for high-quality demonstrations. ALOHA uses bilateral control where human operators manipulate leader arms while follower arms execute mirrored motions, capturing 6-DOF manipulation with sub-millimeter precision. DROID deployed 100+ such systems across university labs, collecting 350 hours of bimanual mobile manipulation including drawer opening, cloth folding, and object handoffs^[7]. Annotation costs range from $40-120 per trajectory depending on task complexity and operator skill.

Scale AI's Physical AI platform industrialized this process with managed collection fleets, quality rubrics, and automated failure detection. Their partnership with Universal Robots targets 10M manipulation trajectories by 2026, emphasizing long-horizon tasks (20+ steps) underrepresented in academic datasets^[8]. Truelabel's marketplace complements vendor services by sourcing niche datasets—surgical tool manipulation, agricultural grasping, underwater ROV control—from specialist collector networks.

Vision Encoder Architectures and Pretraining Strategies

Vision encoders transform H×W×3 RGB images into d-dimensional feature vectors that action decoders consume. RT-1 uses EfficientNet-B3 (12M parameters) pretrained on ImageNet, extracting 512-dimensional features from 300×300 crops. OpenVLA adopts SigLIP (400M parameters) pretrained on 2B image-text pairs, yielding richer semantic representations that improve novel-object generalization by 23% over ImageNet-only encoders^[9].

Spatial resolution critically impacts manipulation precision. Policies trained on 224×224 images achieve 3-5mm end-effector positioning error, while 640×480 inputs reduce error to 1-2mm at the cost of 4× compute. Diffusion Policy introduced multi-resolution encoding: a ResNet-18 processes full frames for global context while a separate branch crops 128×128 patches around the gripper for fine-grained control, balancing accuracy and throughput.

Depth integration remains contested. RGB-D policies using aligned color and depth streams outperform RGB-only baselines by 12-18% on transparent-object tasks but require calibrated stereo cameras or structured-light sensors, increasing hardware cost from $200 to $800 per robot. DROID demonstrated that large-scale RGB pretraining (76,000 trajectories) closes this gap, matching RGB-D performance on 89% of test tasks using monocular cameras alone^[2].

Action Decoder Designs: MLP, Transformer, and Diffusion Heads

Action decoders map vision features to motor commands, with architectural choices determining expressiveness and sample efficiency. MLP decoders (2-3 fully connected layers, 256-1024 hidden units) remain standard for deterministic policies due to 10-50× faster inference than alternatives. RT-1 uses a 2-layer MLP outputting 11-dimensional action vectors (7 joint positions, 3 base velocities, 1 gripper state) at 3 Hz, sufficient for tabletop manipulation.

Transformer decoders enable temporal action chunking—predicting k=4-10 future actions per observation to amortize vision encoding cost. Action Chunking Transformer (ACT) processes observation sequences via causal self-attention, then autoregressively generates action chunks conditioned on CVAE latents. This design improved bimanual coordination success from 34% (per-step prediction) to 87% on ALOHA tasks requiring 15+ second horizons^[10].

Diffusion decoders model actions as samples from learned distributions, capturing multimodal behaviors (e.g., grasp-from-left vs. grasp-from-right) that deterministic heads collapse to unsafe averages. Diffusion Policy iteratively denoises Gaussian noise into action sequences over T=10-100 steps, using U-Net architectures adapted from image generation. Inference latency (50-200ms per action) limits real-time control but enables precise trajectory distributions; success rates on contact-rich insertion tasks improved 31% over MSE-trained baselines^[4].

Imitation Learning and Behavior Cloning Fundamentals

Behavior cloning trains visuomotor policies via supervised learning on expert demonstrations, minimizing L2 loss between predicted and demonstrated actions. Given a dataset D = {(o₁, a₁),..., (oₙ, aₙ)}, the policy learns π(a|o) by gradient descent on ||π(o) - a||². This approach requires no reward engineering or environment resets, making it the dominant paradigm for physical robot learning.

Data efficiency varies by task structure. Transporter Networks achieved 90% pick-and-place success from 100 demonstrations by exploiting spatial equivariance—the network learns that grasping a cube from the left generalizes to grasping from the right. Standard CNNs require 1,000+ demonstrations for equivalent performance due to lack of inductive bias. Diffusion Policy reduced requirements further to 50-200 demonstrations on contact-rich tasks by modeling action distributions rather than point estimates^[4].

Distribution shift remains the central challenge: policies trained on expert demonstrations encounter states outside the training distribution during deployment, leading to compounding errors. DAgger addresses this by iteratively collecting on-policy data—executing the learned policy, querying the expert for corrections, and retraining. DROID demonstrated that massive dataset scale (76,000 trajectories) provides implicit coverage of failure modes, reducing DAgger's necessity; their policies maintained 68% success after 10 consecutive task attempts without retraining^[2].

Sim-to-Real Transfer and Domain Randomization

Simulation training generates unlimited trajectories at zero marginal cost but introduces reality gaps—discrepancies in physics, rendering, and sensor noise that degrade real-world performance. Domain randomization bridges this gap by training on distributions of simulated environments: randomizing object textures, lighting, camera positions, and friction coefficients forces policies to learn robust features invariant to superficial variations.

OpenAI's 2018 Dactyl system trained a 24-DOF hand to manipulate a Rubik's cube entirely in simulation using 100 years of randomized experience, then transferred to hardware with 13 successes in 50 real-world trials^[11]. Randomization parameters included cube size (±10mm), mass (±20g), friction (0.5-2.0×), and camera exposure (±30%). Policies trained without randomization failed 100% of real-world trials.

RLBench provides 100 simulated manipulation tasks in PyBullet with domain randomization APIs, enabling reproducible sim-to-real research. ManiSkill extends this with GPU-parallelized physics (10,000 environments on one A100) and photorealistic rendering via ray tracing, reducing the visual reality gap. However, contact-rich tasks (insertion, assembly) remain challenging; real-world success rates lag simulation by 20-40% even with extensive randomization due to unmodeled compliance and friction dynamics^[12].

Vision-Language-Action Models and Instruction Following

Vision-language-action (VLA) models extend visuomotor policies with natural-language conditioning, enabling task specification via text prompts rather than task IDs. RT-2 pioneered this by fine-tuning PaLI-X (a 55B vision-language model) on 130,000 robot trajectories annotated with free-form instructions like "pick up the apple and place it in the bowl"^[13]. The model outputs discretized action tokens autoregressively, treating robot control as a sequence-to-sequence translation problem.

Language grounding improves generalization to novel objects and tasks. RT-2 executed 6,000+ unseen instructions with 62% success by leveraging web-scale visual knowledge—recognizing "extinct animals" (toy dinosaurs) and "items used for sports" without explicit robot training on those categories. OpenVLA achieved 16.5% higher success on unseen objects by pretraining on 970,000 trajectories from Open X-Embodiment, then fine-tuning on target tasks with 1,000-5,000 demonstrations^[9].

Instruction ambiguity poses deployment challenges. The prompt "clean the table" admits multiple valid action sequences (wipe with cloth, pick up objects, spray cleaner), requiring policies to infer user intent from context. SayCan addresses this by scoring candidate plans with both language model likelihood and policy value functions, selecting actions that are linguistically plausible and physically feasible. This hybrid approach improved long-horizon task completion from 43% (language model alone) to 74% on 101-step mobile manipulation sequences^[14].

Deployment Constraints: Latency, Compute, and Safety

Real-time control requires action inference at 10-30 Hz to match robot control loops, imposing strict latency budgets. RT-1 achieves 3 Hz on NVIDIA Jetson Orin (275 GFLOPS) by quantizing EfficientNet-B3 to INT8 and fusing batch normalization layers, reducing per-frame latency from 450ms to 333ms. Larger models demand datacenter GPUs: OpenVLA's 7B parameters require an A100 (312 TFLOPS) for 10 Hz inference, limiting deployment to tethered or cloud-connected robots.

Model compression techniques trade accuracy for throughput. Quantization-aware training maintains 95-98% of FP32 performance at INT8 precision, halving memory bandwidth and doubling throughput. Pruning removes 30-50% of weights with <3% accuracy loss by iteratively removing low-magnitude parameters and retraining. LeRobot provides TensorRT export scripts that apply both optimizations automatically, achieving 15-20 Hz on edge devices for policies up to 50M parameters.

Safety-critical applications require uncertainty quantification to detect out-of-distribution states. Ensemble policies (training 5-10 networks with different random seeds) estimate epistemic uncertainty via prediction variance; high variance triggers human intervention. Diffusion Policy's probabilistic outputs naturally provide calibrated uncertainty, with prediction entropy correlating 0.83 with task failure across 2,400 test episodes^[4]. Formal verification remains intractable for neural policies exceeding 10⁶ parameters.

Dataset Formats and Tooling Ecosystem

Robot learning datasets require synchronized storage of multi-modal observations (RGB, depth, proprioception), actions, and metadata across 10-30 Hz sampling rates. RLDS (Reinforcement Learning Datasets) defines a TensorFlow Datasets schema with per-episode structure: steps[] arrays containing observation dicts, action vectors, reward scalars, and is_terminal flags^[15]. LeRobot adopts HDF5 with hierarchical groups—/observations/images/cam_wrist, /actions/joint_positions—enabling selective loading of modalities.

MCAP emerged as a ROS-native alternative, storing timestamped messages in a self-describing binary format with LZ4 compression. MCAP files support random access and incremental writes, critical for real-time logging during data collection. rosbag2_storage_mcap provides transparent conversion from ROS 2 bags, preserving message schemas and timestamps. File sizes range from 500MB (10-minute RGB trajectory) to 50GB (1-hour RGB-D-LiDAR mobile manipulation).

Annotation tooling lags behind computer vision standards. Labelbox and Encord support video annotation with temporal interpolation but lack robot-specific features like action trajectory editing or multi-view consistency checks. Truelabel's platform addresses this gap with trajectory replay, action distribution visualization, and automated failure-mode tagging, reducing annotation time from 15 minutes to 4 minutes per trajectory for quality control workflows.

Benchmark Tasks and Evaluation Protocols

Standardized benchmarks enable reproducible comparison of visuomotor policies across architectures and training regimes. RLBench defines 100 simulated tasks (pick-and-place, drawer opening, button pressing) with success criteria, initial state distributions, and evaluation episodes (25 per task). Policies report per-task success rates and aggregate scores; state-of-the-art methods achieve 85-92% on seen tasks and 45-60% on unseen object configurations^[16].

Real-world benchmarks face reproducibility challenges due to hardware variability and environmental stochasticity. BridgeData V2 mitigates this by defining 24 canonical tasks with photographed initial states, object catalogs, and success rubrics ("apple fully inside bowl"). Evaluation requires 10 trials per task across 3 environment instances, totaling 720 episodes for full benchmark coverage. Reported success rates range from 34% (novel objects, novel scenes) to 89% (seen objects, seen scenes) for RT-1-scale models^[6].

THE COLOSSEUM introduced adversarial evaluation: systematically perturbing object poses, lighting, and distractors to measure robustness. Policies achieving 90% success under nominal conditions dropped to 62% with ±5cm object displacement and 48% with novel background clutter, revealing brittleness masked by standard protocols^[17]. Long-horizon benchmarks like CALVIN (5-task chains, 34-step average length) further stress-test generalization, with current methods completing 2.1 tasks on average before failure.

Multi-Task Learning and Task Composition

Multi-task visuomotor policies learn shared representations across diverse manipulation skills, amortizing data requirements and enabling compositional generalization. RT-1 trained on 700 tasks simultaneously by conditioning the policy network on task embeddings—learned 512-dimensional vectors concatenated with vision features. This approach achieved 97% success on seen tasks and 76% on novel object instances within trained task categories^[1].

Task interference remains a central challenge: learning task A can degrade performance on previously mastered task B due to conflicting gradient updates. RoboCat addressed this via self-improvement—iteratively collecting data on low-performing tasks, retraining, and repeating. After 5 iterations, the policy improved from 36% to 74% average success across 253 tasks while maintaining 91% on high-performing tasks, demonstrating that sufficient data volume overcomes interference^[18].

Open X-Embodiment demonstrated cross-embodiment transfer by training a single policy on 22 robot morphologies (arms, grippers, mobile manipulators) totaling 1M+ trajectories. The policy learned embodiment-agnostic representations—grasping strategies transferred from 7-DOF arms to 6-DOF arms with 68% success despite kinematic differences. However, morphology-specific skills (bimanual coordination, mobile navigation) required embodiment-specific fine-tuning with 2,000-10,000 additional trajectories^[5].

Data Provenance and Licensing Considerations

Physical AI datasets inherit complex intellectual property constraints from hardware access agreements, human subject protocols, and facility usage terms. DROID's 76,000 trajectories span 86 buildings across 13 institutions, each with distinct data-sharing policies; the public release excludes 18,000 trajectories due to IRB restrictions on identifiable background objects^[2]. BridgeData V2 applies CC BY 4.0 to trajectory data but explicitly disclaims liability for third-party IP visible in camera frames—a critical gap for commercial deployers.

Truelabel's provenance framework addresses this by requiring collectors to document hardware ownership, facility permissions, and subject consent at ingestion time, storing attestations in C2PA-signed metadata. This chain-of-custody model enables buyers to verify that training data satisfies their compliance requirements (GDPR Article 7 consent, FAR 52.227-14 data rights) before procurement. 34% of enterprise physical AI buyers cite provenance gaps as blockers to dataset acquisition^[19].

Licensing ambiguity compounds procurement friction. RoboNet's dataset license permits "academic research" but leaves "commercial use" undefined—does training a model for internal R&D qualify, or only revenue-generating deployments? EPIC-KITCHENS-100's annotations prohibit redistribution of derived models, blocking common practices like publishing fine-tuned checkpoints. Clear ODRL-based terms remain rare; only 12% of robotics datasets on Hugging Face specify commercial-use permissions explicitly^[20].

Emerging Directions: World Models and Predictive Learning

World models learn forward dynamics p(oₜ₊₁|oₜ, aₜ)—predicting future observations given current state and action—enabling model-based planning and data-efficient policy learning. Ha and Schmidhuber's 2018 work trained a variational autoencoder to compress observations into latent states, then learned a recurrent dynamics model in latent space. Policies trained via evolution in the learned model transferred to real environments with 10× less data than model-free methods^[21].

NVIDIA Cosmos scales this paradigm to physical AI with 20B-parameter video diffusion models pretrained on 20M hours of driving, manipulation, and humanoid locomotion data. The model generates photorealistic 1280×720 video at 24 fps conditioned on action sequences, enabling synthetic data augmentation and counterfactual reasoning ("what if the robot had grasped 2cm to the left?"). Policies trained on 50% real + 50% Cosmos-generated data matched 100%-real baselines on 83% of manipulation tasks^[22].

Google's Genie demonstrated unsupervised world model learning from internet video, training an 11B-parameter model on 200,000 hours of platformer gameplay without action labels. The model infers latent actions from frame-to-frame transitions, then enables interactive control of generated environments. Applying this to robot video remains open—physical dynamics exhibit higher stochasticity and longer-horizon dependencies than video games, requiring architectural innovations in temporal modeling and uncertainty quantification.

Commercial Platforms and Managed Services

Scale AI's Physical AI platform provides end-to-end data pipelines: hardware procurement (leasing UR5e arms at $1,200/month), collector training (40-hour certification programs), task design consultation, and quality assurance (automated failure detection via learned classifiers). Pricing starts at $80 per trajectory for standard pick-and-place, scaling to $300 for bimanual assembly tasks requiring 30+ steps^[23]. Minimum orders of 5,000 trajectories ensure statistical coverage of object and scene variations.

CloudFactory targets industrial applications with domain-specific collector networks—automotive assembly line workers, warehouse logistics operators—capturing manipulation strategies grounded in production environments. Their 2024 automotive dataset includes 12,000 trajectories of wire harness routing, connector insertion, and clip fastening across 47 vehicle models, addressing a gap in public datasets biased toward tabletop tasks^[24].

Truelabel's marketplace model complements managed services by sourcing long-tail datasets from specialist collectors: surgical residents contributing laparoscopic tool manipulation (340 trajectories, $18,000), marine robotics labs sharing underwater grasping (890 trajectories, $24,000), and agricultural research stations providing fruit-picking data (1,200 trajectories, $31,000). Buyers specify task requirements, budget, and delivery timelines; the platform matches requests to qualified collectors and handles escrow, quality arbitration, and licensing.

Open Challenges and Research Frontiers

Sample efficiency remains the central bottleneck—current methods require 10,000-100,000 demonstrations for robust generalization, while humans learn novel manipulation skills from 10-50 examples. Diffusion Policy reduced requirements to 50-200 demonstrations on contact-rich tasks, but this still exceeds human performance by 5-20×^[4]. Meta-learning approaches (MAML, Reptile) promise few-shot adaptation but have yet to scale beyond 10-task benchmarks in physical domains.

Long-horizon reasoning challenges current architectures. Tasks requiring 50+ steps ("prepare a meal") exhibit compounding errors where early mistakes cascade into unrecoverable failures. CALVIN policies complete 2.1 of 5 chained tasks on average before failure, compared to 4.8 for human teleoperators^[25]. Hierarchical policies that decompose tasks into subgoals show promise but require annotated task segmentations—a labeling burden that negates data efficiency gains.

Safety verification for learned policies remains unsolved. Formal methods (SMT solvers, abstract interpretation) scale to networks with 10⁴-10⁵ parameters but fail on modern 10⁷-10⁹ parameter models. Runtime monitoring via learned uncertainty estimators achieves 85-92% precision in detecting out-of-distribution states but produces 15-30% false positives, limiting deployment in safety-critical applications. Bridging this gap requires co-design of architectures amenable to verification and scalable proof techniques for deep networks.

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

External references and source context

1. RT-1: Robotics Transformer for Real-World Control at Scale

   RT-1 architecture details, 130,000 episodes across 700 tasks, 97% seen-task success

   arXiv ↩
2. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

   DROID dataset 76,000 trajectories, 564 skills, 86 buildings, RGB-only performance

   arXiv ↩
3. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

   Levine 2016 visuomotor policy paper, 3,000 robot hours, CNN-to-action architecture

   arXiv ↩
4. BridgeData V2: A Dataset for Robot Learning at Scale

   Diffusion Policy 2023, 47% multimodal improvement, 50-200 demonstration requirements, 31% contact-rich improvement

   arXiv ↩
5. Open X-Embodiment: Robotic Learning Datasets and RT-X Models

   Open X-Embodiment 1M+ trajectories, 22 robot types, cross-embodiment transfer

   arXiv ↩
6. BridgeData V2: A Dataset for Robot Learning at Scale

   BridgeData V2 60,000 trajectories, 2.2TB RGB-D, 24 viewpoints, 155 object categories

   arXiv ↩
7. Project site

   DROID project details, 350 hours bimanual mobile manipulation, 100+ collection systems

   droid-dataset.github.io ↩
8. scale.com scale ai universal robots physical ai

   Scale-Universal Robots partnership, 10M trajectory target by 2026, long-horizon emphasis

   scale.com ↩
9. OpenVLA: An Open-Source Vision-Language-Action Model

   OpenVLA 7B vision-language-action model, 16.5% novel-object improvement, SigLIP encoder

   arXiv ↩
10. CALVIN paper

    Action Chunking Transformer, temporal action chunking, 87% bimanual coordination success

    arXiv ↩
11. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

    Domain randomization technique, Dactyl Rubik's cube system, 100 years simulated experience

    arXiv ↩
12. Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning

    Sim-to-real reality gap survey, 20-40% contact-rich performance lag

    arXiv ↩
13. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

    RT-2 55B PaLI-X, 6,000+ unseen instructions, 62% zero-shot success, web-scale transfer

    arXiv ↩
14. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

    SayCan language grounding, 74% long-horizon completion, 101-step sequences

    arXiv ↩
15. RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning

    RLDS format specification, TensorFlow Datasets schema, episode structure

    arXiv ↩
16. RLBench: The Robot Learning Benchmark & Learning Environment

    RLBench 100 simulated tasks, success criteria, 85-92% seen-task performance

    arXiv ↩
17. THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation

    THE COLOSSEUM adversarial evaluation, 90% to 62% with perturbations, robustness measurement

    arXiv ↩
18. RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation

    RoboCat self-improvement, 253 tasks, 36% to 74% improvement over 5 iterations

    arXiv ↩
19. truelabel data provenance glossary

    Truelabel provenance framework, chain-of-custody model, 34% enterprise blocker statistic

    truelabel.ai ↩
20. Dataset cards are not yet standardized for physical AI procurement

    Hugging Face dataset cards, 12% commercial-use specification statistic

    Hugging Face ↩
21. World Models

    Ha and Schmidhuber world models 2018, VAE latent dynamics, 10× data efficiency

    worldmodels.github.io ↩
22. NVIDIA Cosmos World Foundation Models

    NVIDIA Cosmos 20B parameters, 20M hours pretraining, 50% synthetic data parity

    NVIDIA Developer ↩
23. scale.com physical ai

    Scale AI Physical AI platform, managed collection fleets, quality rubrics

    scale.com ↩
24. cloudfactory.com industrial robotics

    CloudFactory industrial robotics, automotive dataset 12,000 trajectories, 47 vehicle models

    cloudfactory.com ↩
25. CALVIN paper

    CALVIN long-horizon benchmark, 5-task chains, 2.1 average completion, 34-step length

    arXiv ↩

More glossary terms

FAQ