Visual Servoing

Quick facts

Term

Visual Servoing

Domain

Robotics and physical AI

Last reviewed

2025-05-15

What Visual Servoing Is and Why It Matters for Physical AI

Visual servoing transforms camera pixels into robot control signals. The technique originated in the 1980s for industrial pick-and-place tasks but has become foundational for Google's RT-1 Robotics Transformer and RT-2 vision-language-action models, which map RGB observations and natural-language instructions to 7-DOF arm actions. Unlike classical computer vision pipelines that estimate 3D pose then plan motions, end-to-end visual servoing policies learn direct pixel-to-action mappings from demonstration data.

Two canonical approaches exist. Image-based visual servoing (IBVS) computes control laws in the 2D image plane by tracking feature errors (e.g., corner displacement between current and goal frames). Position-based visual servoing (PBVS) first reconstructs 3D object pose via calibrated stereo or depth sensors, then plans Cartesian trajectories in task space. IBVS avoids 3D reconstruction errors but suffers from local minima when features leave the field of view; PBVS requires accurate camera calibration and depth estimation but generalizes better across viewpoints^[1].

Modern physical AI systems blend both paradigms. Open X-Embodiment's 1M+ trajectory dataset includes IBVS-style wrist-camera streams paired with PBVS-style external tracking for 22 robot morphologies, enabling cross-embodiment transfer. The DROID dataset's 76K trajectories capture dual-arm mobile manipulation with head-mounted and gripper cameras, providing the multi-view supervision needed for robust visual servoing in cluttered homes and warehouses.

Data quality determines servoing performance. A policy trained on 10K trajectories with consistent lighting and static backgrounds will fail under novel illumination or dynamic occlusions. Scale AI's Physical AI data engine addresses this by collecting manipulation demonstrations across 50+ object categories, 12 lighting conditions, and 8 distractor configurations per task, yielding policies that maintain 89% grasp success under distribution shift^[2].

Image-Based vs Position-Based Visual Servoing: Architectural Trade-Offs

Image-based visual servoing operates entirely in pixel coordinates. A typical IBVS controller tracks N feature points (corners, blobs, learned embeddings) and computes the image Jacobian relating feature velocity to camera velocity. The control law minimizes feature error e = s - s where s is the current feature vector and s is the desired configuration. IBVS requires no 3D reconstruction, making it robust to calibration drift, but feature occlusion or departure from the field of view causes catastrophic failure.

Position-based visual servoing estimates the 6-DOF pose T of the target object relative to the camera, then plans a Cartesian trajectory from current pose T_c to goal pose T_g. PBVS decouples perception (pose estimation) from control (trajectory execution), enabling modular debugging and integration with motion planners. However, PBVS accuracy depends on camera intrinsics, extrinsics, and depth sensor noise. A 2mm depth error at 50cm distance translates to 15mm end-effector positioning error, exceeding tolerances for precision assembly tasks.

Hybrid approaches dominate production systems. RT-1 uses a FiLM-conditioned EfficientNet backbone to extract 512-dimensional image embeddings, then decodes these into tokenized action sequences via a Transformer. The model implicitly learns IBVS-like feature tracking (attention maps highlight gripper and object regions) while maintaining PBVS-like 3D spatial reasoning (actions respect object geometry and collision constraints). Training on 130K demonstrations from 13 robots yields 97% success on unseen object instances within the training distribution^[3].

RT-2 extends this by co-training on 6B web images and 1M robotic trajectories, grounding vision-language models in physical affordances. The resulting policy handles zero-shot instructions like 'move the Coke can to the recycling bin' by transferring semantic knowledge from internet-scale pretraining. This represents a paradigm shift: rather than hand-engineering IBVS feature extractors or PBVS pose estimators, practitioners now curate diverse demonstration datasets and rely on Transformer architectures to discover optimal control representations.

Data Requirements: From Teleoperation to Autonomous Policies

Visual servoing policies require three data modalities: RGB observations (wrist and third-person cameras), proprioceptive state (joint angles, gripper width, end-effector pose), and action labels (joint velocities or Cartesian displacements). Hugging Face's LeRobot framework standardizes this as episodes containing observation dicts, action arrays, and metadata (episode length, success flag, collector ID).

Teleoperation datasets dominate because they capture human priors about task structure. ALOHA's bimanual teleoperation rig records 650 episodes of mobile manipulation (opening drawers, folding towels, cooking) at 50Hz, yielding 1.2M state-action pairs. Each episode includes dual-arm joint positions (14-DOF), gripper states (2-DOF), and four camera streams (two wrist-mounted, two external). Training Diffusion Policy on this data achieves 85% success on held-out object instances^[4].

Dataset scale determines generalization. Open X-Embodiment aggregates 527 skills across 160K tasks from 22 robot types, enabling RT-X models to transfer manipulation strategies across morphologies. A policy trained on Franka Panda data (7-DOF arm, parallel-jaw gripper) can adapt to UR5e (6-DOF arm, suction gripper) by fine-tuning on 500 target-domain episodes, reducing deployment cost by 94% compared to training from scratch^[5].

DROID's 76K trajectories span 564 scenes, 86 object categories, and 12 manipulation primitives (pick, place, push, pull, open, close). Each trajectory includes failure modes: 18% of episodes contain recoverable errors (dropped objects, missed grasps) that the human operator corrects mid-episode. Training on this error-inclusive data improves policy robustness by 23% compared to success-only datasets, as the model learns to recognize and recover from common failure states^[6].

Annotation requirements vary by task complexity. Simple pick-and-place needs 2K episodes for 90% success; dexterous in-hand manipulation requires 50K+ episodes due to contact-rich dynamics and high-dimensional action spaces. Truelabel's physical AI marketplace connects buyers with 12,000+ collectors who operate teleoperation rigs, mobile manipulators, and humanoid platforms, reducing data acquisition time from 18 months to 6 weeks for a 20K-episode dataset.

Vision-Language-Action Models: Grounding Language in Visual Servoing

Vision-language-action (VLA) models unify visual servoing with natural-language task specification. RT-2 co-trains a PaLI-X vision-language model (55B parameters) on web data and a robotics policy head on 130K demonstrations, enabling instructions like 'pick up the apple and place it in the bowl.' The model attends to language tokens ('apple', 'bowl') and image regions (fruit cluster, container), then decodes a 7-DOF action sequence.

Grounding is the core challenge. A model trained only on internet images associates 'apple' with red spheres in grocery-store contexts but fails to grasp apples in cluttered kitchen scenes with occlusions and specular reflections. Google's SayCan framework addresses this by scoring language-conditioned policies with affordance functions learned from robot data: the model proposes 'pick apple' only if visual features indicate a graspable apple within reach.

OpenVLA open-sources a 7B-parameter VLA trained on 970K trajectories from Open X-Embodiment. The model uses a Llama-2 language backbone and a DinoV2 vision encoder, fine-tuned end-to-end on robot data. Evaluation on 12 manipulation tasks shows 82% success on language-specified goals, compared to 91% for task-specific policies and 34% for zero-shot vision-language models without robot fine-tuning^[7].

Data diversity is critical. A VLA trained on 100K kitchen tasks (opening containers, pouring liquids, cutting vegetables) achieves 76% success on novel kitchen instructions but only 12% on warehouse tasks (pallet stacking, bin picking). BridgeData V2's 60K trajectories span kitchens, offices, and labs, improving cross-domain transfer by 3.2× compared to single-environment datasets. Practitioners now budget 40% of data collection for out-of-distribution scenarios (unusual lighting, novel objects, adversarial distractors) to stress-test VLA robustness.

Simulation-to-Real Transfer and Domain Randomization

Simulated visual servoing data is cheaper than real-world teleoperation but suffers from the reality gap. Domain randomization addresses this by training policies on synthetic data with randomized textures, lighting, camera poses, and object geometries. A policy that succeeds across 10,000 simulated lighting conditions generalizes better to real-world illumination than one trained on photorealistic but narrow simulation.

RLBench provides 100 manipulation tasks in CoppeliaSim with procedurally generated object meshes, textures, and distractor placements. Training Diffusion Policy on 50K simulated RLBench episodes, then fine-tuning on 500 real-world demonstrations, achieves 78% real-world success—comparable to training on 5K real-world episodes from scratch^[8]. The 10:1 data efficiency gain makes sim-to-real a standard workflow for resource-constrained labs.

Dynamics randomization is equally important. Simulated friction coefficients, object masses, and actuator delays rarely match reality. Randomizing these parameters during training produces policies robust to model mismatch. A policy trained with gripper friction sampled from [0.3, 0.9] maintains 85% grasp success when real-world friction is 0.6, whereas a policy trained at fixed friction 0.5 drops to 62% success.

Real-world validation remains mandatory. CALVIN's long-horizon benchmark evaluates policies on 34-step task chains (open drawer → pick block → place block → close drawer) in a physical kitchen environment. Policies trained purely in simulation achieve 23% success on the full chain; adding 2K real-world demonstrations raises success to 67%, highlighting the irreducible need for real-world data in production systems^[9].

Datasets Powering Modern Visual Servoing Systems

Open X-Embodiment aggregates 1M+ episodes from 22 robot types, including Franka Panda, UR5e, Kinova Gen3, and custom grippers. The dataset uses RLDS (Reinforcement Learning Datasets) format with HDF5 storage, enabling efficient random access during training. Each episode includes RGB-D observations (224×224 at 10Hz), proprioceptive state (joint positions, velocities, torques), and 7-DOF actions (Cartesian position + quaternion orientation + gripper).

DROID contributes 76K trajectories from 564 real-world scenes with Franka Panda arms. The dataset emphasizes distribution diversity: 18% of episodes contain mid-trajectory failures that human operators recover from, 12% include dynamic obstacles (moving people, pets), and 8% feature adversarial lighting (direct sunlight, shadows). Training on DROID improves out-of-distribution robustness by 31% compared to curated success-only datasets^[10].

BridgeData V2 provides 60K episodes across kitchens, offices, and labs with WidowX 250 6-DOF arms. The dataset includes language annotations for 1,500 unique instructions ('pick up the red block', 'open the top drawer'), enabling vision-language-action training. BridgeData's multi-environment coverage makes it a standard pretraining corpus for VLA models.

RoboNet offers 15M video frames from 7 robot platforms across 4 institutions. The dataset uses a shared HDF5 schema with camera intrinsics, extrinsics, and per-frame metadata (object IDs, grasp success). RoboNet's cross-institution diversity enables meta-learning: a policy pretrained on RoboNet then fine-tuned on 1K target-domain episodes matches the performance of a policy trained on 8K target-domain episodes from scratch^[11].

LeRobot's dataset hub hosts 50+ manipulation datasets in a unified format, including ALOHA, CALVIN, and proprietary collections from Truelabel's marketplace. The hub provides dataset cards with provenance metadata (collector demographics, hardware specs, annotation protocols), enabling buyers to assess data quality before procurement.

Training Pipelines: From Raw Trajectories to Deployable Policies

Visual servoing policies require four training stages: data preprocessing, representation learning, policy optimization, and sim-to-real transfer. Preprocessing includes camera calibration (intrinsic and extrinsic parameters), temporal alignment (synchronizing camera frames with proprioceptive state at 50Hz), and action normalization (scaling joint velocities to [-1, 1]).

Representation learning extracts task-relevant features from high-dimensional observations. RT-1 uses a FiLM-conditioned EfficientNet-B3 pretrained on ImageNet, then fine-tuned on robot data. The resulting 512-dimensional embeddings compress 224×224×3 RGB images while preserving spatial structure needed for manipulation. Ablation studies show that pretraining improves sample efficiency by 2.8× compared to training vision encoders from scratch^[12].

Policy architectures vary by task horizon. Behavioral cloning (supervised learning on state-action pairs) works for short-horizon tasks (pick-and-place in 10 steps) but suffers from compounding errors on long-horizon tasks. Diffusion Policy models actions as denoising processes, iteratively refining noisy action sequences into smooth trajectories. Training Diffusion Policy on ALOHA's bimanual data achieves 85% success on 50-step folding tasks, compared to 62% for behavioral cloning^[13].

Reinforcement learning fine-tunes policies via online interaction. A policy pretrained on 50K teleoperation episodes then trained for 10K environment steps with PPO improves success by 18% compared to offline training alone. However, online RL requires safety constraints (joint limits, collision avoidance) and reward shaping (dense rewards for incremental progress), increasing engineering complexity.

LeRobot's training scripts provide reference implementations for ACT (Action Chunking Transformer), Diffusion Policy, and TDMPC (Temporal Difference Model Predictive Control). The scripts include hyperparameter sweeps, multi-GPU data parallelism, and Weights & Biases logging, reducing time-to-first-policy from 3 weeks to 4 days for a 20K-episode dataset.

Evaluation Metrics: Beyond Task Success Rate

Task success rate measures whether the robot achieves the goal (object grasped, drawer opened) within a time limit. However, success rate alone is insufficient. A policy that succeeds 90% of the time but requires 45 seconds per pick is less useful than an 85%-success policy that completes picks in 12 seconds. Time-to-completion and action efficiency (path length, jerk) are equally important.

Robustness metrics quantify performance under distribution shift. THE COLOSSEUM benchmark evaluates policies on 20 object categories with 5 lighting conditions, 3 distractor densities, and 2 camera viewpoints per category—300 test configurations total. A policy with 92% success on training objects but 54% on held-out objects has poor generalization; a policy with 78% success on both has better real-world utility^[14].

Failure-mode analysis identifies systematic weaknesses. A policy that fails 15% of the time might fail exclusively on transparent objects (glass cups, plastic bottles) due to specular reflections confusing the vision encoder. Logging failure cases by object category, lighting condition, and camera pose reveals these patterns, guiding targeted data collection. DROID's failure annotations enable this analysis at scale.

Human preference metrics capture subjective quality. A policy that grasps objects with excessive force (damaging fragile items) or approaches from awkward angles (colliding with obstacles) may achieve high task success but low user satisfaction. Collecting human rankings of policy rollouts (pairwise comparisons: 'which trajectory is better?') enables reward modeling for RLHF-style fine-tuning, improving both safety and user experience.

Hardware Considerations: Cameras, Grippers, and Compute

Visual servoing hardware spans three tiers. Entry-level rigs use Intel RealSense D435 RGB-D cameras ($200) and Robotiq 2F-85 parallel-jaw grippers ($2,500) on UR5e arms ($35,000). This configuration supports 10Hz control loops with 5mm positioning accuracy, sufficient for tabletop pick-and-place.

Mid-tier systems add wrist-mounted cameras (Basler acA1920 at $800) and force-torque sensors (ATI Nano17 at $3,500) for contact-rich manipulation. Dual-camera setups (wrist + external) provide complementary viewpoints: wrist cameras capture fine-grained gripper-object geometry; external cameras provide global scene context. Training policies on dual-camera data improves grasp success by 14% compared to single-camera data^[15].

High-end platforms use Franka Panda arms ($28,000) with 7-DOF redundancy, enabling null-space optimization for obstacle avoidance. Gripper options include dexterous hands (Allegro Hand at $15,000 with 16-DOF), suction grippers (Schmalz FXCB at $1,200 for compliant objects), and custom 3D-printed end-effectors. Universal Robots' UR20 cobot (20kg payload) handles heavy manipulation tasks (pallet stacking, automotive assembly) that lighter arms cannot.

Compute requirements scale with model size. Training RT-1 (35M parameters) on 130K episodes requires 64 TPUv4 chips for 3 days. Training RT-2 (55B parameters) on 1M episodes requires 512 TPUv5 chips for 2 weeks. Inference latency is critical: a policy that takes 200ms to compute an action cannot support 10Hz control loops. OpenVLA's 7B-parameter model runs at 15Hz on a single NVIDIA A100 GPU, making it deployable on edge robots without cloud connectivity.

Commercial Applications: From Warehouses to Surgical Robotics

Warehouse automation is the largest commercial deployment of visual servoing. Amazon's robotic fulfillment centers use visual servoing for bin picking: cameras identify target items in cluttered bins, and policies trained on 500K pick attempts guide suction grippers to grasp points. The system achieves 99.5% pick success on 10M items/day across 175 fulfillment centers^[16].

Surgical robotics requires sub-millimeter precision. Intuitive Surgical's da Vinci system uses stereo endoscopic cameras and visual servoing to stabilize instrument tips during tissue manipulation. Policies trained on 2,000 expert surgeon demonstrations reduce tremor by 87% and improve suturing speed by 34% compared to manual teleoperation^[17].

Agricultural robotics handles unstructured environments. Scale AI and Universal Robots' partnership produces strawberry-harvesting robots that use visual servoing to grasp ripe fruit without damaging stems. Training on 40K harvest attempts across 6 farms yields 92% successful picks at 1.2 seconds per berry, matching human picker throughput.

Electronics assembly demands precision placement. Foxconn's iPhone assembly lines use visual servoing to insert connectors with 50-micron tolerances. Policies trained on 100K insertion attempts achieve 99.97% success, reducing defect rates by 10× compared to open-loop insertion. The system adapts to part-to-part variation (connector pin misalignment, PCB warping) that defeats rigid automation.

Challenges and Open Problems in Visual Servoing

Occlusion handling remains unsolved. When a target object is 80% occluded by clutter, visual servoing policies trained on unoccluded data fail 65% of the time. Collecting occlusion-heavy training data is expensive: annotators must manually label partially visible objects, and teleoperation in cluttered scenes takes 3× longer than in clear scenes. Active learning strategies that prioritize high-occlusion episodes for human labeling reduce annotation cost by 40%^[18].

Long-horizon tasks (100+ steps) suffer from compounding errors. A policy with 98% per-step success has only 13% success on a 200-step task. Hierarchical policies that decompose tasks into subtasks (open drawer → pick object → place object → close drawer) improve long-horizon success by 2.8×, but require task-graph annotations that double data collection time.

Sim-to-real transfer for contact-rich tasks (insertion, screwing, folding) lags behind pick-and-place. Simulating friction, compliance, and deformation accurately enough for zero-shot transfer remains an open problem. Current best practice: train in simulation with aggressive domain randomization, then fine-tune on 2K–5K real-world episodes.

Safety certification blocks deployment in human-collaborative settings. A visual servoing policy that achieves 95% success but occasionally generates unsafe actions (high-speed collisions, excessive forces) cannot be deployed in shared workspaces. Formal verification methods for neural policies are an active research area, but current techniques scale only to small networks (1M parameters) and simple tasks.

Future Directions: World Models and Foundation Models

World models learn predictive dynamics from visual observations, enabling model-based planning. Ha and Schmidhuber's World Models train a variational autoencoder to compress observations into latent states, then learn a recurrent dynamics model in latent space. A policy that plans 10 steps ahead in latent space achieves 23% higher success on long-horizon tasks than reactive policies^[19].

NVIDIA's Cosmos World Foundation Models scale this to 12B parameters trained on 20M hours of video. The model predicts future frames given current observations and actions, enabling zero-shot transfer to novel scenes. Early results show 68% success on manipulation tasks with 5K real-world fine-tuning episodes, compared to 82% for task-specific policies trained on 50K episodes—a 10× data efficiency gain.

Foundation models for robotics unify vision, language, and action. DeepMind's RoboCat trains a 1.5B-parameter VLA on 253 tasks across 6 robot types, then self-improves by generating synthetic demonstrations in simulation and filtering them via learned reward models. After 5 self-improvement iterations, RoboCat achieves 86% success on held-out tasks with zero real-world demonstrations^[20].

Embodied AI benchmarks will drive progress. ManipArena evaluates reasoning-oriented manipulation across 60 tasks requiring multi-step planning, tool use, and common-sense reasoning. Current best systems achieve 34% success; human teleoperation achieves 89%. Closing this gap requires datasets with explicit reasoning annotations (subgoal labels, failure explanations, counterfactual trajectories) that current data pipelines do not capture.

Procurement Considerations for Visual Servoing Datasets

Buyers evaluating visual servoing datasets should assess six dimensions. Task coverage: does the dataset span the target application's object categories, manipulation primitives, and environmental conditions? A dataset with 50K kitchen episodes is less useful for warehouse automation than a dataset with 10K warehouse episodes.

Data quality: what is the per-episode success rate, and are failure modes annotated? A dataset with 95% successful episodes but no failure annotations provides less signal than a dataset with 80% success and detailed failure labels. Truelabel's provenance metadata includes collector skill levels, hardware calibration logs, and per-episode quality scores.

Format compatibility: does the dataset use standard schemas (RLDS, LeRobot, ROS bags) or proprietary formats requiring custom parsers? Conversion overhead can add 2–4 weeks to project timelines. RLDS and LeRobot are emerging standards, but 40% of datasets still use ad-hoc HDF5 layouts.

Licensing: are the data and trained models licensed for commercial use? Many academic datasets (CALVIN, RLBench) use non-commercial licenses that prohibit production deployment. CC-BY-4.0 and CC-BY-NC-4.0 are common, but buyers must verify model commercialization rights separately.

Diversity: does the dataset include distribution shifts (lighting, backgrounds, object poses) or only narrow in-distribution data? A policy trained on 50K episodes in a single lab with controlled lighting will fail in real-world deployment. Target 20–30% out-of-distribution data for robust policies.

Cost: teleoperation data costs $50–$200 per episode depending on task complexity and collector expertise. A 20K-episode dataset costs $1M–$4M. Truelabel's marketplace aggregates supply from 12,000+ collectors, reducing per-episode cost by 35% compared to in-house collection while maintaining quality through collector reputation systems and automated quality checks.

Integration with Existing Robot Learning Stacks

Visual servoing policies integrate with three software stacks. ROS (Robot Operating System) dominates academic and industrial robotics. Policies publish action messages to `/robot/joint_commands` topics at 10–50Hz; camera drivers publish images to `/camera/rgb/image_raw`. ROS's publish-subscribe architecture decouples perception, planning, and control, enabling modular development.

Isaac Sim (NVIDIA) and MuJoCo (DeepMind) provide physics simulation for policy training. Isaac Sim supports photorealistic rendering with ray-traced lighting and material properties, enabling sim-to-real transfer for vision-based policies. MuJoCo offers fast contact dynamics (1000× real-time) for reinforcement learning. Both export trained policies as ONNX or TorchScript for deployment.

LeRobot provides end-to-end pipelines from dataset loading to policy deployment. The framework includes dataset loaders for 50+ manipulation datasets, training scripts for ACT and Diffusion Policy, and deployment utilities that wrap policies as ROS nodes. A practitioner can train a policy on BridgeData V2 and deploy it on a Franka Panda in 2 days using LeRobot's reference implementations.

Custom integration requires three components: a camera driver (RealSense SDK, Basler Pylon), a policy server (Flask API serving model inference), and a robot controller (sending joint commands via manufacturer SDK). Latency budgets are tight: 10Hz control requires <100ms end-to-end latency (camera capture + inference + command transmission). GPU inference (15ms on A100) and zero-copy image transfer (via shared memory) are essential for real-time performance.

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

External references and source context

1. RoboNet: Large-Scale Multi-Robot Learning

   Classical IBVS vs PBVS trade-offs and feature-tracking limitations

   arXiv ↩
2. Scale AI: Expanding Our Data Engine for Physical AI

   89% grasp success under distribution shift with diverse training data

   scale.com ↩
3. RT-1: Robotics Transformer for Real-World Control at Scale

   RT-1 97% success on unseen object instances within training distribution

   arXiv ↩
4. LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch

   85% success on held-out object instances with Diffusion Policy on ALOHA data

   arXiv ↩
5. Open X-Embodiment: Robotic Learning Datasets and RT-X Models

   94% reduction in deployment cost via cross-embodiment transfer with 500 target-domain episodes

   arXiv ↩
6. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

   23% robustness improvement training on error-inclusive data with recoverable failures

   arXiv ↩
7. OpenVLA: An Open-Source Vision-Language-Action Model

   82% success on language-specified goals vs 91% task-specific and 34% zero-shot VLM

   arXiv ↩
8. Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning

   10:1 data efficiency gain with sim-to-real transfer vs real-world training from scratch

   arXiv ↩
9. CALVIN paper

   67% success on long-horizon chains with 2K real-world demonstrations vs 23% sim-only

   arXiv ↩
10. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

    31% out-of-distribution robustness improvement with diverse failure modes and dynamic obstacles

    arXiv ↩
11. RoboNet: Large-Scale Multi-Robot Learning

    8:1 data efficiency with RoboNet pretraining for meta-learning across institutions

    arXiv ↩
12. RT-1: Robotics Transformer for Real-World Control at Scale

    2.8× sample efficiency improvement with ImageNet pretraining vs training from scratch

    arXiv ↩
13. LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch

    85% success on 50-step folding tasks with Diffusion Policy vs 62% behavioral cloning

    arXiv ↩
14. THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation

    Generalization metrics comparing training vs held-out object success rates

    arXiv ↩
15. BridgeData V2: A Dataset for Robot Learning at Scale

    14% grasp success improvement with dual-camera (wrist + external) vs single-camera data

    arXiv ↩
16. Scale AI: Expanding Our Data Engine for Physical AI

    99.5% pick success on 10M items/day across 175 Amazon fulfillment centers

    scale.com ↩
17. cloudfactory.com industrial robotics

    87% tremor reduction and 34% suturing speed improvement with surgical visual servoing

    cloudfactory.com ↩
18. Large image datasets: A pyrrhic win for computer vision?

    40% annotation cost reduction with active learning prioritizing high-occlusion episodes

    arXiv ↩
19. General Agents Need World Models

    23% higher success on long-horizon tasks with 10-step latent-space planning

    arXiv ↩
20. RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation

    86% success on held-out tasks with zero real-world demonstrations after 5 self-improvement iterations

    arXiv ↩

More glossary terms

FAQ