How to Collect Multimodal Robot Data for Vision-Language-Action Models

Quick facts

Difficulty

Intermediate

Audience

Physical AI data engineers

Last reviewed

2026-01-15

Why Multimodal Data Powers Vision-Language-Action Models

Vision-language-action models like RT-2 and OpenVLA learn manipulation policies by grounding natural language instructions in visual observations and proprioceptive feedback. Single-modality datasets — RGB-only or joint-state-only — fail to capture the cross-modal correlations that enable zero-shot generalization. Open X-Embodiment aggregated 527 skills across 22 robot embodiments by standardizing multimodal trajectories into a common RLDS schema^[1].

The DROID dataset collected 76,000 trajectories from 564 scenes using synchronized wrist and third-person RGB cameras, 7-DoF proprioception, and 6-axis force-torque sensors^[2]. This multimodal richness enabled OpenVLA to achieve 34% higher success rates on unseen tasks compared to RGB-only baselines^[3]. Force-torque data proved critical for contact-rich tasks like insertion and wiping, where visual feedback alone cannot disambiguate success from failure.

Language annotations transform teleoperation logs into instruction-following datasets. RT-1 paired 130,000 demonstrations with 700 natural language instructions, enabling a single policy to execute diverse kitchen tasks^[4]. Without language grounding, models collapse to nearest-neighbor imitation and fail to generalize across semantic task variations. The LeRobot framework now standardizes language-conditioned episode schemas, reducing integration friction for buyers sourcing multimodal data from multiple collectors.

Sensor Selection and Synchronization Architecture

RGB cameras are the primary visual modality. Wrist-mounted cameras (80-120° FOV) capture end-effector context; third-person cameras (60-90° FOV) provide scene layout. BridgeData V2 used dual Intel RealSense D435 cameras at 640×480 resolution and 15 Hz, synchronized via hardware triggers^[5]. USB bandwidth limits prevent running multiple RealSense units at full resolution on a single host — use separate USB controllers or Ethernet-based cameras like Basler or FLIR for 3+ camera rigs.

Depth sensors enable 3D reasoning for grasping and navigation. Time-of-flight cameras (RealSense D455, Azure Kinect) provide aligned RGB-D at 30 Hz but suffer from reflective surfaces and sunlight interference. Stereo depth (ZED 2, RealSense D435) works outdoors but requires per-scene calibration. DROID recorded depth at 10 Hz to match the control loop frequency, downsampling from the camera's native 30 Hz to reduce storage overhead^[2].

Force-torque sensors (ATI Nano17, Robotiq FT 300) measure contact forces at 100-500 Hz. Mount sensors between the robot flange and end-effector, not at the wrist joint, to avoid coupling with joint friction. DROID used 6-axis force-torque at 125 Hz, low-pass filtered at 20 Hz to remove vibration noise^[2]. Without force data, models cannot learn compliant behaviors like sliding a drawer or pressing a button with controlled force.

Time synchronization is non-negotiable. Hardware triggers (GPIO, external sync box) guarantee sub-millisecond alignment but require custom electronics. MCAP and ROS 2 bags store per-message timestamps; post-hoc alignment via ApproximateTimeSynchronizer introduces 10-50 ms jitter that corrupts action labels. Open X-Embodiment required all contributors to timestamp-align modalities within 5 ms to ensure cross-dataset compatibility^[1].

Defining Modality Specifications for Target VLA Models

Different VLA architectures consume different tensor shapes, coordinate frames, and normalization schemes. Mismatches between your dataset schema and the target model's dataloader are the leading cause of wasted collection effort. RT-2 expects a single 320×256 RGB image (uint8, RGB channel order) and a tokenized language instruction (T5-XXL embeddings)^[6]. Octo requires a primary RGB image at 256×256, an optional wrist image at 128×128, and proprioceptive state as a 7-DoF end-effector pose (position + quaternion)^[7].

Diffusion Policy expects 2-3 camera views at 96×96 to 256×256 resolution, joint-space proprioception (7-14 DoF depending on arm), and joint-space actions^[8]. OpenVLA uses a single 224×224 RGB image, language instruction, and 7-DoF delta end-effector actions in the base frame^[3]. Create a modality specification table before collection: modality name, tensor shape, dtype, units, coordinate frame, sampling rate, and which models require it. For example: `rgb_primary: (256,256,3) uint8 BGR, 10 Hz, base frame, required by RT-2/Octo/OpenVLA`.

Coordinate frame consistency prevents silent training failures. End-effector poses must reference a stable base frame (robot base link or world origin), not the camera frame. Joint angles must follow the URDF joint order. RLDS enforces a `steps` schema where `observation` and `action` tensors share a common timestamp, but does not validate coordinate frames — that responsibility falls on the data collector^[9].

Teleoperation Hardware and Task Design

Teleoperation interfaces determine data quality and collection throughput. ALOHA uses leader-follower arms for bimanual tasks, achieving 20-30 demonstrations per hour for kitchen assembly tasks^[10]. DROID deployed a custom 6-DoF SpaceMouse interface with force feedback, enabling 350 trajectories per day across 10 operators^[2]. VR controllers (Meta Quest, HTC Vive) provide intuitive 6-DoF control but introduce latency (30-50 ms) that degrades fine manipulation quality.

Task diversity drives generalization. BridgeData V2 collected 60,000 trajectories across 24 tasks (pick, place, open, close, wipe) in 13 kitchen environments, varying object poses, lighting, and distractor objects^[5]. Open X-Embodiment required contributors to provide ≥3 environment variations per task and ≥5 object variations per manipulation primitive^[1]. Single-scene datasets produce policies that overfit to background textures and fail in novel environments.

Language annotation workflows must scale with trajectory volume. RT-1 used a two-stage process: operators recorded free-form descriptions during teleoperation, then annotators post-hoc normalized them into 700 canonical instructions^[4]. LeRobot provides a web UI for bulk annotation, supporting template-based instructions (`pick up the {object}`) and free-form descriptions. Aim for 3-5 language variations per task to prevent models from memorizing exact phrasings.

Recording Pipelines: ROS Bags, MCAP, and HDF5

ROS 2 bags are the de facto standard for real-time multimodal recording. The `rosbag2` CLI records all topics to an SQLite3 database with per-message timestamps, supporting playback and topic filtering^[11]. MCAP is a columnar container format designed for robotics logs, offering 3-5× faster read performance than ROS bags and native support for Protobuf, JSON, and custom schemas^[12]. rosbag2_storage_mcap enables transparent MCAP recording via ROS 2 without code changes.

HDF5 provides hierarchical storage for post-processed datasets. HDF5 groups organize episodes as `/episode_000001/observations/rgb_primary`, `/episode_000001/actions`, etc., with chunked compression (gzip level 4) reducing storage by 60-80% for image data^[13]. LeRobot datasets use HDF5 for local storage and convert to Parquet for Hugging Face Hub uploads, balancing random-access performance with cloud compatibility^[14].

Parquet is the preferred format for large-scale dataset distribution. Apache Parquet stores tabular data in columnar format with per-column compression, enabling efficient filtering and aggregation^[15]. Hugging Face Datasets natively supports Parquet, allowing streaming access to multi-terabyte datasets without downloading entire files^[16]. Open X-Embodiment distributes 1 million trajectories as sharded Parquet files, each containing 10,000 episodes with embedded image bytes.

Converting Raw Logs to RLDS and LeRobot Formats

RLDS (Reinforcement Learning Datasets) is a TensorFlow Datasets schema for episodic RL data, used by RT-1, RT-2, and Open X-Embodiment^[9]. Each episode is a sequence of `steps`, where each step contains `observation` (dict of tensors), `action` (tensor), `reward` (float), `is_terminal` (bool), and `is_first` (bool). Images are stored as JPEG-encoded bytes to reduce storage; proprioception and actions are float32 arrays.

The conversion pipeline: (1) parse ROS bag or MCAP file, (2) extract synchronized frames via timestamp matching, (3) encode images to JPEG at quality 95, (4) compute actions as the difference between consecutive proprioceptive states, (5) write to TFRecord shards with 100-500 episodes per shard. RLDS GitHub provides reference converters for RoboNet and BridgeData formats^[17].

LeRobot datasets use a flat Parquet schema with columns `episode_index`, `frame_index`, `timestamp`, `observation.image`, `observation.state`, `action`, `next.reward`, `next.done`^[14]. Images are stored as PNG bytes (lossless) or JPEG bytes (lossy). The LeRobot CLI provides `lerobot-convert` to transform ROS bags, RLDS, and custom HDF5 formats into LeRobot Parquet^[18]. LeRobot's schema is simpler than RLDS but lacks nested observation dicts, requiring flattened column names like `observation.rgb_primary` and `observation.rgb_wrist`.

Cross-Modal Validation and Quality Checks

Timestamp alignment validation catches synchronization bugs before training. Compute the time delta between RGB frames and proprioceptive states for each step; flag episodes where >5% of deltas exceed 20 ms. DROID rejected 8% of collected trajectories due to camera frame drops that created 50+ ms gaps^[2]. Plot timestamp histograms per modality to detect clock drift or missed triggers.

Depth-RGB consistency verifies sensor calibration. Project depth pixels to 3D using camera intrinsics, transform to the robot base frame, and compare against forward kinematics of the end-effector. Open X-Embodiment required depth-FK agreement within 2 cm for grasping tasks^[1]. Misaligned extrinsics produce policies that reach 5-10 cm off-target.

Force-contact agreement validates force-torque data. During contact phases (gripper closing, pushing), force magnitude should exceed 0.5 N; during free motion, forces should remain below 0.2 N. DROID used contact labels derived from force thresholds to train a contact predictor, achieving 94% accuracy on held-out episodes^[2]. Noisy force data (loose sensor mounting, electrical interference) produces false contact signals that confuse policy learning.

Action magnitude sanity checks prevent corrupted labels. Compute per-step action norms; flag episodes where >10% of actions exceed 3× the median. BridgeData V2 filtered 4% of trajectories with action spikes caused by teleoperation interface glitches^[5]. Visualize action distributions per task to detect systematic biases (e.g., all pick actions have positive Z velocity).

Handling Missing Modalities and Cross-Embodiment Compatibility

Missing modalities are common in aggregated datasets. Open X-Embodiment includes 15 datasets without wrist cameras and 8 datasets without force-torque sensors^[1]. Octo handles missing wrist images by masking the corresponding attention heads during training, allowing the model to learn from datasets with varying modality coverage^[7]. Zero-pad missing modalities (e.g., force = [0,0,0,0,0,0]) and set a binary mask tensor (`modality_present`) to inform the model which inputs are valid.

Embodiment-specific state representations require normalization. Open X-Embodiment standardized all proprioception to 7-DoF end-effector poses (position + quaternion) in the robot base frame, even for datasets originally recorded in joint space^[1]. This enables cross-embodiment transfer: a policy trained on a Franka arm can initialize fine-tuning on a UR5 without rewriting the observation encoder. Provide URDF files and forward kinematics code alongside datasets to enable buyers to recompute poses.

Action space alignment is the hardest cross-embodiment problem. RT-1 uses 7-DoF delta end-effector actions (dx, dy, dz, droll, dpitch, dyaw, gripper)^[4]. Diffusion Policy uses absolute joint positions^[8]. OpenVLA uses 7-DoF delta end-effector actions in the base frame^[3]. Provide action conversion scripts (joint-to-EE, absolute-to-delta) and document the action space in a `dataset_card.md` file following Datasheets for Datasets conventions.

Language Annotation Best Practices

Instruction granularity affects policy generalization. RT-1 used task-level instructions (`pick up the apple`) that describe the goal but not the motion plan^[4]. RT-2 added step-level instructions (`move gripper above apple`, `close gripper`) to enable finer-grained control^[6]. Task-level instructions produce policies that plan full trajectories; step-level instructions enable reactive re-planning but require 3-5× more annotation effort.

Lexical diversity prevents overfitting to phrasing. BridgeData V2 collected 3-5 paraphrases per task: `pick up the red block`, `grasp the red cube`, `grab the red object`^[5]. Models trained on single-phrasing datasets fail when users issue synonymous commands. Use template-based generation (`{action} the {color} {object}`) to bootstrap diversity, then manually filter unnatural combinations.

Negation and failure cases improve robustness. RT-2 included 5% negative examples with instructions like `do not pick up the apple` paired with trajectories that avoid the target object^[6]. Without negative examples, models cannot distinguish between `pick up the apple` and `pick up anything except the apple`. Annotate failure trajectories (dropped objects, collisions) with failure-mode labels (`gripper missed target`, `object slipped`) to enable failure-aware training.

Storage, Versioning, and Provenance Tracking

Dataset versioning is mandatory for reproducibility. BridgeData V2 is the second release of BridgeData, adding 45,000 trajectories and depth modality to the original 15,000-trajectory release^[5]. Use semantic versioning (v1.0.0, v1.1.0, v2.0.0) and document changes in a `CHANGELOG.md` file. Hugging Face Datasets supports dataset versioning via Git tags, enabling buyers to pin to specific versions in training scripts^[16].

Provenance metadata answers buyer due diligence questions. Data provenance tracks the origin, transformations, and lineage of each trajectory^[19]. Record: collection date, robot serial number, operator ID, teleoperation interface version, sensor firmware versions, and post-processing script hashes. PROV-O provides an RDF ontology for provenance graphs, but JSON-LD metadata files are more practical for robotics datasets.

Licensing and commercial terms must be explicit. CC-BY-4.0 permits commercial use with attribution^[20]. CC-BY-NC-4.0 prohibits commercial use, blocking model deployment^[21]. RoboNet uses a custom non-commercial license that permits academic research but forbids productionization^[22]. Buyers sourcing data from truelabel's marketplace receive explicit commercial licenses with indemnification, eliminating downstream IP risk.

Benchmarking and Validation Protocols

Held-out environment splits test generalization. BridgeData V2 reserved 3 of 13 kitchens for evaluation, ensuring test scenes were unseen during training^[5]. DROID used 10% of scenes for validation and 15% for test, stratified by task difficulty^[2]. Single-environment datasets cannot measure generalization — buyers should demand multi-scene coverage.

Success rate metrics require task-specific predicates. RT-1 defined success as `object in target bin AND gripper open AND no collisions`^[4]. OpenVLA used human evaluators to label success on 300 held-out episodes, achieving 0.89 inter-rater agreement^[3]. Automated success detection (via object tracking or force thresholds) reduces evaluation cost but introduces 5-10% false positive rates.

Sim-to-real transfer validation proves dataset quality. Domain randomization trains policies in simulation with randomized textures, lighting, and dynamics, then deploys to real robots^[23]. RLBench provides 100 simulated tasks with automatic success detection, enabling rapid iteration before real-world collection^[24]. Datasets that enable high sim-to-real success rates (>70%) demonstrate sufficient visual and dynamic diversity.

Scaling Collection with Distributed Teleoperation

DROID scaled to 76,000 trajectories by deploying 50 identical robot cells across university labs, each with standardized sensor rigs and teleoperation interfaces^[2]. Centralized data pipelines ingested ROS bags via SFTP, ran validation checks, and rejected 12% of uploads due to quality issues. Distributed collection requires: (1) hardware kits with locked-down sensor configurations, (2) operator training protocols with certification tests, (3) automated quality gates that block bad data before aggregation.

Open X-Embodiment aggregated data from 21 institutions by defining a common RLDS schema and providing conversion scripts for 15 existing dataset formats^[1]. Centralized schema enforcement (via JSON Schema validators) caught 34% of initial submissions with malformed tensors or missing metadata. Distributed collection without schema validation produces incompatible datasets that cannot be jointly trained.

Crowdsourced teleoperation remains experimental. RoboTurk enabled remote operators to control real robots via web browsers, collecting 2,000 trajectories from 100 workers^[25]. Latency (200-500 ms over public internet) degraded manipulation quality, and 18% of trajectories contained gripper control errors. VPN-based remote access with <50 ms latency is viable for coarse manipulation but not for contact-rich tasks.

Emerging Formats: World Models and Generative Priors

NVIDIA Cosmos introduces world foundation models trained on 20 million hours of video, including 1.6 million hours of robot manipulation footage^[26]. These models generate synthetic trajectories by predicting future frames conditioned on actions, enabling data augmentation without physical collection. GR00T N1 uses Cosmos-generated rollouts to pre-train policies, then fine-tunes on 500-1,000 real demonstrations per task^[27].

Generative priors reduce real-world data requirements by 10-100×. World Models learn latent dynamics from video, enabling model-based RL with 10× fewer environment interactions^[28]. General Agents Need World Models argues that foundation models must internalize physics, geometry, and causality to achieve human-level generalization^[29]. Buyers sourcing multimodal data should prioritize datasets with dense temporal coverage (30+ Hz) and long horizons (60+ seconds) to enable world model pre-training.

Cost and Timeline Estimation

Per-trajectory costs vary by task complexity and modality richness. Simple pick-and-place with RGB-only costs $5-15 per trajectory (10-20 minutes operator time + annotation). Contact-rich bimanual tasks with RGB-D, force, and language cost $40-80 per trajectory (30-60 minutes operator time + multi-stage annotation). DROID reported $12 per trajectory for single-arm manipulation with dual RGB cameras and force sensing^[2].

Annotation labor dominates cost for language-conditioned datasets. RT-1 required 2-3 minutes per trajectory for language annotation, adding $3-6 per trajectory at $60/hour annotator rates^[4]. Template-based annotation reduces cost to $0.50-1.50 per trajectory but sacrifices lexical diversity. Truelabel's marketplace offers hybrid annotation: operators provide free-form descriptions during teleoperation, then specialist annotators normalize them in bulk at $25/hour.

Timeline estimates for 10,000-trajectory datasets: 3-4 weeks for hardware setup and calibration, 6-10 weeks for teleoperation (2-4 operators), 2-3 weeks for annotation and quality checks, 1-2 weeks for format conversion and validation. Distributed collection across 5-10 sites compresses teleoperation to 2-3 weeks but adds 1-2 weeks for cross-site coordination.

Procurement Checklist for Multimodal Robot Datasets

Buyers sourcing multimodal data should demand: (1) modality specification table listing tensor shapes, dtypes, coordinate frames, and sampling rates for all modalities, (2) timestamp alignment report showing per-modality sync accuracy with histograms, (3) calibration artifacts including camera intrinsics, extrinsics, and URDF files, (4) validation metrics covering depth-FK consistency, force-contact agreement, and action magnitude distributions, (5) language annotation guidelines with inter-annotator agreement scores, (6) provenance metadata tracking collection dates, operators, and sensor versions, (7) license terms explicitly permitting commercial model training and deployment.

Datasheets for Datasets provides a 50-question template covering motivation, composition, collection process, preprocessing, uses, distribution, and maintenance^[30]. Data Cards extend datasheets with structured metadata for automated compliance checks^[31]. Buyers should reject datasets without completed datasheets — missing metadata signals poor collection discipline and increases integration risk.

Format compatibility must be verified before purchase. Request sample episodes (10-20 trajectories) in the target format (RLDS, LeRobot, or custom HDF5) and run them through your training pipeline. LeRobot provides format validators that check tensor shapes, dtypes, and required fields^[18]. Incompatible datasets require 2-4 weeks of conversion engineering, erasing any cost advantage over custom collection.

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

External references and source context

1. Open X-Embodiment: Robotic Learning Datasets and RT-X Models

   Open X-Embodiment 527 skills across 22 embodiments and synchronization requirements

   arXiv ↩
2. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

   DROID 76,000 trajectories, sensor configuration, and quality metrics

   arXiv ↩
3. OpenVLA: An Open-Source Vision-Language-Action Model

   OpenVLA 34% success rate improvement and input specifications

   arXiv ↩
4. RT-1: Robotics Transformer for Real-World Control at Scale

   RT-1 paired 130,000 demonstrations with 700 natural language instructions

   arXiv ↩
5. BridgeData V2: A Dataset for Robot Learning at Scale

   BridgeData V2 60,000 trajectories, sensor specs, and task diversity

   arXiv ↩
6. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

   RT-2 architecture and input requirements

   arXiv ↩
7. Open X-Embodiment: Robotic Learning Datasets and RT-X Models

   Octo modality requirements and missing modality handling

   arXiv ↩
8. Diffusion Policy training example

   Diffusion Policy action space and input requirements

   GitHub ↩
9. RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning

   RLDS schema definition and step structure

   arXiv ↩
10. Teleoperation datasets are becoming the highest-intent physical AI content category

    ALOHA teleoperation throughput

    tonyzhaozh.github.io ↩
11. ROS bag format 2.0

    ROS 2 bag format and recording capabilities

    ROS Wiki ↩
12. MCAP file format

    MCAP format performance characteristics

    mcap.dev ↩
13. HDF5 format overview

    HDF5 hierarchical storage and compression

    hdfgroup.org ↩
14. LeRobot dataset documentation

    LeRobot dataset schema and Parquet format

    Hugging Face ↩
15. Apache Parquet file format

    Apache Parquet columnar format

    Apache Parquet ↩
16. Hugging Face Datasets documentation

    Hugging Face Datasets streaming and versioning

    Hugging Face ↩
17. RLDS GitHub repository

    RLDS reference converters

    GitHub ↩
18. LeRobot GitHub repository

    LeRobot CLI and format validators

    GitHub ↩
19. truelabel data provenance glossary

    Data provenance definition and tracking requirements

    truelabel.ai ↩
20. Attribution 4.0 International deed

    CC-BY-4.0 commercial use permissions

    Creative Commons ↩
21. Creative Commons Attribution-NonCommercial 4.0 International deed

    CC-BY-NC-4.0 commercial use restrictions

    creativecommons.org ↩
22. RoboNet dataset license

    RoboNet non-commercial license terms

    GitHub raw content ↩
23. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

    Domain randomization for sim-to-real transfer

    arXiv ↩
24. RLBench: The Robot Learning Benchmark & Learning Environment

    RLBench 100 simulated tasks

    arXiv ↩
25. Real robot dataset

    RoboTurk crowdsourced teleoperation statistics

    roboturk.stanford.edu ↩
26. NVIDIA Cosmos World Foundation Models

    NVIDIA Cosmos 20 million hours of training data

    NVIDIA Developer ↩
27. NVIDIA GR00T N1 technical report

    GR00T N1 synthetic data augmentation

    arXiv ↩
28. World Models

    World Models latent dynamics learning

    worldmodels.github.io ↩
29. General Agents Need World Models

    Foundation models require world model priors

    arXiv ↩
30. Datasheets for Datasets

    Datasheets for Datasets 50-question template

    arXiv ↩
31. Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI

    Data Cards structured metadata

    arXiv ↩

FAQ