How-to guides for physical AI data

3D point cloud annotation transforms raw LiDAR or depth sensor captures into labeled training data by marking object boundaries, semantic classes, and spatial relationships. Core tasks include 3D bounding box placement (6-DOF cuboids around vehicles, pedestrians, obstacles), semantic segmentation (per-point class labels for road, vegetation, buildings), instance segmentation (separating individual objects within a class), and grasp pose labeling (6-DOF affordance annotations for manipulation). Production pipelines combine manual tooling (Segments.ai, Kognic, Scale 3D Sensor Fusion) with automated pre-labeling from PointNet++ or transformer backbones, then validate via inter-annotator agreement metrics and downstream model performance on held-out test scenes.

• LiDAR annotation
• Semantic segmentation point clouds

The sim-to-real gap is closed through three complementary techniques: domain randomization during simulation training (randomizing visual textures, lighting, physics parameters to force policy robustness), system identification to match simulator parameters to real hardware (measuring friction coefficients, actuator latencies, camera intrinsics), and real-world fine-tuning on 50-500 teleoperation demonstrations collected on target hardware. Policies trained with domain randomization achieve 60-80% baseline transfer success; fine-tuning on real data closes the remaining gap to 85-95% task success rates.

• Domain randomization
• System identification

Building a benchmark dataset requires defining a task suite spanning 8-15 manipulation primitives across three difficulty tiers, specifying initial-state distributions with documented randomization parameters, implementing multi-axis success metrics (task completion, trajectory efficiency, safety margins), collecting 50-200 expert demonstrations per task in standardized formats like RLDS or LeRobot, and publishing evaluation harnesses with reproducible seeding. Strong benchmarks isolate capabilities (grasping, sequencing, force control) rather than bundling them into monolithic tasks where failure modes cannot be diagnosed.

• Robot evaluation
• Manipulation benchmark

A contact-rich manipulation dataset captures force/torque, tactile, and visual streams during tasks like insertion, assembly, or wiping. You need a 6-axis force/torque sensor sampling at 500+ Hz, synchronized RGB-D cameras, optional tactile sensors, a teleoperation interface with force feedback, and a recording pipeline that timestamps all modalities to sub-millisecond precision. The DROID dataset collected 76,000 trajectories across 564 skills using this architecture; Open X-Embodiment aggregated 1 million trajectories from 22 robot embodiments, proving multi-modal contact data scales generalist policies when provenance and sensor metadata are preserved.

• Force torque sensor
• Tactile sensor robotics

Building a humanoid training dataset requires four technical pillars: motion capture or teleoperation hardware to record full-body demonstrations, a kinematic retargeting pipeline that maps human motion to robot joint space, synchronized multi-sensor recording (RGB cameras, depth, IMU, joint encoders) at 30+ Hz, and episode-level quality validation before formatting to RLDS or LeRobot schemas for policy training.

• Motion capture retargeting
• Teleoperation data collection

A language-conditioned dataset pairs natural language instructions with robot demonstrations, enabling vision-language-action (VLA) models to follow free-form commands. Build one by defining a task ontology mapping instructions to behaviors, recording synchronized multi-modal data (RGB-D video, proprioception, audio), collecting demonstrations with concurrent language scaffolding, generating paraphrases to expand linguistic diversity, validating alignment between language and action trajectories, and formatting outputs for VLA training frameworks like LeRobot or RLDS.

• Vision Language Action models
• VLA training data

A manipulation dataset pairs robot trajectories with multi-modal observations (RGB-D, proprioception, force) collected via teleoperation or scripted policies. Production pipelines require task specification, hardware calibration (camera intrinsics, temporal sync), teleoperation interfaces (VR, leader-follower, SpaceMouse), episode recording in RLDS or HDF5 formats, and validation against success metrics before training. The Open X-Embodiment consortium aggregated 1 million trajectories across 22 robot embodiments, demonstrating that cross-embodiment generalization demands standardized action spaces and rich language annotations alongside pixel observations.

• Robot learning dataset
• Teleoperation data collection

Building a preference dataset for RLHF in physical AI requires assembling a diverse trajectory pool spanning expert teleoperation, policy rollouts, and failure modes; designing a pairwise comparison interface with clear success criteria; calibrating annotators on 50-100 gold-standard pairs to achieve 75%+ inter-annotator agreement; collecting 2,000-10,000 preference judgments; training a Bradley-Terry reward model; and validating that learned preferences correlate with downstream task success metrics.

• Reward model training
• Pairwise preference annotation

A safety monitoring pipeline for physical AI systems continuously validates sensor streams, action commands, and environmental state against predefined safety constraints—force/torque thresholds, workspace boundaries, collision proximity, joint limits—and triggers emergency stops when violations occur. Production pipelines combine real-time checks (sub-10ms latency) with offline anomaly detection trained on historical telemetry, logging every intervention for root-cause analysis and model retraining.

• Physical AI safety
• Robot safety monitoring

An object tracking dataset assigns persistent IDs to objects across video frames, enabling models to follow entities through occlusions, viewpoint changes, and multi-camera handoffs. Production pipelines combine pre-annotation from detection models like DETR with human review in platforms like CVAT, enforce temporal consistency through track lifecycle rules, and export to RLDS or LeRobot formats for policy training. Truelabel's marketplace holds 20,000+ collectors capturing multi-modal tracking data across warehouse, kitchen, and outdoor manipulation scenarios.

• Temporal annotation
• Multi Object tracking

Multi-camera calibration establishes intrinsic parameters (focal length, distortion) per camera and extrinsic transforms (rotation, translation) between cameras and robot base frames. Use ChArUco boards printed on rigid substrates for intrinsic calibration, solve hand-eye equations for camera-to-robot transforms, synchronize frames via hardware triggers or PTP, and validate reprojection error <0.5px. Calibration drift detection every 500 episodes prevents systematic pose errors that degrade imitation learning policies.

• Camera intrinsics
• Extrinsic calibration

Force-torque data captures contact forces and moments during manipulation tasks, enabling robots to learn contact-rich skills like insertion, assembly, and tool use. Collection requires mounting a 6-axis F/T sensor between the robot wrist and gripper, calibrating gravity and inertial compensation, recording at 500+ Hz synchronized with visual streams, and formatting episodes in RLDS or LeRobot schemas with per-timestep force vectors and torque vectors alongside RGB-D observations and proprioceptive state.

• F/T sensor calibration
• Contact Rich manipulation data

Kitchen activity data collection requires synchronized multi-view RGB-D cameras (fixed overhead plus wrist-mounted egocentric), temporal annotation of verb-noun action pairs at 1-2 second granularity, and export to RLDS or HDF5 formats compatible with imitation learning pipelines. A production dataset needs 50-200 hours of annotated video across 15-30 recipes, captured in 3-6 distinct kitchen environments to ensure cross-domain generalization for manipulation policies.

• Egocentric video dataset
• Robot manipulation training data

Multimodal robot data collection requires synchronized capture of RGB images, depth maps, proprioceptive state (joint angles or end-effector poses), optional force-torque readings, and natural language instructions. Use hardware-triggered cameras or PTP network sync to align timestamps within 5 ms, record to ROS bags or MCAP containers, then convert to RLDS or LeRobot formats for VLA training.

• Vision Language Action models
• VLA training data

Teleoperation data collection requires four core components: a control interface (VR controllers, leader-follower arms, or spacemouse), synchronized multi-camera recording infrastructure capturing RGB-D streams at 15-30 Hz, trained operators executing task protocols with state randomization, and validation pipelines checking trajectory success rates and action distributions before dataset packaging in RLDS or LeRobot formats.

• Robot demonstration data
• Imitation learning datasets

Warehouse robot data collection requires a synchronized sensor rig (RGB-D cameras, LiDAR, IMU), a task taxonomy mapping manipulation and navigation primitives to observation-action pairs, a ROS2-based recording pipeline capturing 10-20 Hz telemetry, and post-processing into RLDS or HDF5 formats. Successful datasets span 50+ SKU variants, 5+ lighting conditions, and 500+ episodes per task family to support policy generalization across real-world warehouse variability.

• Warehouse robotics training data
• AMR navigation datasets

RLDS (Reinforcement Learning Datasets) is a TensorFlow Datasets extension that standardizes robot demonstration data into episode-structured TFRecords. Converting to RLDS requires auditing source data (HDF5, ROS bags, MCAP), defining a schema with observation/action/reward fields, implementing a TFDS DatasetBuilder that extracts episodes, and validating output against policy training requirements. The format powers 22 datasets in Open X-Embodiment (800K episodes) and models like RT-1, RT-2, and OpenVLA.

• RLDS conversion
• TensorFlow Datasets robot data

Creating a robot demonstration dataset requires five engineering phases: defining the observation-action contract and task specifications, assembling teleoperation hardware with synchronized multi-modal recording, training operators and running pilot collections to validate quality metrics, executing full-scale collection with real-time QA monitoring, and post-processing episodes into training-ready formats like RLDS or LeRobot HDF5 with train-validation splits and metadata.

• Teleoperation dataset
• Imitation learning data

Creating a semantic segmentation dataset requires defining a class taxonomy aligned to robot perception tasks, collecting representative imagery from target environments, annotating pixel-level masks using tools like CVAT or Label Studio with SAM 2 model assistance, validating annotations against inter-annotator agreement thresholds above 85% IoU, and exporting to training-ready formats like COCO JSON or HDF5. For manipulation robots, proven taxonomies include 8-12 classes covering graspable objects, support surfaces, obstacles, robot links, and human hands. Annotation velocity reaches 15-30 images per hour with model-assisted workflows compared to 3-5 images per hour for manual polygon tracing.

• Pixel Level annotation
• Robot perception dataset

Action chunking transforms sequential robot demonstrations into fixed-length temporal windows that policy models consume during training. You audit source trajectories for temporal consistency, select chunk size and horizon parameters based on your target architecture (ACT uses 100-step chunks, Diffusion Policy uses 16-step), implement sliding-window extraction with proper padding, compute per-dimension action normalization statistics, serialize to RLDS or LeRobot format, and validate end-to-end with a training smoke test.

• Action chunking
• Robot demonstration data

Safety-labeled robot data pairs demonstration trajectories with annotations marking constraint violations—collisions, force exceedances, workspace breaches, speed limits. Production workflows combine automated pre-labeling (collision detection, force thresholds) with human review to flag hazardous states. Datasets require 15-25% negative demonstrations showing failure modes, validated against domain-specific taxonomies (ISO 10218 industrial, ISO 15066 collaborative). Format as RLDS episodes with per-timestep safety masks, enabling constraint-aware policy training that generalizes beyond collision-free imitation to real-world deployment constraints.

• Robot safety annotation
• Constraint Aware policy training

Temporal video annotation assigns time-aligned labels to action segments in video streams. For robotics datasets, annotators mark start/end frames for manipulation primitives (reach, grasp, transport, place) using tools like CVAT or Label Studio, then validate boundaries with inter-annotator agreement metrics. The EPIC-KITCHENS dataset uses 97,000 action segments across 700 hours of egocentric video, while DROID contains 76,000 manipulation trajectories with frame-level action labels.

• Action segmentation
• Video labeling for robotics

A production teleoperation interface requires four design pillars: control-mode selection (position, velocity, or hybrid mapping), sub-50ms end-to-end latency, multi-camera operator feedback with task-relevant overlays, and episode workflow automation. Position control via leader-follower arms or VR controllers produces the highest-quality manipulation demonstrations because operators directly specify target poses. The DROID dataset collected 76,000 trajectories across 564 skills using this architecture, proving that interface design directly determines dataset scale and task coverage.

• Robot data collection
• Leader Follower control

Robot policy evaluation requires binary success criteria (e.g., object lifted >5cm AND placed within 3cm of target), controlled variation across object poses and lighting, minimum 50 trials per condition for 80% power at p<0.05, video logging of every trial, and failure-mode taxonomies that map directly to data collection priorities—teams shipping policies without this rigor see 40–60% deployment failure rates.

• Robot manipulation evaluation
• Policy success criteria

Sim-to-real transfer evaluation requires three phases: establish simulation baselines across 1,000+ episodes measuring success rates and action distributions, execute controlled real-world trials with matched initial conditions while logging visual and dynamics discrepancies, then attribute performance gaps to specific sources—visual domain shift, physics mismatch, or actuation errors—using diagnostic metrics like CLIP embedding distance and trajectory RMSE to guide domain randomization or fine-tuning interventions.

• Domain randomization
• Reality gap

Training data quality evaluation requires measuring 12 quantifiable dimensions across episode and dataset levels: temporal synchronization between sensors (≤16ms jitter for 60Hz control), action trajectory smoothness (acceleration variance <0.8 m/s³), observation completeness (≥98% frame presence), label consistency (≥95% inter-annotator agreement), state-space coverage (Shannon entropy ≥4.2 bits for manipulation tasks), and domain diversity (≥8 distinct environment configurations per task). Statistical validation combines per-episode metrics with dataset-level distribution analysis to predict downstream policy performance before expensive training runs.

• Robot training data quality
• Physical AI dataset validation

Fine-tuning a vision-language-action model requires converting your robot demonstrations into RLDS format with 256×256 RGB observations and normalized 7-DoF actions, configuring LoRA adapters with rank 32–64 to reduce VRAM from 80GB to under 24GB, training for 5,000–15,000 steps on 4–8 A100 GPUs over 12–48 hours, and validating success rates above 70% on held-out tasks before deploying the policy to your physical robot with 10–30Hz control loops.

• Vision Language Action model training
• OpenVLA fine Tuning

Synthetic robot data generation combines physics simulation (MuJoCo, Isaac Gym, PyBullet) with domain randomization to produce training episodes at 10,000+ per hour on GPU clusters. Teams implement visual randomization (lighting, textures, camera poses) and physical randomization (mass, friction, actuator noise) to bridge the sim-to-real gap, then validate transfer quality by measuring task success rates on real hardware. Optimal training mixes 60-80% synthetic episodes with 20-40% real teleoperation data to achieve 85-92% real-world success rates across manipulation benchmarks.

• Domain randomization
• Sim To Real transfer

Data versioning for robotics requires tracking both raw sensor streams (camera frames, joint states, force-torque readings) and derived artifacts (annotations, model checkpoints, evaluation metrics) across collection cycles. Use Git for metadata and code, DVC or LFS for large binary files, and structured formats like HDF5, MCAP, or RLDS for episode storage. Embed provenance metadata (collector ID, robot serial, calibration version) in every episode file. Maintain a dataset registry mapping version tags to training runs, enabling reproducible experiments and rollback when model performance degrades. The Open X-Embodiment dataset aggregates 1M+ trajectories from 22 robot embodiments using this approach.

• Robot dataset versioning
• Robotics data provenance

Grasp success labeling requires a binary outcome decision (success/failure) anchored to task-specific criteria: object lifted above threshold height, held for minimum duration, or placed within target tolerance. Modern pipelines combine force-torque sensor thresholds with visual confirmation (object in gripper, stable pose) and encode outcomes as boolean flags in episode metadata. The DROID dataset labels 76,000 manipulation trajectories with per-step success annotations; Open X-Embodiment aggregates 22 datasets totaling 1M+ episodes with grasp outcome labels. Production systems automate detection via gripper state + object tracking, then route edge cases (partial grasps, slippage, re-attempts) to human review queues with pre-filled suggestions to maintain 95%+ inter-annotator agreement.

• Grasp outcome annotation
• Manipulation dataset labeling

Multi-site data collection distributes robot teleoperation and sensor capture across geographically separate facilities to accelerate dataset growth and capture environmental diversity. Success requires four pillars: standardized hardware manifests and software containers at each site, automated quality gates that reject malformed episodes before upload, a central aggregation layer that reconciles coordinate frames and timestamps, and continuous monitoring dashboards that surface collection velocity and error rates in real time.

• Robot data pipeline
• Distributed teleoperation

Inter-annotator agreement (IAA) quantifies consistency between multiple human annotators labeling the same data. For physical AI datasets, measure IAA by designing a 15-25% overlap protocol where annotator pairs independently label identical episodes, then compute metric-specific scores: Cohen's kappa or Fleiss' kappa for categorical labels (object classes, grasp types), intraclass correlation coefficient (ICC) for continuous values (force measurements, trajectory smoothness), and Krippendorff's alpha for temporal or ordinal annotations. Scores above 0.80 indicate strong agreement; 0.60-0.80 moderate; below 0.60 signals taxonomy ambiguity or insufficient training requiring immediate remediation.

• Cohen's kappa
• Fleiss' kappa

Dataset diversity optimization requires measuring coverage across visual (lighting, viewpoint, occlusion), spatial (workspace zones, approach angles), object (geometry, material, articulation), and behavioral (trajectory curvature, contact force, failure recovery) dimensions. Effective protocols combine stratified sampling (target 80+ distinct scene configurations per task), active learning (prioritize high-uncertainty regions), and continuous monitoring (track per-dimension entropy). The Open X-Embodiment dataset demonstrates this: 22 robot embodiments, 527 skills, 160,000 tasks across 21 institutions yield 30% better zero-shot transfer than single-lab collections[ref:ref-open-x-embodiment].

• Robot learning dataset diversity
• Physical AI data collection

Point cloud preprocessing transforms raw depth sensor output into training-ready 3D representations for robot manipulation policies. The pipeline includes depth-to-point conversion using camera intrinsics, statistical outlier removal, multi-view registration via ICP or TSDF fusion, table plane segmentation with RANSAC, voxel downsampling to target point counts (typically 1,024–8,192 points), coordinate frame normalization, and packaging in formats like HDF5 or Parquet for batch training.

• Depth sensor calibration
• Point cloud filtering

Bimanual demonstration recording captures synchronized dual-arm manipulation trajectories for training policies like ALOHA and RT-X. Core requirements: hardware synchronization across two robot arms (≤5ms timestamp drift), teleoperation interfaces that map human bimanual input to dual end-effectors, and storage formats (RLDS, HDF5, MCAP) that preserve per-arm action-observation tuples with shared episode metadata. Quality hinges on temporal alignment, action space consistency across arms, and operator training for coordinated two-hand tasks.

• Dual Arm teleoperation
• ALOHA dataset

A mobile manipulation rig combines a wheeled base with a mounted robotic arm to collect navigation and manipulation data simultaneously. Core steps: select a mobile platform (differential-drive or omnidirectional), mount a 6-7 DoF arm with end-effector cameras, synchronize all sensors to a shared clock, configure ROS2 or MCAP recording pipelines, and collect teleoperated demonstrations across varied environments to generate training data for vision-language-action models.

• Mobile manipulation data collection
• Robot teleoperation setup

A data quality pipeline for physical AI datasets automates validation across collection, session, and release stages. Real-time checks catch sensor dropouts and synchronization drift during teleoperation. Session-level statistical validation flags outlier episodes by duration, action smoothness, and frame completeness. Human review workflows triage flagged data for re-collection or repair. Dataset-level validation enforces schema compliance and provenance metadata before release, reducing downstream training failures by 40-60%.

• Robot dataset validation
• Teleoperation data quality

A teleoperation rig captures human demonstrations for robot imitation learning by pairing an input device (leader arm, VR controller, or SpaceMouse) with a follower robot, synchronized cameras, and a recording pipeline that logs joint states, end-effector poses, RGB-D streams, and task metadata into MCAP or HDF5 containers at 15-30 Hz. Production rigs balance interface fidelity (leader-follower arms yield 85-90% task success vs 60-75% for VR), hardware cost ($800-$16,000 per station), and operator throughput (20-50 episodes per 8-hour shift). The DROID dataset collected 76,000 trajectories across 564 skills using this architecture[ref:ref-droid-paper], while Open X-Embodiment aggregated 1 million episodes from 22 robot embodiments[ref:ref-open-x-embodiment].

• Robot teleoperation data collection
• Leader Follower arm setup

A domain randomization pipeline systematically varies visual, physics, and dynamics parameters during synthetic data generation to train policies that generalize from simulation to real hardware. The pipeline requires a physics simulator (Isaac Sim, MuJoCo, or RLBench), randomization APIs for lighting/textures/friction/mass, a training loop that samples parameter distributions per episode, and real-world validation to tune ranges and identify sim-to-real gaps.

• Sim To Real transfer
• Synthetic training data

Training a diffusion policy requires a demonstration dataset of 100+ episodes with synchronized observations and actions, a vision encoder (ResNet-18 or ViT), and a conditional denoising network (U-Net or Transformer). Normalize actions to [-1,1], configure a DDPM or DDIM noise schedule with 10-100 diffusion steps, set observation horizon to 2-4 frames and action horizon to 8-16 steps, then train with AdamW optimizer for 50,000-200,000 gradient steps while monitoring MSE loss and success rate on held-out validation episodes.

• Diffusion policy training
• Robot manipulation policy

Action label validation ensures robot learning datasets contain physically plausible, temporally consistent control signals. Core validation steps: verify action-observation alignment via forward kinematics, check joint limits and velocity bounds against URDF specifications, detect timestamp drift between sensor streams, apply statistical outlier detection to catch encoder noise, and run end-to-end trajectory replay in simulation to surface labeling errors before training begins.

• Robot learning datasets
• Action label validation

RLDS and LeRobot are the two dominant serialization standards for robot learning datasets. RLDS wraps TensorFlow Datasets with trajectory semantics for RL agents; LeRobot uses HDF5 + Parquet for Hugging Face ecosystem integration. Converting between them requires mapping observation/action schemas, resampling timestamps to match episode boundaries, and validating tensor shapes against target model APIs. Teams typically maintain dual-format exports: RLDS for Google Research pipelines (RT-1, RT-2), LeRobot for OpenVLA and diffusion policies.

• Robot learning data formats
• RLDS conversion