NVIDIA PhysicalAI GR00T X-Embodiment Sim Dataset

This dataset provides 9,000 simulated bimanual manipulation trajectories generated in NVIDIA Isaac Sim, specifically designed for post-training the GR00T N1 robotics foundation model. The collection spans three distinct manipulation tasks—threading, tray lifting, and three-piece assembly—performed across two robot embodiments: Panda arms equipped with parallel-jaw grippers and Panda arms with anthropomorphic hands. Each task contributes 1,000 trajectories per embodiment configuration, creating a structured benchmark for cross-embodiment policy learning. The trajectories capture full state information including joint positions, end-effector poses, object states, and action sequences, enabling both behavior cloning and inverse dynamics learning. The data was generated as part of NVIDIA's PhysicalAI initiative to develop general-purpose humanoid and manipulation capabilities, with simulation allowing rapid iteration and perfect ground-truth labels unavailable in real-world capture scenarios.

The dataset is released under CC-BY-4.0, one of the most permissive open licenses available. This explicitly grants rights for commercial use, modification, distribution, and creation of derivative works, with the sole requirement being appropriate attribution to NVIDIA. You can train proprietary models on this data, deploy those models in commercial products, and retain full ownership of the resulting system without share-alike obligations. For compliance, attribution typically involves citing NVIDIA as the dataset creator in technical documentation, model cards, or research papers. Unlike GPL-style licenses, CC-BY-4.0 does not require you to release your model weights or disclose training procedures, making it fully compatible with commercial product development where intellectual property protection is essential.

This dataset is particularly valuable for teams developing vision-language-action models or bimanual manipulation policies that must generalize across different robot morphologies. If you are building foundation models that need to transfer skills between robots with different end-effectors—such as adapting a gripper-trained policy to dexterous hands—the explicit cross-embodiment structure provides controlled experimental scaffolding. Teams working on humanoid robots with dual-arm capabilities will find the task diversity and coordination patterns directly applicable. Organizations in the post-training or fine-tuning phase of foundation model development gain the most immediate value, as the 9,000 trajectory scale suits adaptation workflows better than pre-training from scratch. The simulation environment also benefits teams performing extensive ablation studies or hyperparameter sweeps, where deterministic replay and perfect state observability accelerate debugging compared to noisy real-world data.

If your deployment environment involves single-arm manipulation, mobile manipulation, or outdoor unstructured settings, this dataset provides limited relevance as it focuses exclusively on stationary bimanual tasks. Teams requiring massive-scale pre-training data for foundation models should look elsewhere, as 9,000 trajectories undershoots the millions of demonstrations typically needed for training large transformer policies from random initialization. This serves better as a supplementary fine-tuning resource than a primary training corpus. The simulation provenance also makes it unsuitable as a sole data source for production deployment without significant real-world bridging. If you lack the infrastructure for sim-to-real transfer techniques like domain randomization, real-world fine-tuning, or adversarial adaptation, the distribution gap will likely cause policy failures on physical hardware. Teams working with deformable objects, precise force control, or tasks sensitive to contact dynamics should be especially cautious, as Isaac Sim's rigid-body assumptions diverge substantially from real material behavior.

The dataset follows standard trajectory formats compatible with behavior cloning and offline reinforcement learning workflows common in VLA development. Each trajectory includes observation sequences, action labels, and state information that can be directly ingested by frameworks like OpenVLA, RoboMimic, or custom transformer architectures. The cross-embodiment structure allows for multi-task training where a single policy network learns task semantics while embodiment-specific adapter layers handle morphology differences, a pattern increasingly common in foundation model architectures. For teams already training on real-world data, this simulation collection serves as an effective augmentation source to improve sample efficiency and provide coverage of rare edge cases difficult to capture physically. The consistent task definitions enable curriculum learning strategies where models first master simulated versions before fine-tuning on smaller real-world datasets. However, practitioners should implement observation normalization and domain randomization during training to prevent overfitting to Isaac Sim's specific rendering and physics characteristics.

While the dataset provides structured trajectories, production training pipelines typically require several preprocessing steps. You will likely need to resample control frequencies to match your target robot's actuation rates, as simulation often uses higher frequencies than physical hardware supports. Observation spaces may require normalization or projection to match your robot's sensor suite, particularly if your cameras have different intrinsics or your proprioceptive sensors report in different coordinate frames than the Panda embodiments used here. Teams pursuing sim-to-real transfer should implement domain randomization during data loading, adding synthetic noise to proprioceptive readings, varying lighting and texture in visual observations, and introducing small perturbations to action execution. The perfect ground-truth state information should be degraded to match real-world sensor noise profiles, preventing the policy from relying on unrealistically precise signals. Budget engineering time for these augmentation pipelines, as naive training on unmodified simulation data typically yields policies that fail immediately when deployed to physical robots due to the distribution shift.