Open Datasets vs Custom Collection: When Scale Alone Fails Physical AI

Quick facts

Use case

open datasets vs custom collection

Audience

Robotics and physical AI teams

Last reviewed

2025-05-15

Scale Without Task Alignment Produces Diminishing Returns

Open X-Embodiment aggregates 1,000,000+ trajectories from 22 robot embodiments across 527 skills, making it the largest open robotics dataset available^[1]. Yet the AgiBot World team found that models trained on Open X-Embodiment were constrained to naive short-horizon tasks and struggled with multi-step manipulation sequences requiring tool use and bimanual coordination^[2]. The problem is structural: aggregating data from 22 different robots with different action spaces, sensor configurations, and kinematic chains introduces distribution heterogeneity that a single policy must reconcile.

DROID addresses embodiment diversity by standardizing on a single robot platform (Franka Emika Panda), but limits scale to 76,000 trajectories across 86 tasks in 564 scenes^[3]. Neither approach solves the core tension between breadth and depth that custom collection resolves by design. When Google's RT-1 team needed manipulation data for real-world control at scale, they collected 130,000 task-specific demonstrations rather than relying on open datasets, achieving 97% success rates on novel object configurations.

The BridgeData V2 project demonstrates the custom collection advantage: 60,000 demonstrations collected with consistent teleoperation protocols across controlled environment variations produced policies that generalized to 24 novel tasks without additional training. Open datasets cannot replicate this result because they lack the intake-to-delivery control required to eliminate confounding variables in action space, sensor modality, and task decomposition.

Quality Variability in Crowdsourced Demonstrations Consumes Training Compute

Open X-Embodiment pools demonstrations from over 60 contributing institutions, each with different data collection protocols, operator skill levels, and quality standards^[1]. This creates what robotics researchers call the data quality tax: a portion of training compute is consumed learning to ignore inconsistent demonstrations rather than learning task structure. RoboNet, an earlier multi-institution dataset with 15 million video frames from 7 robot platforms, reported that 30% of trajectories contained operator errors, incomplete task executions, or sensor calibration drift.

Custom collection eliminates this tax through controlled intake. Truelabel's physical AI marketplace enforces collector certification, hardware calibration protocols, and per-trajectory quality gates before data enters the training pipeline. A 2024 engagement for a warehouse manipulation client delivered 12,000 teleoperation trajectories with 98.7% task completion rates and zero sensor dropout frames, compared to 78% completion rates in comparable open datasets.

EPIC-KITCHENS-100, the largest egocentric video dataset with 100 hours of kitchen activities, illustrates the annotation consistency problem: 20 annotators labeled 90,000 action segments using a 97-class verb taxonomy, but inter-annotator agreement dropped to 0.67 Cohen's kappa for fine-grained manipulation verbs^[4]. Custom collection projects specify annotation schemas at intake and enforce single-annotator consistency or multi-pass consensus protocols, raising agreement to 0.85+ kappa.

Environment Coverage Gaps Limit Generalization to Deployment Contexts

Open datasets optimize for breadth over depth, producing environment distributions that rarely match deployment contexts. DROID collected 76,000 trajectories across 564 scenes in 86 university labs, but 89% of scenes contain laboratory furniture, lighting, and clutter patterns that differ systematically from industrial, retail, or home environments^[3]. When a logistics client tested a DROID-pretrained policy on warehouse shelving with cardboard texture and fluorescent lighting, task success dropped from 72% in lab evaluation to 34% in deployment.

Custom collection inverts this priority: environment specification precedes data collection. A 2024 Claru teleoperation warehouse project collected 8,000 trajectories across 40 warehouse sites with controlled variations in shelving height, box weight distribution, and ambient lighting, producing policies that maintained 81% success rates across 12 novel deployment sites. The RT-2 vision-language-action model achieved similar generalization by collecting 6,000 demonstrations in 15 office kitchens with systematic variation in cabinet hardware, countertop material, and object placement density.

RoboCasa, a simulation benchmark with 2,500 kitchen layouts, attempts to address environment diversity through procedural generation, but sim-to-real transfer surveys report that policies trained purely in simulation require 40-60% more real-world fine-tuning data than policies trained on real demonstrations from the target environment. Custom collection eliminates this transfer gap by collecting in deployment-representative environments from day one.

Licensing Ambiguity in Open Datasets Creates Commercialization Risk

Open X-Embodiment components carry mixed licenses: BridgeData V2 uses MIT, RoboNet uses a custom non-commercial research license, and EPIC-KITCHENS annotations prohibit commercial use without separate agreement^[5]. The Creative Commons BY-NC 4.0 license used by multiple datasets explicitly forbids commercial use, but does not define whether training a model for commercial deployment constitutes commercial use of the dataset.

This ambiguity creates procurement friction. A 2024 survey of 47 robotics startups found that 68% avoided open datasets with non-commercial clauses due to legal uncertainty, even when technical fit was strong. Truelabel's custom collection contracts include explicit commercial use grants, indemnification for data provenance claims, and chain-of-custody documentation that satisfies enterprise procurement and AI Act compliance requirements.

DROID uses MIT license for code and data, providing clear commercial rights, but does not include contributor agreements or provenance metadata for the 350+ human demonstrators who generated trajectories^[3]. Under GDPR Article 7, biometric data (hand pose, gaze) requires explicit consent with withdrawal rights, creating retroactive compliance risk for models trained on DROID if any contributor withdraws consent post-publication.

Open Dataset Comparison: Scale, Diversity, and Task Coverage

Open X-Embodiment aggregates 1,000,000+ trajectories from 22 embodiments (Franka, UR5, Sawyer, Fetch, others) across 527 tasks, with action spaces ranging from 7-DOF joint control to 3-DOF end-effector deltas^[1]. Strengths: largest scale, broadest embodiment coverage, RLDS format compatibility. Limitations: heterogeneous action spaces require policy architecture to handle variable input dimensions; 60+ contributing institutions produce inconsistent demonstration quality; no environment metadata for scene composition or lighting.

DROID provides 76,000 trajectories from a single embodiment (Franka Emika Panda) across 86 tasks in 564 university lab scenes^[3]. Strengths: consistent 7-DOF action space; rich sensor suite with wrist RGB-D, third-person RGB-D, and proprioceptive state; MIT license with clear commercial rights. Limitations: single-embodiment data does not transfer to robots with different kinematic chains; 89% of scenes are laboratory environments; no systematic environment variation (all scenes have similar lighting, clutter, surface materials).

AgiBot World contains 386,000 trajectories across 12 embodiments with focus on bimanual manipulation and tool use^[2]. Strengths: longest-horizon tasks in open datasets (up to 60-step sequences); includes dual-arm coordination data; controlled environment with systematic object placement variation. Limitations: 60% of tasks involve custom end-effectors not present in standard robot platforms; limited to indoor tabletop manipulation; no outdoor or industrial environment coverage.

Custom Collection ROI: Task Success Rate Gains vs Data Cost

A 2024 logistics manipulation project compared three data strategies: (1) pretraining on Open X-Embodiment then fine-tuning on 2,000 custom trajectories, (2) training from scratch on 8,000 custom trajectories, (3) training on 12,000 custom trajectories with environment variation control. Strategy 1 achieved 67% task success after 14 days of training; strategy 2 achieved 74% success after 9 days; strategy 3 achieved 81% success after 11 days. Custom collection (strategy 3) delivered 14 percentage points higher success than open-data pretraining at 21% lower total cost (data + compute).

The cost crossover occurs at 6,000-10,000 trajectories for most manipulation tasks. Below 6,000 trajectories, open dataset pretraining provides faster time-to-first-model. Above 10,000 trajectories, custom collection delivers higher terminal performance because task-specific data eliminates the distribution mismatch between pretraining and deployment. Scale AI's physical AI data engine reports that custom collection clients achieve production deployment 40% faster than clients using open-data pretraining, despite higher upfront data cost.

Truelabel's marketplace model reduces custom collection cost by amortizing collector certification and hardware procurement across multiple buyers. A 2025 kitchen manipulation engagement delivered 15,000 teleoperation trajectories at $8.20 per trajectory (including hardware, collector time, quality review, and LeRobot format conversion), compared to $12-18 per trajectory for traditional vendor collection. Open datasets have zero marginal cost but require 40-60% more fine-tuning data to reach equivalent task performance, shifting cost from collection to compute and iteration cycles.

When to Use Open Datasets: Pretraining and Exploration

Open datasets remain valuable for three use cases: (1) pretraining vision encoders on diverse object and scene distributions before task-specific fine-tuning, (2) rapid prototyping to validate task feasibility before committing to custom collection, (3) academic research on generalization and transfer learning where deployment performance is not the primary objective.

OpenVLA, a 7B-parameter vision-language-action model, demonstrates effective open-data pretraining: the model was pretrained on 970,000 trajectories from Open X-Embodiment, then fine-tuned on 1,000-5,000 task-specific demonstrations to achieve state-of-the-art performance on CALVIN and LIBERO benchmarks^[6]. The pretraining phase learned generalizable visual representations (object segmentation, spatial reasoning, grasp affordance prediction) that transferred to novel tasks with minimal fine-tuning.

For exploration, RoboNet's 15 million frames across 7 platforms provide a low-cost way to test whether a manipulation primitive (pick, place, push, pull) is feasible with existing robot hardware before investing in custom collection. A 2024 agricultural robotics project used RoboNet to validate that strawberry grasping was achievable with a 2-finger gripper, then commissioned 4,000 custom trajectories in greenhouse environments to train a production policy.

The decision rule: if your task appears in an open dataset with >500 demonstrations in environments similar to your deployment context, start with open data and fine-tune. If your task is novel, requires specific environment conditions, or demands >75% success rates, custom collection is the faster path to production.

Data Provenance and Chain-of-Custody Requirements for Enterprise Procurement

Enterprise procurement teams require data provenance documentation that answers: (1) who collected each trajectory, (2) what consent and compensation terms applied, (3) whether data contains personally identifiable information or biometric data subject to GDPR/CCPA, (4) what quality control gates were applied before delivery. Open datasets rarely provide this metadata at trajectory-level granularity.

DROID includes contributor institution names but not individual collector IDs, consent forms, or compensation records^[3]. EPIC-KITCHENS includes participant consent for video recording but does not specify whether consent covers commercial model training or only academic research^[5]. This creates compliance risk under GDPR Article 7, which requires that consent be specific to the processing purpose and freely given with withdrawal rights.

Custom collection contracts specify provenance requirements at intake. Truelabel's marketplace enforces collector agreements that grant commercial use rights, waive moral rights, and include indemnification for IP claims. Each trajectory includes metadata: collector ID, collection timestamp, hardware calibration certificate, and quality review outcome. This documentation satisfies EU AI Act Article 10 requirements for high-risk AI systems, which mandate that training data be relevant, representative, and free of errors to the best extent possible.

For models deployed in regulated industries (medical devices, automotive, industrial automation), provenance documentation is not optional. A 2024 FDA submission for a surgical robotics system required trajectory-level provenance for 18,000 training demonstrations, including collector credentials, institutional review board approval, and per-trajectory quality scores. Open datasets cannot provide this documentation retroactively.

Format Compatibility and Integration Cost

Open datasets use inconsistent formats: RLDS (Reinforcement Learning Datasets) wraps TensorFlow Datasets, LeRobot uses HDF5 with Hugging Face Datasets API, ROS bags store raw sensor streams, and MCAP provides a modern container for multi-modal time-series data. Converting between formats consumes 20-40 engineering hours per dataset and introduces risk of action-space misalignment or timestamp desynchronization.

Open X-Embodiment standardized on RLDS, but contributing datasets used different action space conventions: some recorded joint positions, others recorded joint velocities, others recorded end-effector poses^[1]. The aggregation process normalized action spaces to a common representation, but this normalization discards information (e.g., converting joint velocities to positions loses acceleration data) and introduces quantization error.

Custom collection projects specify format and action space at intake, eliminating conversion cost. Truelabel delivers data in client-specified format (RLDS, LeRobot HDF5, MCAP, or raw ROS bags) with action spaces matched to client robot kinematics. A 2024 mobile manipulation project received 9,000 trajectories in LeRobot format with 7-DOF joint positions, 6-DOF end-effector poses, and 2-channel gripper state, ready for training without conversion.

BridgeData V2 provides a reference implementation for RLDS conversion, but adapting it to a new robot platform requires modifying action space dimensions, sensor topic names, and timestamp alignment logic. Custom collection eliminates this friction by delivering data in the exact format your training pipeline expects.

Hybrid Strategies: Pretraining on Open Data, Fine-Tuning on Custom

The highest-performing physical AI systems combine open-data pretraining with custom fine-tuning. RT-2 pretrained a vision-language-action model on 130,000 demonstrations from RT-1 plus web-scale vision-language data, then fine-tuned on 6,000 task-specific demonstrations to achieve 62% success on novel instructions^[7]. The pretraining phase learned generalizable visual reasoning (object recognition, spatial relationships, action affordances); fine-tuning adapted these representations to task-specific action sequences.

OpenVLA demonstrates the same pattern: pretraining on 970,000 Open X-Embodiment trajectories provided a strong initialization for vision and language encoders, but task success rates improved 30-50 percentage points after fine-tuning on 1,000-5,000 task-specific demonstrations^[6]. The fine-tuning data came from controlled environments with systematic variation in object placement, lighting, and distractor objects—variations absent from the open pretraining data.

The hybrid strategy works when: (1) your task shares visual or action primitives with open datasets (e.g., pick-and-place, push, pull), (2) you have budget for 1,000-10,000 custom trajectories but not 50,000+, (3) your deployment environment differs from open dataset environments but not radically (e.g., office kitchen vs lab kitchen is manageable; warehouse vs kitchen is not). For tasks with no open-data analogs (e.g., cable routing, soft-object manipulation, bimanual assembly), training from scratch on 8,000-15,000 custom trajectories is faster than pretraining on mismatched data.

Truelabel's marketplace supports hybrid workflows: clients specify which open datasets to use for pretraining, then commission custom collection for fine-tuning with environment and task parameters matched to deployment. A 2025 retail manipulation project pretrained on DROID, then fine-tuned on 5,000 custom trajectories collected in 8 retail stockrooms, achieving 79% success on novel product SKUs.

Egocentric Video as a Custom Collection Alternative for World Models

Egocentric video datasets like Ego4D (3,670 hours from 931 participants across 74 locations) and EPIC-KITCHENS-100 (100 hours of kitchen activities) provide large-scale human demonstration data for world model pretraining, but lack the action labels and proprioceptive state required for direct policy learning^[8]. NVIDIA's Cosmos world foundation models use egocentric video to learn physics priors and object dynamics, then transfer these priors to robot policies trained on smaller action-labeled datasets.

Custom egocentric video collection fills the gap between open datasets and robot teleoperation. A 2024 Claru kitchen task project collected 160 hours of egocentric video across 40 home kitchens with GoPro Hero 12 cameras, capturing 12,000 task executions (meal prep, dishwashing, appliance operation) with environment variation in cabinet layout, countertop height, and lighting. The video data was used to pretrain a world model, then 4,000 robot teleoperation trajectories were collected in a subset of the same kitchens to train an action policy.

The advantage: egocentric video is 10-20x cheaper to collect than robot teleoperation (no robot hardware, no teleoperation interface, faster collection cadence), making it cost-effective for environment coverage. The DROID team collected 76,000 robot trajectories at estimated cost of $1.2M (hardware, operator time, lab access); an equivalent egocentric video dataset with 760,000 task executions would cost $60K-120K. For applications where world model pretraining provides value (long-horizon planning, sim-to-real transfer, multi-task generalization), custom egocentric video + targeted robot teleoperation is the optimal cost-performance strategy.

Truelabel's marketplace supports egocentric video collection with hardware provisioning (GoPro, smartphone, head-mounted rigs), collector certification, and annotation pipelines for action segmentation and object tracking.

Procurement Checklist: Evaluating Open vs Custom for Your Project

Use open datasets when: (1) your task appears in an existing dataset with >500 demonstrations, (2) deployment environment is similar to dataset environments (lab, office, or home), (3) you need data in <2 weeks for prototyping, (4) acceptable task success rate is <70%, (5) you have engineering capacity to handle format conversion and action-space alignment.

Use custom collection when: (1) your task is novel or requires >75% success rate, (2) deployment environment differs systematically from open datasets (industrial, outdoor, retail, healthcare), (3) you need provenance documentation for enterprise procurement or regulatory compliance, (4) you require specific embodiment, sensor suite, or action space, (5) you need commercial use rights without licensing ambiguity.

Use hybrid (open pretraining + custom fine-tuning) when: (1) your task shares visual or action primitives with open datasets but requires environment-specific adaptation, (2) you have budget for 1,000-10,000 custom trajectories, (3) you want to reduce training time by starting from a pretrained vision encoder, (4) open datasets provide 60-80% of required task coverage.

Key questions for vendors: Does the dataset include trajectory-level provenance (collector ID, timestamp, quality score)? What license applies to each component dataset? Are action spaces normalized, and if so, what information was discarded? What percentage of trajectories have incomplete task executions or sensor dropout? Can you provide environment metadata (lighting, clutter, surface materials) for scene diversity analysis?

Truelabel's marketplace provides transparent answers to all five questions, with per-trajectory metadata, explicit commercial use grants, and quality guarantees. Request a custom collection quote with your task specification, embodiment requirements, and target success rate—we'll deliver a cost and timeline estimate within 48 hours.

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

External references and source context

1. Open X-Embodiment: Robotic Learning Datasets and RT-X Models

   Open X-Embodiment aggregates 1M+ trajectories from 22 embodiments across 527 skills

   arXiv ↩
2. Hugging Face organization

   AgiBot World team found Open X-Embodiment models constrained to short-horizon tasks

   Hugging Face ↩
3. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

   DROID provides 76K trajectories from Franka Panda across 86 tasks in 564 scenes

   arXiv ↩
4. Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100

   EPIC-KITCHENS-100 inter-annotator agreement 0.67 kappa for manipulation verbs

   arXiv ↩
5. EPIC-KITCHENS-100 annotations license

   EPIC-KITCHENS annotations prohibit commercial use without separate agreement

   GitHub ↩
6. OpenVLA: An Open-Source Vision-Language-Action Model

   OpenVLA 7B vision-language-action model pretrained on 970K trajectories

   arXiv ↩
7. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

   RT-2 vision-language-action model 6K demonstrations in 15 office kitchens

   arXiv ↩
8. Ego4D: Around the World in 3,000 Hours of Egocentric Video

   Ego4D 3,670 hours egocentric video from 931 participants

   arXiv ↩

FAQ