How to Evaluate Robotics Data Vendors

The short answer: score seven axes against a sample, not a slide

A procurement committee evaluating robotics data vendors is not buying hours of footage. It is buying a claim that those hours will move a policy or a benchmark, and that claim is only testable on your task, in your schema, against your acceptance rules. Every vendor deck leads with a total — hours captured, trajectories collected, collectors on the network. Treat that total as the least informative number in the room.

The reason is structural. The largest open real-robot dataset in existence, Open X-Embodiment, reaches its 1M+ trajectories across 22 embodiments only by pooling 60 separate datasets from 34 research labs. That corpus is a triumph of aggregation, but it is also a warning: a big number is usually a mosaic of inconsistent capture conditions, sensor stacks, labeling conventions, and rights regimes. When a vendor quotes you a similar aggregate, the useful question is not "how much" but "how consistent" — and the only way to answer it is to score the vendor on the axes below and then buy a small, paid sample packet before you commit budget.

Why headline volume is the wrong first question

Aggregate size tells you almost nothing about fit, and it can actively mislead. Two vendors can both claim "10,000 hours" while one delivers a narrow, internally consistent collection captured on one rig with one consent regime, and the other delivers a heterogeneous pile stitched from dozens of sources with mismatched timing, labeling, and licenses. The second number is larger and worse.

The published research bears this out. The team behind ManipArena — a standardized real-robot evaluation of 20 tasks, 10,812 expert trajectories, and 13.5M frames — reports that real-robot outcomes depend not only on model architecture but on model provenance, fine-tuning regime, data sampling, and annotation granularity. In other words, the properties that decide whether data helps are exactly the ones a headline hour-count hides. The same paper names calibration error and latency as first-class sources of the reality gap, which is the strongest possible argument for asking a vendor about clocks and calibration before you ask about scale.

• Two collections of equal size can differ 10x in usable yield after your acceptance test rejects mismatched episodes.
• Aggregated corpora inherit the weakest rights regime in the pool — one non-commercial source can taint a commercial program.
• Provenance, sampling, and annotation granularity are the levers that move evals; hour-counts are not.

The seven-axis rubric

A rubric only works if every vendor is scored on the same axes with the same weighting, decided before the first call. Weight the axes to your program: a team fine-tuning a deployed manipulation policy weights sensor fidelity and provenance heavily; a team pretraining and running ablations may weight scale and diversity more. Score each axis 1 to 5 from evidence the vendor shows, not claims they make — a weak vendor answers at the level of a brochure, a strong one shows a rejected episode and explains why it was rejected.

Sensor and synchronization fidelity: ask about clocks before scale

This is the axis most procurement processes skip and most policies fail on. A manipulation or teleoperation demonstration is only useful if its observation and action streams are aligned tightly enough that a controller can learn the mapping between them. When RGB frames, depth, proprioception, and gripper state drift apart by even tens of milliseconds, the episode teaches the wrong correspondence, and the failure surfaces later as a policy that looks fine offline and misses grasp transitions on the bench.

The container format is the tell. A vendor delivering MCAP or an equivalent timestamped, multi-channel log is treating synchronization as a first-class property; a vendor delivering a folder of MP4s and a CSV is not. Ask three concrete questions: what clock are all streams stamped against, what is the stated synchronization tolerance, and how is each rig calibrated and how often is calibration re-verified. ManipArena naming calibration error and latency as reality-gap sources is not academic hand-wringing — it is the reason a synchronized delivery is worth more per hour than an unsynchronized one.

• Common clock: every modality timestamped against one source, not per-device wall clocks.
• Stated tolerance: a number for acceptable cross-stream drift, not 'they line up'.
• Calibration discipline: documented intrinsics/extrinsics and a re-verification cadence, not a one-time setup.

Provenance and rights: use DROID as the granularity bar

Provenance is where teams get burned after the money is spent, when legal or a partner asks a question the vendor cannot answer. The bar for what "traceable" means is public. The DROID dataset documents its 76,000 trajectories down to 564 scenes, 86 tasks, 50 named data collectors, and 13 institutions over 12 months. That is the level of decomposition a serious vendor can produce on demand: which session, which operator, which environment, under which consent. If a vendor can quote you a total but cannot break it down along those lines, the collection is not auditable, and an unauditable collection is a compliance liability regardless of its technical quality.

Rights are inseparable from provenance. A commercial robotics program needs per-trajectory consent artifacts, location releases where people or private spaces appear, and usage scope documented at the session level — not a blanket assurance that "everything is cleared." Buying only shifts the rights burden off your team if the vendor surfaces a clean chain; if they do not, you have simply moved the liability without resolving it. Ask to see the actual consent artifact and metadata record that ships with a single delivered episode.

Format and delivery: demand a sample in your exact schema

Format mismatch is the quietest failure mode in the whole evaluation, because a "close enough" delivery imports without an error and then silently drops the field your training loop needed. The defense is specific: before you approve scale, require one sample episode in the exact structure you will train on. If your stack expects RLDS, the episode should arrive as a nested dataset of steps that preserves observations, actions, rewards, and per-step metadata — not a flattened export you have to reconstruct. If you standardize on LeRobot, ask for a LeRobot dataset you can load with the library, not a folder you have to convert.

The reason to insist on your schema rather than the vendor's preferred one is reproducibility. A conversion path you maintain is a conversion path that breaks on the next delivery, and format drift between batches is a common source of the "the data changed and we can't tell why" problem. A vendor that can deliver RLDS, LeRobot, or MCAP cleanly on the first sample is demonstrating pipeline maturity; a vendor that promises to "figure out the format" during scale is deferring risk onto you.

1. 01

   Buy a small packet, not a demo reel

   Pay for a bounded sample sized to your reject conditions. A curated highlight reel tests the vendor's editing, not their capture. A paid packet tests the capture you would actually receive at scale.

2. 02

   Run your own intake QA

   Load the sample in your exact schema, check synchronization on real episodes, verify consent artifacts and metadata resolve per trajectory, and run your acceptance rules against it as if it were a production batch.

3. 03

   Score, then decide on scale

   Record yield: what fraction of the sample passed your acceptance test unmodified. Feed the failures back while the collection is still small enough to correct, and only scale with the vendor whose sample cleared intake in your schema.

Running the evaluation, and where a marketplace fits honestly

Put the axes together into a workflow a committee can defend. Write the spec, score every candidate on the same seven axes with weights fixed in advance, buy a paid sample from the top two or three, run identical intake QA on each, and let measured yield in your schema break the tie. Unit cost anchors the budget conversation but should never lead it: reported ranges put complex multi-sensor manipulation capture at roughly $80 to $150 per hour and simple 2D teleoperation at $15 to $30 per hour, and a cheaper hour that fails your acceptance test is the most expensive data you can buy, because it forces a second collection.

A marketplace is one honest answer to the sourcing problem, and it is worth being precise about what it does and does not solve. A marketplace does not remove the need for this rubric — you still write the spec, still score suppliers, still run the sample test. What it changes is the search and vetting cost: instead of qualifying capture partners one at a time, you post a spec once and matched suppliers return samples against it. Truelabel is built for exactly that buy-side workflow: a physical-AI data marketplace where robotics and embodied-AI teams post a spec and receive rights-cleared egocentric, exocentric, teleoperation, and directed captures from 100+ vetted partners drawn from around 10,000 collectors across 100 countries, with sample packets, consent artifacts, per-trajectory provenance, and RLDS, LeRobot, or MCAP delivery to S3, GCS, or Azure — so the seven axes above are what you evaluate on, not what you have to chase down.

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

FAQ