Physical AI Data Engineering
RGB-D Datasets for Robot Manipulation
RGB-D data for robot manipulation pairs each color frame with a spatially aligned depth map, giving a grasping policy the 3D geometry it needs to plan contact. Start from public sets that already ship depth: DROID (76,000 stereo-depth trajectories on a Franka arm), BridgeData V2 (60,096 trajectories with an over-the-shoulder RGB-D view), and DEX-YCB for hand-object grasping. When your embodiment, task, or commercial-use rights diverge from those sets, commission custom capture and take delivery in RLDS, LeRobot, or MCAP with the depth channel and camera intrinsics intact.
Quick facts
- Topic
- Sourcing RGB-D datasets for robot manipulation
- Audience
- Robotics / ML engineers, procurement leads
- Deliverable
- Dataset shortlist + custom-capture spec + RGB-D delivery acceptance checklist
Why manipulation policies need the depth channel, not just RGB
A gripper closes on geometry, not pixels. When a policy plans where to make contact, it needs the metric distance to the surface it is about to touch, and a color image alone does not carry that. Two mugs that look identical in RGB can sit 8 cm apart in depth, and a policy that never saw the depth channel will confidently miss the near one. That is why RGB-D is the default modality for indoor manipulation: the aligned depth map gives per-pixel distance, and back-projecting it through the camera intrinsics yields a point cloud you can reason about in 3D.
Depth earns its keep hardest on contact-rich tasks — insertion, stacking, tool use, anything where a few millimeters decides success or a jam. It also disambiguates occlusion: when one object is partly behind another, depth tells you which is in front. And it makes grasp sampling tractable, because a 6-DoF grasp planner works over surface normals estimated from the point cloud rather than guessing pose from a single view.
The practical consequence for sourcing is simple. If you buy or download a manipulation dataset that is RGB-only and then discover your task needs depth, you cannot recover it after the fact — depth has to be captured (or, at a quality cost, estimated) at collection time. So the first question to ask of any candidate dataset is not how many trajectories it has, but whether the depth channel exists, how it was produced, and how much of the data actually carries it.
How the depth channel gets produced — and why it matters for what you buy
There is no single "depth sensor." There are four ways a depth map lands in a dataset, and each fails differently.
Passive stereo computes disparity between two calibrated cameras. It works outdoors and at range, degrades on textureless surfaces (a blank white table is its worst case), and its accuracy falls off with distance squared. DROID's rig is passive stereo: a Franka Panda 7DoF arm with two ZED 2 cameras for the scene view and a wrist-mounted ZED Mini, so every DROID depth map is triangulated, not measured[1].
Active stereo adds a projected IR pattern so stereo matching works on textureless surfaces. The Intel RealSense D455 is the workhorse here — cheap, USB, easy to mount on a rig — which is why it shows up across so many robot-lab captures.
Time-of-flight and structured light (Azure Kinect, older Kinect) measure depth directly and give clean indoor maps, but interfere with each other in multi-camera setups and struggle with sunlight and specular or transparent objects. Glass and shiny metal read as holes in the depth map regardless of sensor.
Monocular estimation infers depth from a single RGB frame with a network like Depth Anything V2. This is the fallback when a dataset was captured RGB-only. It is genuinely useful for relative structure, but treat metric claims skeptically: scale is ambiguous from one view, and a policy trained on estimated depth inherits the estimator's errors on exactly the thin, reflective, and near-field surfaces manipulation cares about.
Why this matters at purchase time: the production method determines where the depth is trustworthy. Stereo data is fine at a meter but noisy up close; ToF is crisp indoors but blind in sun; monocular is a stopgap, not a sensor. Match the method to your task before you match the task label.
Public RGB-D manipulation datasets worth starting from
Do not commission capture for data that already exists. Three public sets cover a large fraction of tabletop and hand-object manipulation, and each ships depth — with caveats worth reading before you download 400 GB.
DROID is the broadest real-robot manipulation set: 76,000 demonstration trajectories, about 350 hours, across 564 scenes and 84 tasks, gathered by 50 collectors over 12 months[2]. Single embodiment (Franka Panda), stereo depth from the ZED cameras, and real scene diversity make it a strong pretraining base. The catch is the same as its strength: everything is one arm, so cross-embodiment transfer is on you.
BridgeData V2 gives 60,096 trajectories across 24 environments on a low-cost WidowX 250 arm[3]. It has an over-the-shoulder RGB-D camera — but here is the gotcha that burns teams: the majority of episodes include only the primary fixed view, and very little of the data includes all four camera views[4]. If your model assumes a dense multi-view depth stream on every episode, BridgeData V2 will not give it to you uniformly. Audit the per-episode view coverage before you build a loader around it.
DEX-YCB is the one to reach for when the task is grasping and hand-object interaction: it captures hand grasping of YCB objects from 10 subjects with multiple calibrated RGB-D cameras, and ships benchmarks for 6D object pose and 3D hand pose[5]. It is a perception and grasp-planning resource more than a policy-rollout set — different job, so slot it accordingly.
When you want scale beyond any single set, Open X-Embodiment aggregates data from 22 embodiments across 21 institutions spanning 527 skills[6] — but depth availability and format are wildly uneven across its member datasets, so treat it as a menu to filter, not a uniform corpus. And for controlled coverage of rare configurations, simulators render perfect RGB-D on demand: ManiSkill, CALVIN, and robomimic all emit depth, at the price of the sim-to-real gap you then have to close.
| Dataset | Embodiment | Depth source | Best for | Watch out for |
|---|---|---|---|---|
| DROID | Franka Panda 7DoF | Passive stereo (ZED) | Broad real-world pretraining | Single embodiment |
| BridgeData V2 | WidowX 250 6DoF | Over-the-shoulder RGB-D | Low-cost tabletop skills | Most episodes = primary view only |
| DEX-YCB | Human hands + YCB objects | Multi-view RGB-D | Grasp / 6D pose / hand pose | Perception set, not policy rollouts |
| ManiSkill / CALVIN / robomimic | Simulated arms | Rendered depth | Rare configs, perfect labels | Sim-to-real gap |
When public data runs out: commissioning custom RGB-D capture
Public sets stop being enough at a predictable moment: your embodiment is not a Franka or a WidowX, your task distribution is not on the menu, or your legal team will not clear a research-only license for a commercial model. That is the boundary where sourcing shifts from download to capture; the depth data for robotics sourcing hub lays out the broader choice among public corpora, learned depth, commodity sensors, LiDAR, and custom collection.
A custom RGB-D manipulation spec is worth writing down before anyone records a frame. Pin the sensor class to the task (active stereo for cluttered indoor tabletops, ToF where lighting is controlled, stereo where you need range), and state the depth resolution and frame rate you need aligned to RGB. Require intrinsics and extrinsics per camera and per session — without them the depth map is unprojectable and the whole capture is worthless for 3D reasoning. Specify calibration cadence, because a rig that drifts mid-session silently corrupts every downstream point cloud. Define how transparent and specular objects (the depth-hole cases) are handled: flagged, re-shot, or annotated. And set the sync tolerance between RGB, depth, and proprioception; a manipulation policy needs streams registered inside roughly a 10 ms window, not "close enough."
This is the work Truelabel's physical AI data marketplace is built for. You post a spec, vetted capture partners return sample packets before any scale commitment, and delivery is rights-cleared — contributor consent and per-trajectory provenance attached, so the data survives a commercial-use audit instead of stranding a model in legal review. Truelabel draws on around 10,000 consented collectors across 100 countries, which is the practical way to get depth data spanning real homes, factories, and object distributions rather than one lab bench.
Delivery: getting depth into your training loop without a rewrite
Depth data that arrives in the wrong shape costs you a week of ETL before a single training step. Fix the format at the spec, not after delivery.
Three schemas cover almost everything. RLDS stores episodes as nested observation dictionaries, so a depth channel lives beside RGB in the same step record — the format behind DROID and Open X-Embodiment. LeRobot stores per-step observation frames with camera metadata and is the path of least resistance if you train with the Hugging Face stack. MCAP is the choice for ROS 2 teams: it carries synchronized RGB and depth streams with per-message timestamps, which preserves the alignment a manipulation policy depends on. For custom pipelines, HDF5 packs RGB frames, depth arrays, and metadata into one random-access file.
Beyond the container, nail down the depth encoding itself. Depth ships either as uint16 millimeters or float32 meters, and mixing them up scales your whole scene by 1000. Confirm the invalid-pixel sentinel (zero versus NaN) so your loader masks holes instead of training on them. Keep depth aligned to the RGB frame, not the raw depth-sensor frame, or every pixel correspondence is off. Then back-projection to a metric point cloud is just the intrinsics and a matrix multiply, exactly as the Point Cloud Library documents — and from there you can run the 3D annotation passes that grasp labels need.
- 01
Depth channel present and populated
Confirm depth exists on the fraction of episodes you actually need — not just that the schema has a depth field.
- 02
Camera intrinsics and extrinsics included
Per camera, per session; without them the depth map cannot be projected to metric 3D.
- 03
Encoding and units documented
uint16 mm vs float32 m, plus the invalid-pixel sentinel (0 or NaN) so holes get masked.
- 04
RGB-depth alignment and sync verified
Depth registered to the RGB frame and timestamped within roughly a 10 ms window across streams.
- 05
License clears your use
Research-only and non-commercial datasets block commercial model training — resolve this before you build on the data.
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
External references and source context
- Project site
DROID hardware is a Franka Panda 7DoF arm with two ZED 2 stereo cameras and one wrist-mounted ZED Mini stereo camera, so its depth is stereo-derived
droid-dataset.github.io ↩ - DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
DROID contains 76,000 demonstration trajectories (350 hours) across 564 scenes and 84 tasks, collected by 50 data collectors over 12 months
arXiv ↩ - BridgeData V2: A Dataset for Robot Learning at Scale
BridgeData V2 contains 60,096 trajectories collected across 24 environments on a low-cost robot arm
arXiv ↩ - Project site
BridgeData V2 uses a WidowX 250 6DoF arm with one over-the-shoulder RGB-D camera plus RGB cameras; most episodes carry only the primary fixed view and few include all four
rail-berkeley.github.io ↩ - Project site
DEX-YCB captures hand grasping of YCB objects from 10 subjects with multiple calibrated RGB-D cameras, with benchmarks for 6D object pose and 3D hand pose
dex-ycb.github.io ↩ - Open X-Embodiment: Robotic Learning Datasets and RT-X Models
Open X-Embodiment aggregates data from 22 robot embodiments across 21 institutions spanning 527 skills
arXiv ↩
FAQ
What depth sensor should custom RGB-D manipulation data use?
Match the sensor class to the task. Active-stereo cameras like the Intel RealSense D455 are the default for cluttered indoor tabletops because the projected IR pattern gives depth even on textureless surfaces. Passive stereo (as in DROID's ZED rig) works outdoors and at range but is noisy up close. Time-of-flight and structured-light sensors give clean indoor maps but interfere in multi-camera setups and fail in sunlight. No sensor reads glass or shiny metal reliably — plan for depth holes on transparent and specular objects.
Is monocular depth estimation good enough for manipulation instead of a depth sensor?
As a stopgap, not a substitute. Models like Depth Anything V2 recover useful relative structure from a single RGB frame, which helps when a dataset was captured RGB-only. But metric scale is ambiguous from one view, and estimated depth is least reliable on thin, reflective, and near-field surfaces — precisely the cases contact-rich manipulation depends on. If depth is load-bearing for grasp planning, capture it with a sensor rather than estimating it after the fact.
Which public RGB-D dataset is best for a manipulation pretraining base?
DROID is the strongest single starting point for real-robot breadth: 76,000 trajectories across 564 scenes and 84 tasks with stereo depth, all on a Franka Panda. BridgeData V2 adds 60,096 low-cost WidowX trajectories but ships dense depth on only a subset of episodes. DEX-YCB is the pick for grasping and hand-object perception rather than policy rollouts. Most teams pretrain on a broad set and then fine-tune on a few thousand task-specific trajectories captured for their own embodiment.
Does BridgeData V2 include depth on every episode?
No. BridgeData V2 has an over-the-shoulder RGB-D camera, but the majority of episodes include only that primary fixed view, and very little of the data includes all four camera views. If your loader assumes a dense multi-view depth stream on every episode, audit the per-episode view coverage first — this is a common source of silent training bugs.
What format should RGB-D manipulation data be delivered in?
RLDS for TensorFlow-style episodic pipelines (it stores depth beside RGB in nested observation dicts and backs DROID and Open X-Embodiment), LeRobot if you train on the Hugging Face stack, and MCAP for ROS 2 teams that need synchronized RGB and depth with per-message timestamps. HDF5 suits custom pipelines. Whichever you choose, require camera intrinsics/extrinsics, a documented depth encoding (uint16 mm or float32 m), and depth aligned to the RGB frame.
Can I use public RGB-D datasets like DEX-YCB for a commercial robot product?
Check the license per dataset — this is the step teams skip and regret. Many academic RGB-D datasets carry research-only or non-commercial terms that block training weights you intend to ship. When a needed dataset is non-commercial, or when your embodiment and task diverge from what is public, the clean path is custom capture under a buyer-owned commercial license with consent and provenance attached, which clears legal review at first pass.
Looking for RGB-D datasets for robot manipulation?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request RGB-D data for manipulation on Truelabel