Buyer intake
Depth data for robotics
Depth data for robotics is per-pixel distance information — RGB-D frames, LiDAR point clouds, or learned monocular depth maps — that tells a policy how far every surface is, not just what it looks like. Robots use it for grasp planning and collision checking in manipulation, and for obstacle avoidance and SLAM in navigation. It comes in three shapes with very different accuracy, cost, and licensing: sensor RGB-D from active-stereo or structured-light cameras (Intel RealSense, Kinect v2, Zed), sparse LiDAR point clouds, and dense monocular depth predicted from ordinary RGB by models like Depth Anything V2. Most public RGB-D corpora are research-licensed, so commercial training usually needs custom capture with consent and commercial rights. Truelabel matches buyer depth specs to 100+ vetted capture partners and delivers depth-channel data in RLDS, LeRobot, and MCAP with camera intrinsics/extrinsics, per-frame color-depth alignment, and rights-cleared provenance.
Quick facts
- Request type
- Off-the-shelf or NET_NEW capture
- Modality
- Aligned RGB-D (color + depth) + camera intrinsics/extrinsics
- Sensor class
- Active stereo / structured light / stereo (RealSense, Kinect v2, Zed)
- Environment
- Warehouse aisles, kitchen counters, sidewalks, cluttered tabletops
- Rights
- Commercial training license with consent artifacts
- Delivery
- Depth channel in RLDS/LeRobot/MCAP to your S3, GCS, or Azure bucket
Comparison
| Where to get depth data | Best for | Watch out |
|---|---|---|
| Public RGB-D datasets (DIML, NYU-Depth, ScanNet) | Pretraining, ablations, benchmarks | Research/NC licenses; fixed scenes; no commercial rights |
| Learned monocular depth (Depth Anything V2) | Dense depth from RGB you already have | Relative, not metric, scale; needs calibration before grasp planning |
| Commodity depth cameras (RealSense / Kinect v2 / Zed) | In-house capture you fully control | Depth dropouts on reflective/transparent surfaces; range limits |
| LiDAR / point clouds | Outdoor navigation, long range, mobile robots | Sparse returns, higher cost, harder to fuse with dense RGB |
| Truelabel custom capture | Your embodiment, task distribution, and commercial rights | Requires a clear spec and sample-packet QA before scale |
What counts as depth data for robotics?
Depth data answers a question RGB alone cannot: how far away is each surface? A pixel value of "0.62 m" on the mug handle is what lets a policy decide where to close the gripper — colour tells it what the object is, depth tells it where. Three representations dominate, and buyers routinely conflate them.
RGB-D is a colour frame plus a per-pixel depth frame, aligned to the same camera model. It is the default for tabletop manipulation because it is dense, cheap, and directly consumable by a policy network. Point clouds are unordered sets of 3D points, usually from LiDAR or fused stereo — the canonical 3D representation standardized in the Point Cloud Library, and the right choice for outdoor navigation and long range. Learned monocular depth predicts a dense depth map from a single ordinary RGB image; Depth Anything V2 offers models from 25M to 1.3B parameters and runs more than 10x faster than diffusion-based depth, which makes it attractive when you already have RGB and cannot re-capture with a depth rig.
The catch: monocular depth is usually relative, not metric. It tells you the mug is closer than the wall, not that the handle is 0.62 m away. For grasp planning you need metric scale — so learned depth either needs a metric fine-tune or a calibration pass against a real sensor. That distinction decides whether a dataset is usable for manipulation at all.
Where depth actually changes robot behavior
Depth is not a nice-to-have channel; it removes ambiguity that RGB physically cannot resolve. In manipulation, grasp pose selection and collision checking both live in 3D — a two-finger gripper approaching a cup needs the cup's geometry and the free space around it, not a flat image. Contact-rich tasks amplify this: the last centimetre before contact is where monocular RGB is weakest and metric depth matters most. In navigation, depth drives obstacle avoidance and feeds SLAM, where consistent metric depth across frames is what keeps the map from drifting. And for world models and video-prediction training, depth supervises geometry so the model learns that objects have extent and occlude each other, rather than treating the scene as a flat texture.
The practical implication for a data buy: the use case dictates the representation. A grasping program wants dense, metric, well-aligned RGB-D at close range. A sidewalk-delivery robot wants longer-range point clouds tolerant of sunlight. Buying the wrong representation is the most common depth-data mistake we see — a beautiful indoor RGB-D corpus is nearly useless for outdoor navigation, and vice versa.
Sensor depth vs learned depth: what you're buying
Depth data is produced two ways, and the production method is baked into the data's failure modes. Sensor depth comes from hardware: active-stereo and structured-light cameras like the Intel RealSense D455 and Kinect v2 project or triangulate to recover depth directly; passive stereo like the Zed infers it from a calibrated camera pair. Sensor depth is metric out of the box but degrades predictably — reflective and transparent surfaces (glass, polished metal, dark plastics) return garbage or voids, and every sensor has a usable range envelope. Learned depth comes from a model run over RGB; it never drops out on glass but inherits the model's biases and, for most models, only relative scale.
The table below maps the sensor and model classes onto what they cost you in practice.
| Depth source | Scale | Strength | Failure mode |
|---|---|---|---|
| Active stereo / structured light (RealSense, Kinect v2) | Metric | Dense, cheap, plug-and-play indoors | Voids on transparent/reflective surfaces; limited range |
| Passive stereo (Zed) | Metric | Works outdoors, longer range | Needs texture; weak on blank walls and low light |
| LiDAR point cloud | Metric | Long range, sunlight-robust | Sparse; expensive; harder to fuse with dense RGB |
| Learned monocular (Depth Anything V2) | Relative | Dense depth from any RGB; no rig needed | Not metric without fine-tune/calibration |
Public RGB-D datasets vs custom capture
There is real public RGB-D data, and for pretraining or ablations you should use it. The DIML/CVL RGB-D dataset spans 200+ indoor and outdoor scenes with synchronized frames from Kinect v2 indoors and a Zed stereo camera outdoors, plus per-pixel disparity confidence maps — offices, dormitories, streets, and roads, which is why researchers matching a depth-dataset spec (scene type, depth range, sensor) keep landing on it. NYU-Depth v2 and ScanNet cover dense indoor RGB-D at scale. These are excellent for learning geometry priors.
Where they stop working is deployment. Most public RGB-D corpora carry research or non-commercial licenses, their scenes are fixed (you cannot add your warehouse, your gripper camera height, your lighting), and the sensor may not match your embodiment. That is the gap custom capture fills: a program scoped to your task distribution, your camera model, and a commercial license with consent artifacts attached. The decision is not "public or custom" — it is "public for pretraining, custom for the last mile that touches a deployed model."
How depth ships in robotics formats
Depth is not a separate deliverable — it is an observation channel that rides inside the episode alongside RGB, proprioception, and actions. RLDS stores a per-step observation dictionary, so a depth map is just another key next to the RGB frame, and LeRobotDataset stores multi-camera image and state features per frame the same way. Open X-Embodiment standardizes these observation and action formats across many robots and datasets, which is why a depth channel captured to spec drops into an existing training pipeline without a bespoke loader. For log-style multimodal capture, MCAP keeps synchronized, timestamped channels together so depth stays aligned with everything else that happened in the same instant.
Two things separate usable depth delivery from a folder of PNGs. First, camera intrinsics and extrinsics must travel with the data — without the intrinsic matrix you cannot back-project a depth map into a metric point cloud, and without extrinsics you cannot fuse multiple cameras. Second, colour and depth must be alignment-verified per frame; an unregistered depth frame silently poisons grasp labels. Truelabel's deliveries carry intrinsics/extrinsics, per-frame alignment, and provenance so the depth is trainable on arrival, not after a week of cleanup.
How Truelabel sources rights-cleared depth data
Truelabel is a physical-AI data marketplace: you post a depth spec — sensor class, environment, range, alignment, commercial rights — and matched capture partners return sample packets before you fund scale. A network of around 10,000 consented collectors across 100 countries captures to spec, from cluttered kitchen counters to warehouse aisles to sidewalks, and every clip carries per-session consent and provenance so the footage is licensable for a deployed model and auditable for governance. A calibration pilot returns a first batch early; accepted batches then ship on a recurring cadence, each gated against your acceptance rubric before scale-up. Delivery is in RLDS, LeRobot, or MCAP with the depth channel, camera intrinsics/extrinsics, and metadata attached, to your S3, GCS, or Azure bucket. If you are validating a supplier or a sensor assumption, start with a small depth eval bundle before committing to a full program.
Explore the depth-data cluster
The links below group the depth library by where you are in the decision: Define (what depth and RGB-D data are), Consider (representations and models), and Decide (the sourcing marketplace and eval path).
Related pages
Use these to move from category-level context into specific task, dataset, format, and comparison detail.
FAQ
What is depth data for robotics?
Depth data is per-pixel distance information for a scene — delivered as RGB-D frames, LiDAR point clouds, or learned monocular depth maps. It tells a robot how far every surface is, which is what makes grasp planning, collision avoidance, and SLAM-based navigation possible. RGB alone cannot recover the 3D geometry these tasks depend on.
What's the difference between RGB-D data and a point cloud?
RGB-D is a dense, image-shaped representation: a colour frame plus an aligned per-pixel depth frame, best for tabletop manipulation. A point cloud is an unordered set of 3D points, usually from LiDAR or fused stereo, best for long-range outdoor navigation. RGB-D is denser and cheaper to consume; point clouds cover longer range and are more robust to sunlight but are sparser and harder to fuse with dense RGB.
Is learned monocular depth good enough for manipulation?
Usually not on its own. Models like Depth Anything V2 produce dense depth from any RGB image, but the output is typically relative, not metric — it ranks surfaces by distance without telling you the true metres. Grasp planning needs metric scale, so learned depth for manipulation needs either a metric fine-tune or a calibration pass against a real depth sensor.
Can I use public RGB-D datasets like DIML or NYU commercially?
Often not directly. Public RGB-D corpora such as DIML/CVL, NYU-Depth v2, and ScanNet are typically research- or non-commercial-licensed, and their scenes are fixed. They are excellent for pretraining and ablations, but training a deployed commercial model usually requires custom capture with explicit consent and a commercial license.
Why do depth cameras fail on glass and shiny objects?
Active-stereo and structured-light cameras recover depth from projected or triangulated patterns. Transparent surfaces let the pattern pass through and reflective surfaces scatter it, so the sensor returns noise or voids instead of a valid distance. This is why datasets for environments with glass, polished metal, or dark plastics benefit from edge-case coverage and, sometimes, multi-sensor fusion.
What formats does Truelabel deliver depth data in?
Depth ships as an observation channel inside the episode — in RLDS, LeRobot, or MCAP — alongside RGB, proprioception, and actions, with camera intrinsics/extrinsics and per-frame color-depth alignment included. Delivery goes to your S3, GCS, or Azure bucket with per-session provenance and consent artifacts attached.
How do I request depth or RGB-D data?
Post a spec describing the sensor class, environment, depth range, alignment requirements, and commercial rights you need. Matched capture partners return a sample packet for review before you fund scale. If you are validating a supplier or a sensor assumption first, request a small depth eval bundle to test quality before committing to a larger program.
Looking for depth data for robotics?
Specify modality, task, environment, rights, and delivery format. Truelabel matches you with vetted capture partners and helps scope consent artifacts and commercial licensing requirements before delivery.
Request depth/RGB-D data