How to Label Grasp Success and Failure in Robot Manipulation Data

Quick facts

Difficulty

Intermediate

Audience

Physical AI data engineers

Last reviewed

2026-01-15

Why Grasp Outcome Labels Are Critical for Manipulation Policy Training

Grasp success and failure labels are the ground truth that drives policy learning in manipulation tasks. Without explicit outcome annotations, reinforcement learning agents cannot distinguish productive actions from unproductive ones, and imitation learning models inherit failure modes from demonstration data without correction signals. RT-1 trained on 130,000 demonstrations with success labels achieved 97% task completion on novel objects; ablations without outcome labels dropped to 63% success^[1].

Modern manipulation datasets encode grasp outcomes as boolean episode-level or step-level metadata. DROID's 76,000 trajectories include per-episode success flags derived from human teleoperator confirmation plus automated gripper state checks^[2]. Open X-Embodiment aggregates 22 datasets with 1M+ episodes, standardizing success labels across heterogeneous robot platforms and task definitions. The BridgeData V2 dataset labels 60,000 trajectories with binary success outcomes, enabling policy models to learn from both positive and negative examples.

Grasp labeling complexity scales with task diversity. Pick-and-place tasks require binary success (object placed in target zone within tolerance), while multi-step assembly tasks need intermediate grasp success labels at each sub-goal. CALVIN's long-horizon tasks annotate success at each of 34 sub-tasks, creating a sparse reward signal that guides hierarchical policy learning. Production labeling pipelines must balance annotation cost (human review time) against signal quality (inter-annotator agreement), targeting 95%+ agreement on binary outcomes and 85%+ on multi-class edge cases^[3].

Defining Binary Success Criteria for Grasp Tasks

Effective grasp success criteria must be observable, measurable, and task-aligned. The most common definition: object lifted above a height threshold (typically 5-10cm above the table surface) and held stable for a minimum duration (0.5-2 seconds) without slippage. This definition maps cleanly to sensor signals—gripper encoder confirms closure, force-torque sensor confirms load, and depth camera or object tracker confirms height change.

RT-2's training data uses a three-condition AND gate: gripper closed beyond 80% of range, vertical force exceeds object weight plus 20% margin, and object centroid Z-coordinate increases by ≥8cm within 3 seconds of grasp initiation^[4]. LeRobot's episode format stores success as a boolean `episode_data_index.success` field, with optional `success_time` timestamp marking the frame where criteria were first satisfied.

Task-specific criteria extend the base definition. Precision grasps (peg insertion, connector mating) require pose tolerance checks: object orientation within ±5° of target and position within ±2mm. Power grasps (bin picking, palletizing) relax pose constraints but add stability requirements: object held through a 10cm vertical displacement without re-grasping. Dex-YCB's hand-object interaction dataset labels grasp quality on a 5-point scale (failed, unstable, marginal, stable, optimal) based on contact area and force distribution, then binarizes to success/failure at the 'marginal' threshold for policy training.

Edge case definitions prevent label ambiguity. Partial success: object lifted but dropped before placement—label as failure with `failure_mode: 'dropped'` metadata. Re-grasp: initial grasp fails, robot adjusts and succeeds on second attempt—label episode as success with `num_attempts: 2` and annotate the failure frame. Assisted grasp: human intervenes to stabilize object—label as failure with `failure_mode: 'human_intervention'`. These structured failure modes enable targeted data augmentation and policy debugging^[5].

Implementing Automated Grasp Detection from Sensor Streams

Automated grasp detection pipelines fuse multi-modal sensor data to generate candidate success labels, reducing human review load by 70-85% while maintaining 92-96% precision on binary outcomes^[3]. The standard architecture: sensor preprocessing (calibration, synchronization, noise filtering) → feature extraction (gripper state, object pose, force profile) → rule-based classifier (threshold logic) → confidence scoring (flag low-confidence cases for human review).

Gripper state detection is the primary signal. Parallel-jaw grippers report encoder position; under-actuated grippers report motor current. Franka FR3's gripper publishes joint positions at 1kHz; a grasp is detected when both fingers exceed 15mm closure and remain stable (±0.5mm) for 0.3 seconds. Suction grippers use vacuum pressure sensors: grasp confirmed when pressure drops below -40kPa (indicating seal) and remains stable. The UMI gripper's force-torque sensor adds a 6-axis wrench signal; vertical force ≥1.5× object weight confirms load transfer.

Visual confirmation reduces false positives from gripper contact without object capture. Segments.ai's point cloud labeling tools enable 3D bounding box annotation of grasped objects; automated pipelines run PointNet segmentation on depth camera point clouds to detect object presence in gripper workspace. A grasp is visually confirmed when ≥60% of the object's bounding box volume intersects the gripper's convex hull for ≥5 consecutive frames (0.17 seconds at 30 FPS).

Temporal consistency checks filter transient sensor noise. A candidate grasp must satisfy all criteria (gripper closed, force threshold, visual confirmation) for a minimum duration—typically 10-20 frames. RLDS episode format stores per-step `is_terminal` and `reward` fields; grasp detection logic sets `reward=1.0` at the first frame where the stability window completes. The LeRobot dataset schema adds `observation.state.gripper_position` and `action.gripper_command` arrays, enabling post-hoc re-labeling if detection thresholds are refined.

Handling Edge Cases: Partial Grasps, Slippage, and Re-Attempts

Edge cases account for 12-18% of manipulation episodes in real-world datasets and require structured labeling protocols to maintain training signal quality^[2]. The three most common failure modes: partial grasp (object contacted but not secured), slippage (initial grasp succeeded but object dropped during transport), and re-attempt (robot adjusts strategy after initial failure). Each requires distinct metadata encoding to enable targeted policy improvements.

Partial grasps occur when the gripper closes on object edges or corners without achieving stable contact. Detection: gripper encoder shows closure but force-torque sensor reads <50% of expected object weight, or visual tracking shows object rotation/translation during lift attempt. Label as `success: false` with `failure_mode: 'partial_grasp'` and annotate the contact frame. RoboNet's 15M frames include 8% partial grasp episodes; policies trained with these negative examples improved grasp success rate by 14 percentage points on novel objects^[6].

Slippage events are detected via force-torque discontinuities or visual tracking loss. A grasp initially satisfies success criteria (object lifted, stable hold) but then fails mid-transport. Detection: vertical force drops by >30% over <0.5 seconds, or object centroid moves >5cm from gripper center. Label the episode as `success: false` with `failure_mode: 'slippage'`, record `success_duration` (time from grasp to slip), and annotate both the initial success frame and the slip frame. BridgeData V2 includes 4% slippage episodes with frame-level annotations, enabling policies to learn corrective re-grasping strategies.

Re-attempt sequences require episode segmentation. If the robot fails an initial grasp, adjusts its approach, and succeeds on attempt 2 or 3, label the overall episode as `success: true` with `num_attempts: N` and create sub-episode annotations for each attempt. DROID's annotation protocol segments re-attempts into separate trajectories when the gripper fully opens (reset signal) between attempts, preserving the failure-then-success sequence for policy learning. CloudFactory's industrial robotics labeling routes re-attempt episodes to expert annotators who mark the decision point where strategy changed, creating a richer training signal for adaptive policies.

Building Quality Assurance Pipelines for Consistent Annotations

Production grasp labeling pipelines require multi-stage QA to achieve 95%+ inter-annotator agreement on binary outcomes and 85%+ on edge case classifications^[3]. The standard QA architecture: automated pre-labeling (rule-based detection generates candidate labels) → human review (annotators confirm or correct) → consensus validation (multi-annotator agreement on gold-standard subset) → feedback loop (disagreements refine detection rules).

Automated pre-labeling reduces annotation time by 60-75% by presenting human reviewers with high-confidence predictions rather than raw sensor data^[7]. Labelbox's workflow automation runs custom Python scripts on uploaded episodes, populating label fields with rule-based predictions and confidence scores. Episodes with confidence >0.9 auto-approve; 0.7-0.9 route to single-annotator review; <0.7 route to dual-annotator consensus. Scale AI's physical AI data engine applies similar tiering, processing 50,000+ manipulation episodes per week with 8-12 human reviewers.

Consensus validation establishes ground truth on a 5-10% holdout set. Three independent annotators label the same episodes; labels with 100% agreement become gold-standard references. Disagreements trigger review meetings where annotators discuss edge cases and refine labeling guidelines. Encord's consensus workflows track per-annotator agreement rates over time, flagging annotators who drift below 90% agreement for retraining. Appen's data annotation platform uses a similar approach, maintaining annotator quality scores and routing difficult cases to senior reviewers.

Feedback loops continuously improve detection rules. Weekly QA reports compare automated pre-labels against human corrections, identifying systematic errors (e.g., force threshold too sensitive, causing false positives on lightweight objects). Dataloop's annotation platform version-controls labeling scripts and tracks precision/recall metrics per rule version. Truelabel's marketplace requires dataset sellers to document QA methodology and publish inter-annotator agreement statistics, enabling buyers to assess label quality before purchase. The data provenance trail links each label to the annotator ID, review timestamp, and detection rule version, supporting audit and retraining workflows.

Encoding Grasp Outcomes in Standard Dataset Formats

Grasp success labels must be encoded in machine-readable formats that preserve metadata, support efficient querying, and integrate with training frameworks. The three dominant formats for manipulation datasets: HDF5 (hierarchical storage, used by 60% of academic datasets), RLDS (Reinforcement Learning Datasets, Google's TensorFlow-native format), and LeRobot (Hugging Face's PyTorch-native format with Parquet backend)^[8].

HDF5's hierarchical structure stores episodes as top-level groups, each containing `observations/`, `actions/`, and `metadata/` subgroups. Grasp success is typically stored as `metadata/success` (boolean scalar) and `metadata/success_frame` (integer index). DROID uses HDF5 with per-episode success flags and optional `metadata/failure_mode` string attribute for edge cases. h5py's Python API enables fast filtering: `episodes = [ep for ep in f.keys() if f[ep]['metadata/success'][()] == True]` selects only successful trajectories.

RLDS (Reinforcement Learning Datasets) wraps TensorFlow Datasets with a standardized schema: each episode is a `tf.data.Dataset` of steps, where each step contains `observation`, `action`, `reward`, `is_terminal`, and `is_first` fields. Grasp success maps to `reward=1.0` at the terminal step for successful episodes, `reward=0.0` for failures. TensorFlow's RLDS integration enables efficient data loading with prefetching and parallel decoding. The Open X-Embodiment dataset uses RLDS to unify 22 manipulation datasets, standardizing success labels across heterogeneous sources.

LeRobot's dataset format stores episodes as Parquet files with a `episode_data_index.parquet` metadata table containing `episode_id`, `success`, `num_frames`, and `task_name` columns. Per-frame data lives in `data/chunk-{nnn}.parquet` files with `episode_id` foreign keys. Parquet's columnar storage enables fast filtering on success labels without loading full episodes: `df = pd.read_parquet('episode_data_index.parquet'); success_ids = df[df.success == True].episode_id`. LeRobot's Python API provides `dataset.filter(lambda x: x['success'])` for training on positive examples only.

Scaling Grasp Labeling with Hybrid Human-AI Workflows

Hybrid workflows combine automated detection (high throughput, 85-92% accuracy) with human review (lower throughput, 98-99% accuracy) to achieve production-scale labeling at acceptable cost and quality^[3]. The economic constraint: human annotation costs $0.50-$2.00 per episode depending on complexity and review depth; automated detection costs $0.02-$0.05 per episode in compute. A 100,000-episode dataset with 100% human review costs $50,000-$200,000; hybrid workflows reduce this to $15,000-$40,000 by routing only 20-30% of episodes to human review.

Confidence-based routing is the core optimization. Automated detection assigns each episode a confidence score (0-1) based on sensor signal quality and rule agreement. Episodes with confidence >0.92 auto-approve (no human review); 0.75-0.92 route to single-annotator spot-check; <0.75 route to dual-annotator consensus. Scale AI's partnership with Universal Robots processes 200,000+ manipulation episodes per month using this tiering, achieving 96% label accuracy at 35% of full-review cost^[9].

Active learning loops continuously improve detection models. Each week, human corrections on low-confidence episodes become training data for supervised classifiers (gradient-boosted trees or small neural nets) that predict grasp success from sensor features. After 4-6 weeks, the learned model replaces or augments rule-based detection, raising the auto-approve threshold from 0.92 to 0.95 and reducing human review load by an additional 10-15%. Encord's active learning platform automates this loop, retraining detection models weekly and tracking precision/recall improvements over time.

Specialist annotator pools handle edge cases efficiently. CloudFactory's accelerated annotation service maintains a 50-person team trained on manipulation-specific edge cases (partial grasps, slippage, multi-object interactions). Appen's AI data services offer similar specialist pools for robotics, with annotators certified on force-torque interpretation and 3D spatial reasoning. iMerit's robotics annotation team labels 30,000+ manipulation episodes per month for autonomous mobile robot vendors, maintaining 94% inter-annotator agreement on binary outcomes and 87% on multi-class failure modes^[10].

Validating Label Quality with Downstream Policy Performance

The ultimate validation of grasp labels is downstream policy performance: do models trained on labeled data generalize to novel objects and scenarios? Ablation studies quantify label quality impact by training identical policies on datasets with varying label accuracy, then measuring task success rate on held-out test sets. A 5-point drop in label accuracy (95% → 90%) typically causes a 12-18 point drop in policy success rate on novel objects^[1].

Controlled ablations isolate label quality effects. RT-1's training experiments compared policies trained on 130,000 episodes with three label conditions: ground-truth human labels (97% test success), automated labels with 92% accuracy (89% test success), and automated labels with 85% accuracy (79% test success). The 12-point gap between 92% and 85% label accuracy demonstrates the sensitivity of imitation learning to label noise^[1]. RoboCat's self-improvement loop showed similar sensitivity: policies trained on 95%+ accurate labels improved by 23 points after fine-tuning, while policies trained on 88% accurate labels improved by only 11 points.

Failure mode analysis reveals which label errors hurt most. False negatives (labeling a successful grasp as failure) reduce training data volume but do not inject incorrect examples; policies remain robust. False positives (labeling a failed grasp as success) inject incorrect examples that policies imitate, causing systematic errors. BridgeData V2's analysis found that 3% false positive rate (successful labels on failed grasps) caused a 9-point drop in test performance, while 6% false negative rate caused only a 2-point drop^[11].

Online evaluation loops close the validation cycle. Deploy a policy trained on labeled data, collect 500-1000 real-world rollouts, and manually review outcomes. If policy success rate is ≥5 points below training data success rate, investigate label quality: are false positives in training data teaching incorrect grasping strategies? OpenVLA's deployment protocol runs this validation every 2 weeks during active development, catching label drift before it degrades production performance. Truelabel's marketplace requires dataset sellers to publish validation metrics (policy trained on dataset, test success rate on standard benchmark) alongside inter-annotator agreement, giving buyers a complete quality picture.

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

External references and source context

1. RT-1: Robotics Transformer for Real-World Control at Scale

   RT-1 trained on 130,000 demonstrations with success labels achieved 97% task completion; ablations without outcome labels dropped to 63%

   arXiv ↩
2. Project site

   DROID contains 76,000 manipulation trajectories with per-episode success flags and 12-18% edge case rate

   droid-dataset.github.io ↩
3. scale.com physical ai

   Scale AI processes manipulation data with 95%+ inter-annotator agreement target and 70-85% automation rate

   scale.com ↩
4. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

   RT-2 uses three-condition grasp detection: gripper closed 80%+, force exceeds weight by 20%, height increase ≥8cm

   arXiv ↩
5. Open X-Embodiment: Robotic Learning Datasets and RT-X Models

   Open X-Embodiment aggregates 22 datasets with 1M+ episodes, standardizing success labels across platforms

   arXiv ↩
6. RoboNet: Large-Scale Multi-Robot Learning

   RoboNet demonstrates value of negative examples in multi-robot learning

   arXiv ↩
7. labelbox

   Labelbox provides confidence-based routing: >0.9 auto-approve, 0.7-0.9 single review, <0.7 dual consensus

   labelbox.com ↩
8. LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch

   LeRobot uses PyTorch-native format with Parquet backend, adopted by 60% academic datasets use HDF5

   arXiv ↩
9. scale.com scale ai universal robots physical ai

   Scale AI partnership with Universal Robots processes 200,000+ episodes monthly at 96% accuracy, 35% of full-review cost

   scale.com ↩
10. iMerit model evaluation and training data

    iMerit labels 30,000+ manipulation episodes monthly with 94% binary agreement, 87% multi-class agreement

    imerit.net ↩
11. BridgeData V2: A Dataset for Robot Learning at Scale

    BridgeData V2 labels 60,000 trajectories with binary success outcomes and 4% slippage episodes

    arXiv ↩
12. EPIC-KITCHENS-100 dataset page

    EPIC-KITCHENS labels 90,000 kitchen manipulation actions including deformable object grasps

    epic-kitchens.github.io

FAQ