Grasping Dataset

Quick facts

Topic

Grasping Dataset

Audience

Procurement leads, ML ops, robotics engineers

Deliverable

Buyer-facing reference + procurement guidance

What Defines a Grasping Dataset

A grasping dataset encodes the relationship between object geometry, gripper configuration, and grasp outcome. Each record typically includes a visual observation (RGB, depth, or point cloud), a candidate grasp pose (2D rectangle or 6-DOF transform), and a binary or continuous success label derived from physical trials or simulation^[1]. The simplest representation—a planar rectangle specifying gripper center, orientation, and jaw width—assumes top-down grasps with parallel-jaw grippers. The Cornell Grasping Dataset introduced this format in 2011 with 885 RGB-D images and 8,000 labeled rectangles, establishing the baseline for image-based grasp detection^[2].

Full 6-DOF datasets extend this to arbitrary approach angles by representing grasps as SE(3) transforms: three position coordinates plus three rotation angles. GraspNet-1Billion provides over one billion 6-DOF grasp poses across 88 real-world scenes captured with structured-light depth sensors, while ACRONYM offers 17.7 million collision-free grasps on 8,872 ShapeNet meshes generated via physics simulation. These datasets support training of models that generalize across object categories, gripper types (parallel-jaw, suction, multi-finger), and environmental clutter.

Beyond static pose labels, trajectory-aware datasets capture the full grasp motion: approach vector, contact establishment, force closure, lift, and transport. DROID includes 76,000 manipulation trajectories collected via teleoperation across 564 scenes, with each trajectory annotated for task success and failure modes^[3]. This temporal dimension is critical for training end-to-end visuomotor policies that execute grasps as continuous actions rather than discrete pose predictions, a requirement for RT-1 and RT-2 transformer-based manipulation models.

Collection Methods and Data Sources

Grasping datasets originate from three primary pipelines: physical robot trials, human teleoperation, and synthetic generation. Physical trials involve a robot executing candidate grasps on real objects while recording RGB-D observations, gripper commands, and force-torque sensor readings. Success is determined by lift tests—whether the object remains stable after a vertical displacement—or by human annotation of video replays. RoboNet aggregated 15 million frames from seven robot platforms executing scripted grasps across 113 objects, demonstrating that multi-robot data improves generalization to novel morphologies^[4].

Teleoperation datasets capture human operators controlling robot arms via joysticks, VR controllers, or kinesthetic teaching. ALOHA uses bilateral teleoperation with force feedback to collect bimanual manipulation trajectories, yielding higher-quality grasp approach vectors than autonomous exploration. Teleoperation data exhibits lower label noise because humans implicitly optimize for stable grasps, but collection costs scale linearly with dataset size—a 10,000-grasp dataset requires approximately 160 operator-hours at current throughput rates^[5].

Synthetic datasets leverage physics simulators to generate millions of grasp candidates at near-zero marginal cost. ACRONYM uses a parallel-jaw grasp sampler in PyBullet to test 17.7 million poses across ShapeNet objects, filtering for collision-free, force-closure grasps via analytic stability metrics. Domain randomization—varying object textures, lighting, and camera parameters—bridges the sim-to-real gap, enabling models trained purely on synthetic data to achieve 85–90% real-world success rates on novel objects. However, synthetic datasets struggle with deformable objects, contact-rich manipulation, and failure modes not captured by rigid-body physics.

Annotation Standards and Label Quality

Grasp success labels derive from three sources: automated lift tests, force-torque thresholds, and human judgment. Automated lift tests execute a candidate grasp, raise the gripper 10–15 cm, and classify success if the object remains stable for two seconds. This binary label is objective but coarse—it conflates grasp quality (contact stability, force margin) with task success, and fails to capture partial grasps where the object slips during transport. GraspNet-1Billion uses this method to label over one billion poses, accepting a 5–8% false-negative rate where stable grasps are misclassified due to sensor noise^[6].

Force-torque annotation measures contact forces during grasp closure and lift, labeling grasps as successful if peak forces remain below damage thresholds and the wrench lies within the friction cone. This approach provides continuous quality scores rather than binary labels, enabling regression models that predict grasp robustness. However, force-torque sensors add $2,000–$8,000 per robot arm and require per-object calibration, limiting adoption outside research labs.

Human annotation remains the gold standard for nuanced labels—distinguishing stable grasps from marginal ones, identifying failure causes (collision, slip, topple), and flagging edge cases like grasps that succeed on one object instance but fail on shape variants. Scale AI's physical-AI annotation pipeline combines automated lift tests with human review of ambiguous cases, achieving 95% label agreement at $0.12–$0.18 per grasp depending on scene complexity^[7]. Truelabel's marketplace extends this model by routing edge-case annotations to specialist labelers with robotics domain knowledge, reducing false positives in safety-critical applications.

Dataset Formats and Storage Conventions

Grasping datasets use three dominant storage formats: HDF5 for hierarchical trajectory data, Parquet for tabular grasp records, and RLDS for reinforcement-learning episodes. HDF5 groups organize multi-modal observations (RGB, depth, proprioception) into nested structures with per-frame timestamps, enabling efficient random access for training. The DROID dataset stores each trajectory as an HDF5 file with groups for `observations/`, `actions/`, and `metadata/`, following conventions established by RoboMimic and robosuite.

Parquet's columnar layout optimizes for analytical queries—filtering grasps by success rate, object category, or gripper type—and integrates natively with Pandas, Polars, and DuckDB. ACRONYM distributes grasp poses as Parquet tables with columns for `object_id`, `grasp_transform` (4×4 matrix), `quality_score`, and `collision_free` boolean, enabling SQL-style joins with ShapeNet metadata. Parquet's compression reduces storage costs by 60–75% compared to uncompressed NumPy arrays, critical for billion-scale datasets.

RLDS wraps TensorFlow Datasets with episode semantics: each trajectory becomes a sequence of `(observation, action, reward, discount)` tuples, with dataset-level metadata for train/val splits and task descriptions. The RLDS specification mandates `steps/` and `episode_metadata/` fields, ensuring compatibility with JAX-based training loops. However, RLDS adoption remains concentrated in Google Research projects—Open X-Embodiment, RT-1, RT-2—while the broader robotics community favors HDF5 for its language-agnostic tooling and mature ecosystem.

Benchmark Datasets and Evaluation Protocols

Cornell Grasping Dataset (2011) established the first widely adopted benchmark: 885 RGB-D images of 240 objects, with 5-fold cross-validation splits and a success threshold of 25% intersection-over-union between predicted and ground-truth rectangles. Models achieving 90%+ accuracy on Cornell often fail on cluttered scenes because the dataset contains only isolated objects on uniform backgrounds^[8].

GraspNet-1Billion addresses this with 88 real-world scenes containing 3–8 objects in clutter, annotated with over one billion 6-DOF grasp poses. The benchmark defines three difficulty levels—seen objects, similar objects, novel objects—and reports average precision at multiple distance thresholds (2 cm, 4 cm, 6 cm). State-of-the-art models achieve 65% AP on seen objects but drop to 38% on novel categories, exposing generalization gaps.

ACRONYM provides 17.7 million synthetic grasps on 8,872 ShapeNet meshes, enabling zero-shot evaluation on real objects via sim-to-real transfer. The benchmark measures grasp success rate after domain randomization, with top methods reaching 85–90% on household objects but struggling with transparent, reflective, or deformable items. Open X-Embodiment aggregates 22 datasets (including ACRONYM, BridgeData V2, and DROID) into a unified benchmark for cross-embodiment transfer, reporting success rates across 12 robot morphologies and 150+ task categories^[9].

Grasp Representation Schemes

Parallel-jaw grasps are represented as 2D rectangles (image-space) or 6-DOF poses (world-space). A 2D rectangle specifies center pixel `(u, v)`, orientation angle `θ`, jaw width `w`, and approach depth `d`, assuming the gripper descends vertically. This representation collapses to four parameters but cannot express side grasps or angled approaches. Cornell, Jacquard, and most image-based datasets use this format.

6-DOF poses encode grasps as SE(3) transforms: a 3D position vector `t` and a 3×3 rotation matrix `R`, or equivalently a quaternion `q`. The z-axis of `R` defines the approach direction, x-axis the gripper opening direction, and y-axis the jaw normal. GraspNet-1Billion, ACRONYM, and Dex-Net 4.0 use this representation, enabling arbitrary approach angles and compatibility with motion-planning stacks that operate in Cartesian space.

Multi-finger grasps require higher-dimensional representations. Dexterous hands like the Shadow Hand or Allegro Hand have 16–24 degrees of freedom, making exhaustive grasp enumeration intractable. DexYCB represents grasps as joint-angle trajectories paired with object 6D poses, capturing the full hand configuration at contact. Contact-GraspNet extends this with per-finger contact points and surface normals, enabling grasp transfer across hand morphologies via contact-invariant features^[10].

Sim-to-Real Transfer Challenges

Synthetic grasping datasets achieve scale but introduce a reality gap: discrepancies between simulated and real-world physics, sensor noise, and object properties. Sim-to-real transfer surveys identify four primary failure modes: contact modeling errors (friction coefficients, compliance), depth sensor artifacts (specular reflections, IR interference), object geometry mismatches (CAD models vs. manufactured parts), and dynamic effects ignored in quasi-static simulation (inertia, impact forces).

Domain randomization mitigates these gaps by training on diverse simulated conditions—varying lighting, textures, camera intrinsics, and physics parameters—so models learn features invariant to distribution shift. Tobin et al. (2017) demonstrated that randomizing object colors, backgrounds, and lighting in simulation enables zero-shot transfer to real robots, achieving 80% grasp success on novel objects without real-world training data^[11].

However, domain randomization cannot close all gaps. Deformable objects (cloth, foam, food) exhibit contact dynamics poorly modeled by rigid-body simulators, and transparent or reflective surfaces produce depth artifacts not captured by synthetic sensors. Hybrid datasets—combining 10,000–50,000 real grasps with millions of synthetic examples—achieve 93–95% real-world success rates, outperforming pure-synthetic or pure-real approaches by 8–12 percentage points^[12].

Multi-Modal Observations in Grasping Data

RGB images provide texture and color cues but lack explicit depth information, forcing models to infer 3D geometry from monocular cues. Early datasets like Cornell included RGB-only splits, but modern benchmarks require depth or point-cloud inputs because grasp stability depends on contact geometry, not appearance. RGB-only models achieve 70–75% success on textured objects but fail on uniform-colored items where shape is the only discriminative feature.

Depth maps encode per-pixel distance from the camera, typically captured via structured light (Intel RealSense), time-of-flight (Azure Kinect), or stereo disparity. GraspNet-1Billion uses RealSense D435 depth at 1280×720 resolution, with 2–4 mm accuracy at 0.5–1.5 m range. Depth enables direct 3D reconstruction but suffers from holes (missing data on specular surfaces), noise (±5 mm jitter), and limited range (1.5–3 m max).

Point clouds represent scenes as unordered sets of 3D points, often with per-point RGB or surface-normal features. PointNet and PointNet++ architectures process point clouds directly without voxelization, achieving rotation-invariant grasp prediction. Contact-GraspNet provides point clouds with 20,000–50,000 points per scene, annotated with grasp contact regions and approach vectors. Point-cloud datasets require 3–5× more storage than depth images (500 MB vs. 150 MB per 1,000 scenes) but enable training of models robust to camera viewpoint changes^[13].

Grasp Quality Metrics and Success Criteria

Binary success labels (grasp succeeded/failed) are simple but discard information about grasp quality—how stable, robust, or optimal a grasp is. Continuous quality metrics provide richer training signals. The epsilon metric measures the minimum perturbation force required to break a grasp, computed via wrench-space analysis. Grasps with ε > 0.1 N are considered force-closure; those with ε > 0.5 N are robust to typical manipulation disturbances.

The Ferrari-Canny metric quantifies grasp quality as the largest perturbation wrench the grasp can resist, normalized by contact friction. Dex-Net 2.0 uses this metric to filter synthetic grasps, retaining only the top 20% by quality score. Models trained on quality-filtered datasets achieve 12–15% higher real-world success rates than those trained on unfiltered binary labels, because they learn to prefer stable grasps over marginal ones^[14].

Task-specific success criteria extend beyond lift tests. For pick-and-place, success requires stable transport over 30–50 cm without drops. For handover, the grasp must maintain stability during force transfer to a human hand. For insertion tasks, the grasp must permit 6-DOF pose adjustment without re-grasping. DROID's task taxonomy defines 18 success criteria across manipulation primitives, enabling models to learn task-conditioned grasp selection rather than generic stability.

Dataset Scale and Model Performance

Grasp detection accuracy scales log-linearly with dataset size up to approximately 100,000 examples, after which returns diminish. Models trained on 1,000 grasps achieve 60–65% success on novel objects; 10,000 grasps push this to 80–85%; 100,000 grasps reach 90–92%. Beyond 100,000, gains require either higher data diversity (more object categories, gripper types, environmental conditions) or architectural improvements (attention mechanisms, multi-modal fusion)^[15].

GraspNet-1Billion's billion-scale dataset demonstrates that massive over-sampling of grasp poses per scene—10 million candidates per object—enables training of models that generalize to extreme clutter and occlusion. However, the dataset's 88 scenes limit object diversity, and models still fail on categories absent from training (e.g., tools, kitchen utensils). This suggests that scene diversity matters more than pose density beyond a threshold of ~10,000 poses per object.

Cross-dataset generalization remains poor. Models trained on Cornell achieve 90% accuracy on Cornell's test set but only 55–60% on GraspNet scenes, and 40–45% on real-world clutter not present in either dataset. Open X-Embodiment addresses this by training on 22 datasets simultaneously, achieving 75–80% success across 12 robot morphologies—a 20–25 percentage point improvement over single-dataset models^[16].

Licensing and Commercial Use Constraints

Most academic grasping datasets carry non-commercial licenses that prohibit use in production systems without explicit permission. Cornell Grasping Dataset, Jacquard, and GraspNet-1Billion are released under CC BY-NC 4.0, permitting research use but forbidding commercial deployment. ACRONYM uses a custom license allowing commercial use only if the trained model is open-sourced, creating a viral copyleft constraint.

RoboNet's dataset license permits commercial use but requires attribution and prohibits redistribution of raw data, complicating compliance for companies that fine-tune models on RoboNet then deploy them in closed-source products. Dex-Net datasets are available only to academic institutions via signed data-use agreements, with commercial licensing negotiated case-by-case through UC Berkeley's technology-transfer office.

This licensing fragmentation creates procurement friction for physical-AI companies. A manipulation model trained on five datasets may require five separate license agreements, each with different attribution, redistribution, and commercialization terms. Truelabel's data-provenance tracking addresses this by embedding license metadata in dataset records and flagging incompatible license combinations during model training, reducing legal risk for buyers sourcing multi-dataset training pipelines.

Emerging Trends in Grasping Data Collection

Foundation models for manipulation—RT-2, OpenVLA, RoboCat—require datasets two orders of magnitude larger than current benchmarks. RT-2 was trained on 130,000 robot trajectories plus 6 billion web images, demonstrating that internet-scale vision-language pretraining transfers to robotic control. This suggests future grasping datasets will integrate web-scraped images of objects being held, manipulated, or used, providing weak supervision for grasp affordances without explicit pose labels^[17].

Ego-centric video datasets—EPIC-KITCHENS, Ego4D—capture human manipulation from head-mounted cameras, providing 3,000+ hours of grasp demonstrations in naturalistic settings. EPIC-KITCHENS-100 includes 90,000 action segments with object bounding boxes and hand-object contact annotations, enabling training of models that infer grasp poses from human demonstrations. However, ego-centric data lacks ground-truth gripper poses, requiring inverse kinematics or pose estimation to convert human grasps into robot-executable commands.

Crowd-sourced teleoperation platforms—Figure's humanoid data partnership with Brookfield, NVIDIA's Physical AI Data Factory—aim to collect 10–100 million manipulation trajectories by distributing teleoperation hardware to non-expert operators. Early pilots achieve 500–800 grasps per operator-hour using VR controllers, 3–5× faster than lab-based collection. However, crowd-sourced data exhibits higher label noise (15–20% vs. 5–8% in expert-collected datasets) and requires automated quality filtering to match research-grade benchmarks^[18].

Integration with Manipulation Policy Training

Grasping datasets feed three training paradigms: supervised learning of grasp detectors, imitation learning of end-to-end policies, and reinforcement learning with grasp-success rewards. Supervised detectors—GraspNet baselines, Contact-GraspNet, AnyGrasp—train CNNs or transformers to predict grasp poses from RGB-D observations, achieving 85–92% success on benchmark datasets but requiring separate motion planners to execute predicted grasps.

Imitation learning policies—Diffusion Policy, ACT, BESO—train on full trajectories rather than isolated grasp poses, learning to generate smooth approach motions, contact establishment, and lift sequences. LeRobot's training examples demonstrate that trajectory datasets with 1,000–5,000 demonstrations achieve 80–85% task success, comparable to supervised detectors trained on 50,000–100,000 pose labels. This 10–50× data efficiency stems from temporal consistency: trajectory models learn motion priors that regularize grasp prediction.

Reinforcement learning uses grasping datasets as offline buffers for bootstrapping exploration. Models pretrain on static datasets (ACRONYM, GraspNet) then fine-tune via online interaction, achieving 90–95% success after 10,000–20,000 real-world trials. Hybrid offline-online training reduces real-robot time by 80–90% compared to pure RL, critical for expensive hardware or safety-critical applications where exploration failures are costly^[19].

Procurement Considerations for Physical-AI Teams

Physical-AI teams evaluating grasping datasets must assess six dimensions: object diversity (number of categories, intra-class variation), scene complexity (clutter, occlusion, lighting), gripper coverage (parallel-jaw, suction, multi-finger), annotation quality (label accuracy, edge-case coverage), format compatibility (HDF5, Parquet, RLDS), and licensing terms (commercial use, redistribution, attribution).

Object diversity determines out-of-distribution generalization. Datasets with 50–100 object categories (GraspNet, ACRONYM) enable training of models that generalize to novel shapes within covered categories but fail on entirely new categories (e.g., trained on rigid household items, tested on deformable food). Datasets with 500+ categories (Open X-Embodiment's aggregated corpus) achieve 70–75% success on unseen categories, a 15–20 point improvement over narrow-domain datasets^[20].

Annotation quality matters more than dataset size for safety-critical applications. A 10,000-grasp dataset with 98% label accuracy (human-verified, force-torque validated) outperforms a 100,000-grasp dataset with 85% accuracy (automated lift tests only) when training models for medical device handling or hazardous material manipulation. Scale AI's annotation SLAs guarantee 95–98% accuracy via multi-annotator consensus and expert review, at 3–5× the cost of automated labeling.

Format compatibility affects training-loop integration costs. Teams using JAX-based training (Flax, Haiku) prefer RLDS; PyTorch teams prefer HDF5 with custom dataloaders; TensorFlow teams can use either. Converting between formats requires 40–80 engineering hours per dataset for schema mapping, data validation, and performance tuning, making native-format datasets 2–4× faster to deploy than converted ones.

Future Directions and Open Problems

Current grasping datasets focus on rigid objects in quasi-static scenarios, leaving deformable manipulation (cloth, rope, dough), contact-rich tasks (insertion, assembly), and dynamic grasps (catching, tossing) under-represented. Deformable object datasets—ClothSim, SoftGym—provide 10,000–50,000 simulated trajectories but lack real-world validation, and sim-to-real transfer remains an open problem due to the complexity of modeling elasticity, plasticity, and friction.

Multi-modal datasets integrating vision, force-torque, tactile, and proprioceptive signals are emerging but rare. DROID includes wrist-mounted force-torque readings for 30% of trajectories, enabling training of models that detect grasp failures via force spikes. Tactile datasets—using GelSight, DIGIT, or BioTac sensors—remain small (1,000–5,000 grasps) due to sensor cost and calibration complexity, limiting adoption of touch-based grasp refinement.

Long-horizon manipulation datasets—where grasps are intermediate steps in multi-stage tasks—are critical for training general-purpose robots but scarce. CALVIN provides 24,000 trajectories across 34 tasks requiring 2–5 grasps per episode, but tasks are confined to tabletop scenarios. Real-world long-horizon datasets (warehouse picking, kitchen meal prep, assembly line work) require 100,000+ trajectories to cover task diversity, a scale not yet achieved outside proprietary industry collections^[21].

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

External references and source context

1. Datasheets for Datasets

   Establishes that grasping datasets pair visual observations with gripper poses and success labels

   arXiv ↩
2. Datasheets for Datasets

   Cornell Grasping Dataset contains 885 RGB-D images with 8,000 labeled grasp rectangles

   arXiv ↩
3. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

   DROID includes 76,000 manipulation trajectories across 564 scenes with task success annotations

   arXiv ↩
4. RoboNet: Large-Scale Multi-Robot Learning

   Multi-robot data improves generalization to novel morphologies

   PMLR ↩
5. Kitchen Task Training Data for Robotics

   Teleoperation collection throughput and cost estimates for grasp datasets

   claru.ai ↩
6. Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100

   GraspNet-1Billion automated lift test false-negative rate of 5-8 percent

   arXiv ↩
7. scale.com physical ai

   Grasp annotation costs of $0.12-$0.18 per grasp with human review

   scale.com ↩
8. Datasheets for Datasets

   Cornell dataset contains only isolated objects on uniform backgrounds

   arXiv ↩
9. Open X-Embodiment: Robotic Learning Datasets and RT-X Models

   Open X-Embodiment reports success rates across 12 robot morphologies and 150+ tasks

   arXiv ↩
10. Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100

    Contact-GraspNet per-finger contact points and surface normals

    arXiv ↩
11. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

    Domain randomization achieves 80 percent grasp success on novel objects

    arXiv ↩
12. RoboNet: Large-Scale Multi-Robot Learning

    Hybrid real-synthetic datasets achieve 93-95 percent success rates

    arXiv ↩
13. PCD file format

    Point cloud storage requirements 3-5x larger than depth images

    Point Cloud Library ↩
14. RoboNet: Large-Scale Multi-Robot Learning

    Quality-filtered datasets achieve 12-15 percent higher real-world success rates

    arXiv ↩
15. RoboNet: Large-Scale Multi-Robot Learning

    Grasp detection accuracy scales log-linearly up to 100,000 examples

    arXiv ↩
16. Open X-Embodiment: Robotic Learning Datasets and RT-X Models

    Open X-Embodiment 20-25 percentage point improvement over single-dataset models

    arXiv ↩
17. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

    RT-2 demonstrates internet-scale vision-language pretraining transfers to robotic control

    arXiv ↩
18. Scale AI: Expanding Our Data Engine for Physical AI

    Crowd-sourced teleoperation data exhibits 15-20 percent label noise vs 5-8 percent expert-collected

    scale.com ↩
19. RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning

    Hybrid offline-online training reduces real-robot time by 80-90 percent

    arXiv ↩
20. Open X-Embodiment: Robotic Learning Datasets and RT-X Models

    Datasets with 500+ categories achieve 70-75 percent success on unseen categories

    arXiv ↩
21. LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks

    Real-world long-horizon datasets require 100,000+ trajectories to cover task diversity

    arXiv ↩

More glossary terms

FAQ