How to Build an Object Tracking Dataset for Physical AI

Quick facts

Difficulty

Intermediate

Audience

Physical AI data engineers

Last reviewed

2025-06-15

Why Object Tracking Datasets Power Next-Generation Physical AI

Object tracking datasets teach robots to maintain spatial awareness of dynamic entities—humans, tools, obstacles—across time. Unlike single-frame detection, tracking preserves identity through occlusions and camera motion, enabling manipulation policies to reason about object permanence and predict future states. Google's RT-1 Robotics Transformer trained on 130,000 episodes uses temporal object embeddings to generalize pick-and-place across 700+ tasks^[1]. Open X-Embodiment aggregates 22 datasets with 527,000 trajectories, many requiring persistent track IDs for multi-step assembly and handoff tasks^[2].

Physical AI applications demand richer annotations than autonomous vehicle tracking. Kitchen robots must track ingredient states (chopped vs. whole), tool orientations (knife blade angle), and hand-object interactions simultaneously. EPIC-KITCHENS-100 provides 90,000 action segments with 20 million bounding boxes across 100 hours of egocentric video, but lacks the 6-DOF pose and force data needed for contact-rich manipulation^[3]. DROID addresses this with 76,000 teleoperation trajectories capturing gripper state, wrist-mounted RGB-D, and third-person tracking of 398 object instances across 564 scenes^[4]. Warehouse datasets like those on truelabel's marketplace add LiDAR point clouds and multi-camera synchronization for pallet tracking under forklift occlusion.

Tracking annotation costs scale non-linearly with object count and occlusion frequency. A 10-minute warehouse sequence with 15 simultaneous objects and 40% occlusion rate requires 8-12 annotator hours using model-assisted tools versus 25-30 hours for manual frame-by-frame labeling. Scale AI's Physical AI platform reports 60% annotation time reduction through foundation model pre-labeling, but human review remains mandatory for ID consistency across occlusion boundaries^[5].

Define Tracking Taxonomy and Annotation Specification

Tracking taxonomies must balance granularity with annotator reliability. Kitchen robotics datasets typically define 25-40 object classes (utensils, containers, ingredients, appliances) plus 2-4 hand classes (left/right, open/closed). Warehouse datasets need 15-25 classes (box sizes, pallet types, vehicle categories, human roles). CALVIN's manipulation benchmark uses 34 object classes with sub-types for drawer states (open/closed) and object poses (upright/sideways)^[6].

Annotation formats split between bounding boxes and segmentation masks. Axis-aligned boxes (x, y, width, height) in MOT Challenge format suffice for rigid objects with predictable orientations. Rotated boxes (x, y, width, height, angle) reduce background pixels for elongated tools like spatulas or wrenches. Instance segmentation masks in COCO RLE encoding capture irregular shapes—crumpled cloth, deformable food—but increase annotation time 3-5× and storage requirements 8-12× versus boxes. BridgeData V2 uses masks for target objects and boxes for distractors, balancing precision with dataset size (60,096 trajectories at 2.4 TB)^[7].

Track metadata schemas must encode visibility states and lifecycle events. Each track carries a persistent integer ID, category label, per-frame visibility flag (visible/partially_occluded/fully_occluded/out_of_frame), and bounding geometry. Occlusion flags trigger different loss weights during training—RT-2 masks gradients for fully occluded frames to prevent hallucination^[8]. Track lifecycle rules define when to spawn new IDs (object enters frame, emerges from full occlusion >2 seconds) versus maintain existing IDs (brief occlusion <1 second, partial visibility). Inconsistent ID assignment across annotators is the primary cause of tracking metric collapse—HOTA scores drop 15-25 points when re-identification rules lack frame-level precision.

Prepare Video Data and Generate Pre-Annotations

Video preprocessing standardizes frame rates, resolution, and color spaces before annotation. Physical AI datasets typically use 10-30 FPS—higher rates capture fast motions (throwing, catching) but increase annotation cost linearly. DROID uses 15 FPS for teleoperation, balancing temporal resolution with the 1.1 TB dataset size across 350 hours of recording^[4]. Extract frames with FFmpeg using consistent encoding (H.264 at CRF 18-23) to avoid compression artifacts that confuse pre-annotation models.

Organize data into annotation-platform-compatible structures. CVAT ingests video files directly or frame sequences in directories; Labelbox requires cloud storage URLs with signed access. Partition sequences into 30-90 second clips—shorter clips reduce annotator fatigue and enable parallel assignment, but increase overhead from track ID reconciliation across clip boundaries. RLDS format stores episodes as TFRecord shards with embedded video, simplifying downstream training pipelines^[9].

Pre-annotation with detection and tracking models reduces human labeling time 50-70%. Run a foundation model like DETR or Grounding DINO for per-frame detection, then apply a tracking algorithm like ByteTrack or DeepSORT to assign preliminary IDs. Export predictions in platform-native formats—CVAT XML, Label Studio JSON, or MOT Challenge CSV. Pre-annotation quality directly impacts review efficiency: 85% precision at 70% recall is optimal (annotators correct false positives faster than they find false negatives). Calibrate confidence thresholds on 500-1000 representative frames before bulk processing. Encord Active provides model-assisted annotation with active learning to prioritize ambiguous frames^[10].

Execute Multi-Pass Annotation with Quality Controls

Multi-pass annotation separates detection, tracking, and quality assurance into specialized roles. Pass 1 (detection): annotators draw bounding boxes or masks on keyframes (every 10-30 frames depending on motion speed), assigning category labels but not track IDs. Pass 2 (tracking): a separate annotator or the same annotator in a second session links detections across frames, assigning persistent IDs and marking occlusion states. Pass 3 (QA): a senior annotator reviews 10-20% of tracks, focusing on occlusion boundaries, ID switches, and category errors. This pipeline reduces per-frame error rates from 12-18% (single-pass) to 3-6% (three-pass) in Sama's computer vision workflows^[11].

Annotation platforms must support temporal navigation and ID propagation. CVAT's track mode lets annotators create a box on frame N, then automatically interpolates to frame N+K, with manual adjustment at occlusion points^[12]. Segments.ai adds multi-sensor views—synchronized RGB, depth, LiDAR—for warehouse datasets where 3D context resolves ambiguous 2D overlaps^[13]. Keyboard shortcuts for ID assignment (number keys 1-9 for frequent classes, search for rare classes) and occlusion toggling (O key) double annotation throughput versus mouse-only interfaces.

Quality metrics track annotator performance and dataset health. Measure per-annotator ID switch rate (target <2 switches per 1000 frames), box IoU consistency across frames (target >0.85 for rigid objects), and occlusion flag agreement with ground truth (target >90% on validation sets). Dataloop's consensus workflows require 2-3 annotators per ambiguous sequence, auto-resolving agreements and escalating disagreements to experts^[14]. Track annotation velocity (frames per hour) and flag outliers—annotators 2× faster than median often sacrifice ID consistency for speed.

Handle Occlusion, Re-Identification, and Edge Cases

Occlusion handling separates amateur from production-grade tracking datasets. Partial occlusion (object 30-70% visible) requires continued tracking with reduced box confidence or mask coverage. Full occlusion (object 0% visible) triggers a visibility flag but maintains the track ID if re-appearance is expected within 2-3 seconds. Prolonged occlusion (>3 seconds) terminates the track; re-appearance spawns a new ID unless visual features (color, shape, last-known trajectory) enable confident re-identification.

Re-identification rules must be explicit and testable. For rigid objects with distinctive appearance (red mug, yellow drill), maintain ID across occlusions up to 5 seconds. For generic objects (identical cardboard boxes), spawn new IDs after any full occlusion—tracking algorithms cannot reliably re-identify without unique features. Ego4D's 3,670 hours of egocentric video includes re-identification annotations for 5,000+ object instances, enabling training of appearance-based re-ID models^[15]. Hand tracking in manipulation datasets requires special rules: hands frequently occlude objects and each other, so track IDs persist across self-occlusion but reset when hands leave frame for >1 second.

Edge cases demand written guidelines with visual examples. Reflections in mirrors or shiny surfaces: annotate the real object only, not the reflection. Object splits (cutting an apple): terminate the original track, spawn two new IDs for the halves. Object merges (stacking boxes): maintain individual IDs if boxes remain distinguishable, else merge into a single aggregate track. Deformable objects (cloth, dough): use masks instead of boxes, accept 10-15% IoU variance across frames as normal. RoboNet's 15 million frames across 7 robot platforms include deformable object annotations with per-frame mask quality scores^[16].

Validate Annotations with Automated Metrics

Automated validation catches systematic errors before training. Temporal consistency checks flag tracks with impossible motion (object teleports >50 pixels between frames at 30 FPS without occlusion), sudden category changes (mug becomes plate), or duplicate IDs in the same frame. Geometric checks flag boxes with aspect ratios outside expected ranges (width/height >5:1 for most objects suggests annotation error), boxes extending beyond frame boundaries, or masks with disconnected components.

Tracking-specific metrics quantify dataset quality. HOTA (Higher Order Tracking Accuracy) combines detection accuracy, association accuracy, and localization accuracy into a single score; production datasets target HOTA >75 on held-out validation sets. ID switches per track (target <0.1) measure re-identification consistency. Track fragmentation (number of track segments per ground-truth object, target <1.2) reveals premature track termination. MOT Challenge benchmarks define standard metrics and provide evaluation scripts^[17].

Cross-annotator agreement validates ambiguous sequences. Select 200-500 frames spanning diverse scenarios (heavy occlusion, fast motion, crowded scenes) and have 2-3 annotators label independently. Compute pairwise IoU for boxes (target >0.80) and ID overlap for tracks (target >0.85). Low agreement (<0.70) indicates unclear taxonomy or insufficient guidelines. Appen's quality frameworks use consensus labeling for 5-10% of data, then train annotators on disagreement cases^[18]. Re-annotation of low-agreement sequences with refined guidelines typically lifts agreement 10-15 points.

Format for Training Pipelines and Benchmark Submission

Export formats must match downstream training frameworks. RLDS (Reinforcement Learning Datasets) stores episodes as TFRecord files with nested tensors for observations (images, proprio), actions, rewards, and metadata^[19]. Each episode contains a steps array; each step contains an observation dict with image (H×W×3 uint8), track_ids (N int32), track_boxes (N×4 float32), and track_categories (N int32). LeRobot format uses Parquet for tabular data (timestamps, actions, states) and separate video files with frame-to-row indices, optimizing for random access during training^[20].

Benchmark submissions require standardized splits and metadata. MOT Challenge uses 50/50 train/test splits with held-out test annotations. Open X-Embodiment defines task-specific splits (seen objects/unseen objects, seen environments/unseen environments) to measure generalization^[2]. Include a datasheet documenting collection conditions (lighting, camera specs, object set), annotation process (platform, annotator count, QA steps), and known limitations (missing small objects <20 pixels, no nighttime data). Datasheets for Datasets provides a template covering 57 questions across motivation, composition, collection, preprocessing, uses, distribution, and maintenance^[21].

Data provenance tracking ensures reproducibility and compliance. Record source video checksums, annotation platform versions, model versions for pre-annotation, and annotator IDs (anonymized). Truelabel's provenance system links every annotation to collector hardware, timestamp, and geographic region, enabling buyers to filter by collection conditions^[22]. C2PA content credentials embed cryptographic provenance in media files, preventing training on synthetic or manipulated data^[23].

Scale Annotation with Model-Assisted Pipelines

Model-assisted annotation pipelines combine foundation models, active learning, and human review to scale to millions of frames. Foundation models like Grounding DINO or SAM provide zero-shot detection and segmentation; fine-tune on 500-1000 domain-specific examples to lift recall from 60-70% (zero-shot) to 85-92% (fine-tuned). Active learning selects frames where model confidence is low (entropy >0.6) or predictions disagree across ensemble members, routing only these frames to human annotators. Encord Active reports 3-5× annotation efficiency gains versus random sampling^[10].

Tracking-by-detection pipelines separate spatial localization from temporal association. Run a detector (YOLO, Faster R-CNN) on every frame, then apply a tracker (SORT, ByteTrack, StrongSORT) to link detections into tracks. Detector precision matters more than recall—false positives create ghost tracks that annotators must delete, while false negatives are caught during human review. Tracker hyperparameters (IoU threshold, max age, min hits) require tuning on 1000-2000 validation frames; default settings often produce 20-30% excess ID switches.

Human-in-the-loop workflows iterate between model predictions and annotator corrections. Annotators review model tracks, correcting ID switches, adding missed objects, and deleting false positives. Corrections feed back into model training every 5,000-10,000 frames, progressively reducing error rates. Labelbox's model-assisted labeling reduces annotation time 40-60% for object tracking versus manual annotation^[24]. V7's automation suite claims 90% pre-annotation accuracy on custom object classes after 2,000 training examples^[25].

Benchmark Datasets and Performance Targets

Public benchmarks establish quality baselines and evaluation protocols. MOT Challenge provides 11,000 frames across pedestrian tracking scenarios with dense annotations (50+ simultaneous tracks), targeting MOTA >65 and IDF1 >60 for competitive methods. EPIC-KITCHENS-100 offers 90,000 action segments with object bounding boxes, but lacks persistent track IDs across actions—a gap for manipulation policy training^[3].

Robotics-specific benchmarks emphasize manipulation-relevant tracking. DROID's 76,000 trajectories include 398 object instances tracked across 564 scenes with wrist-mounted and third-person cameras, enabling evaluation of multi-view tracking consistency^[4]. BridgeData V2 provides 60,096 trajectories with target object masks and distractor boxes, but no persistent IDs for distractors—limiting multi-object reasoning tasks^[7]. CALVIN includes 34 object classes with state annotations (drawer open/closed, object upright/sideways) across 24,000 language-conditioned tasks^[6].

Performance targets vary by application. Warehouse robotics requires MOTA >80 and ID switch rate <1 per 1000 frames for reliable pallet tracking under forklift occlusion. Kitchen robotics tolerates MOTA 70-75 but demands higher localization precision (IoU >0.85) for grasp planning. Egocentric manipulation datasets like Ego4D accept MOTA 60-65 due to severe hand-object occlusion, but require re-identification accuracy >75% for objects returning to view^[15]. Truelabel's marketplace filters datasets by these metrics, enabling buyers to match quality to task requirements.

Common Annotation Failure Modes and Mitigations

ID switches at occlusion boundaries are the most frequent tracking error. Annotators lose track of object identity when multiple similar objects (identical boxes, same-model robots) pass behind occluders. Mitigation: require annotators to mark uncertainty flags on ambiguous re-identifications, then route flagged tracks to senior reviewers. Use appearance features (color, texture, size) as tie-breakers; when features are insufficient, spawn new IDs rather than guess. MOT Challenge data shows ID switch rates drop 40-60% when annotators mark uncertainty versus making forced choices^[17].

Inconsistent bounding box tightness across frames degrades localization metrics. Annotators vary in how closely they fit boxes to object boundaries—some leave 5-10 pixel margins, others fit pixel-perfect. Mitigation: define box-fitting rules with visual examples (box edges should bisect boundary pixels, include shadows only if attached, exclude motion blur). Automated checks flag boxes with frame-to-frame area changes >20% without corresponding object scale change. CVAT's interpolation smooths box dimensions between keyframes, reducing jitter^[12].

Missing small objects (<20 pixels) biases datasets toward large, salient entities. Annotators overlook distant objects, small tools, or low-contrast items against busy backgrounds. Mitigation: use multi-pass annotation where pass 1 focuses on large objects, pass 2 specifically searches for small objects with zoomed views. Pre-annotation models often miss small objects (recall <40% for <15 pixel boxes), so human review must compensate. RoboNet excludes objects <15 pixels from evaluation to avoid penalizing methods for inherently ambiguous cases^[16].

Ghost tracks on background clutter or shadows waste annotation budget and confuse training. Annotators occasionally track non-objects—floor patterns, wall textures, lighting artifacts—especially when pre-annotation models hallucinate detections. Mitigation: train annotators on negative examples (what NOT to track), implement per-track review where QA checks the first and last frame of every track for object validity. Automated checks flag tracks with <5 frames (likely false positives) or tracks that never move (likely background).

Multi-Sensor Tracking for Physical AI

Multi-sensor tracking fuses RGB, depth, LiDAR, and proprioceptive data for robust object localization. RGB provides appearance features for re-identification; depth resolves occlusion ordering (which object is in front); LiDAR gives metric 3D positions immune to lighting changes. DROID combines wrist RGB-D with third-person RGB, enabling tracking of objects in gripper (wrist view) and workspace (third-person view) simultaneously^[4]. Warehouse datasets add overhead LiDAR for pallet tracking across 50+ meter ranges where camera resolution degrades.

Synchronization across sensors is critical—misaligned timestamps create phantom motion or duplicate objects. Hardware-triggered capture (all sensors triggered by a master clock) achieves <1 ms sync; software timestamps introduce 10-50 ms jitter depending on OS scheduling. MCAP format stores multi-sensor data with nanosecond timestamps and schema definitions, enabling precise temporal alignment during annotation^[26]. ROS bag files provide similar capabilities but lack built-in schema versioning, complicating long-term dataset maintenance^[27].

Annotation platforms must render multi-sensor views simultaneously. Segments.ai displays RGB, depth, and LiDAR point clouds in synchronized panes; annotators draw 3D bounding boxes in point cloud view, which project to 2D boxes in RGB view^[13]. Kognic's platform specializes in automotive and robotics annotation with sensor fusion, supporting cuboid annotation in 3D with automatic 2D projection^[28]. 3D annotation is 2-3× slower than 2D but provides metric dimensions (length, width, height in meters) required for grasp planning and collision avoidance.

Dataset Licensing and Commercialization

Licensing terms determine whether datasets can train commercial models. Creative Commons BY 4.0 permits commercial use with attribution; CC BY-NC 4.0 restricts commercial use, blocking deployment in products^[29]. Many academic datasets use CC BY-NC, requiring companies to negotiate separate licenses or collect proprietary data. EPIC-KITCHENS-100 annotations use a custom non-commercial license prohibiting model commercialization without permission^[30].

Data provenance affects legal risk and model performance. Datasets scraped from YouTube or social media may contain copyrighted content, biased distributions, or privacy violations. GDPR Article 7 requires explicit consent for personal data collection; egocentric datasets showing faces or license plates need consent forms or anonymization^[31]. Truelabel's marketplace provides cryptographic provenance for every data point, linking to collector consent and collection conditions^[22].

Procurement workflows for physical AI datasets differ from web data. Buyers specify task requirements (object classes, scene diversity, occlusion frequency), quality thresholds (MOTA >75, ID switch rate <2%), and format (RLDS, LeRobot, custom). Truelabel's request system matches requirements to collectors, who capture and annotate data on-spec, with payment contingent on passing automated quality checks^[32]. Scale AI's data engine offers similar services with managed annotation teams, targeting 2-4 week delivery for 10,000-50,000 frame datasets^[5].

Tracking Annotation Formats and Interoperability

MOT Challenge format stores tracks as CSV with columns: frame_id, track_id, x, y, width, height, confidence, class_id, visibility. Each row is one detection; tracks are implicit sequences of rows with the same track_id. Simple and human-readable, but lacks support for masks, 3D boxes, or multi-sensor data. MOT Challenge benchmarks use this format exclusively, making it the de facto standard for 2D pedestrian tracking^[17].

COCO format extends JSON with per-frame annotations: images array (file paths, dimensions), annotations array (bbox, segmentation, category_id, track_id), and categories array (id, name). Segmentation uses RLE (run-length encoding) for masks. COCO supports instance segmentation but lacks temporal metadata (occlusion flags, track lifecycle events). EPIC-KITCHENS-100 uses COCO-style JSON with added action_id and verb/noun labels^[3].

RLDS format embeds tracking annotations in episode tensors: observation['image'] (H×W×3), observation['track_ids'] (N), observation['track_boxes'] (N×4), observation['track_masks'] (N×H×W). Tracks are first-class citizens alongside actions and rewards, enabling end-to-end policy training without format conversion^[19]. LeRobot separates video (MP4) from tabular data (Parquet), storing track_ids and track_boxes as columns indexed by frame number^[20]. Parquet's columnar layout accelerates filtering (e.g., load only frames where track_id==5) versus row-based formats.

Interoperability tools convert between formats. RLDS repository includes scripts to import MOT Challenge CSV and COCO JSON into TFRecord episodes^[33]. LeRobot's data loaders parse RLDS, Parquet, and HDF5, unifying access across dataset sources^[34]. Conversion losses occur when target format lacks source features—COCO masks downgrade to bounding boxes in MOT format, RLDS 3D boxes project to 2D in COCO.

Active Learning and Data Efficiency

Active learning selects the most informative frames for annotation, reducing dataset size 3-5× for equivalent model performance. Uncertainty sampling annotates frames where model entropy is highest (prediction confidence <0.6 across classes). Diversity sampling clusters frame embeddings (e.g., CLIP features) and samples from under-represented clusters, ensuring coverage of rare scenarios. Encord Active combines both strategies, prioritizing high-uncertainty frames from diverse clusters^[10].

Error-driven sampling focuses annotation on model failure modes. Run a baseline tracker on unlabeled video, compute per-frame metrics (detection recall, ID switch rate), and annotate the worst-performing 10-20%. This targets annotation budget where it improves performance most. Dataloop's active learning workflows re-train models every 2,000-5,000 annotations, progressively shifting sampling toward remaining hard cases^[14].

Data efficiency metrics quantify annotation ROI. Plot model performance (MOTA, HOTA) versus annotation hours; the curve should be concave (diminishing returns). If performance plateaus after 5,000 frames, additional annotation is wasted—invest in data diversity (new scenes, lighting, objects) instead. BridgeData V2 demonstrates that 13 diverse environments with 4,600 trajectories each outperform 60,000 trajectories from a single environment^[7]. Truelabel's marketplace enables buyers to request specific diversity axes (geographic regions, lighting conditions, object sets) rather than bulk undifferentiated data.

Temporal Consistency and Interpolation

Temporal consistency ensures smooth track evolution without jitter or teleportation. Annotators mark keyframes (every 10-30 frames depending on motion speed), and interpolation fills intermediate frames. Linear interpolation works for constant-velocity motion; cubic splines handle acceleration. CVAT's track mode uses linear interpolation by default, with manual keyframe insertion at direction changes or occlusions^[12].

Interpolation quality depends on keyframe density. For slow-moving objects (stationary tools, parked robots), 30-frame keyframe spacing suffices. Fast-moving objects (thrown items, human hands) require 5-10 frame spacing to avoid interpolation errors >10 pixels. Automated checks flag tracks with inter-keyframe motion >50 pixels, prompting annotators to add intermediate keyframes. Labelbox's interpolation uses optical flow to predict intermediate boxes, reducing keyframe requirements 30-40%^[24].

Occlusion boundaries break interpolation. When an object disappears behind an occluder, interpolation must stop; when it re-appears, a new interpolation segment begins. Annotators mark occlusion start/end frames explicitly, preventing interpolation from drawing boxes on occluded objects. MOT Challenge guidelines require visibility flags (0=fully occluded, 1=partially occluded, 2=fully visible) on every frame to guide interpolation logic^[17]. Tracks with >50% occluded frames often have 2-3× higher ID switch rates, indicating re-identification difficulty.

Evaluation Metrics for Tracking Datasets

MOTA (Multiple Object Tracking Accuracy) combines false positives, false negatives, and ID switches: MOTA = 1 - (FP + FN + IDS) / GT, where GT is ground truth object count. MOTA penalizes all error types equally, making it sensitive to detection quality. Target MOTA >75 for production robotics datasets. MOT Challenge uses MOTA as the primary leaderboard metric^[17].

HOTA (Higher Order Tracking Accuracy) separately measures detection accuracy (DetA), association accuracy (AssA), and localization accuracy (LocA), then combines as HOTA = sqrt(DetA × AssA). HOTA is more interpretable than MOTA—low DetA indicates missed objects, low AssA indicates ID switches, low LocA indicates poor box fit. Target HOTA >70 for research datasets, >80 for production. HOTA metrics are becoming the standard for robotics tracking benchmarks.

IDF1 (ID F1 score) measures the ratio of correctly identified detections to total detections, emphasizing long-term identity preservation. IDF1 = 2 × IDTP / (2 × IDTP + IDFP + IDFN), where IDTP counts detections with correct ID and IoU >0.5. Target IDF1 >60 for crowded scenes (10+ simultaneous objects), >75 for sparse scenes. MOT Challenge reports IDF1 alongside MOTA^[17].

Per-class metrics reveal taxonomy imbalances. Compute MOTA, HOTA, IDF1 separately for each object class; classes with <50 instances or <10 tracks often have unreliable metrics. EPIC-KITCHENS-100 reports per-noun metrics across 300 object classes, showing 40-point MOTA variance between frequent (mug, plate) and rare (whisk, colander) classes^[3]. Stratify evaluation by object size (<32 pixels, 32-96 pixels, >96 pixels) to diagnose small-object tracking failures.

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

External references and source context

1. RT-1: Robotics Transformer for Real-World Control at Scale

   RT-1 trained on 130,000 episodes uses temporal object embeddings to generalize across 700+ tasks

   arXiv ↩
2. Open X-Embodiment: Robotic Learning Datasets and RT-X Models

   Open X-Embodiment aggregates 22 datasets with 527,000 trajectories requiring persistent track IDs

   arXiv ↩
3. Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100

   EPIC-KITCHENS-100 provides 90,000 action segments with 20 million bounding boxes across 100 hours

   arXiv ↩
4. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

   DROID contains 76,000 teleoperation trajectories with 398 object instances across 564 scenes

   arXiv ↩
5. Scale AI: Expanding Our Data Engine for Physical AI

   Scale AI reports 60% annotation time reduction through foundation model pre-labeling

   scale.com ↩
6. CALVIN paper

   CALVIN uses 34 object classes with sub-types for drawer states and object poses

   arXiv ↩
7. BridgeData V2: A Dataset for Robot Learning at Scale

   BridgeData V2 contains 60,096 trajectories at 2.4 TB with masks for targets and boxes for distractors

   arXiv ↩
8. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

   RT-2 masks gradients for fully occluded frames to prevent hallucination

   arXiv ↩
9. RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning

   RLDS format stores episodes as TFRecord shards with embedded video

   arXiv ↩
10. encord.com active

    Encord Active provides model-assisted annotation with active learning

    encord.com ↩
11. sama.com computer vision

    Sama's multi-pass annotation workflows reduce error rates from 12-18% to 3-6%

    sama.com ↩
12. CVAT polygon annotation manual

    CVAT track mode with interpolation and temporal navigation features

    docs.cvat.ai ↩
13. Segments.ai multi-sensor data labeling

    Segments.ai multi-sensor annotation with synchronized RGB, depth, and LiDAR views

    segments.ai ↩
14. dataloop.ai annotation

    Dataloop consensus workflows and quality management features

    dataloop.ai ↩
15. Ego4D: Around the World in 3,000 Hours of Egocentric Video

    Ego4D contains 3,670 hours of egocentric video with re-identification annotations for 5,000+ objects

    arXiv ↩
16. RoboNet: Large-Scale Multi-Robot Learning

    RoboNet contains 15 million frames across 7 robot platforms with deformable object annotations

    arXiv ↩
17. CVAT polygon annotation manual

    MOT Challenge format and evaluation protocols for pedestrian tracking

    docs.cvat.ai ↩
18. appen.com data annotation

    Appen quality frameworks using consensus labeling and golden sets

    appen.com ↩
19. RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning

    RLDS ecosystem for generating, sharing, and using reinforcement learning datasets

    arXiv ↩
20. LeRobot dataset documentation

    LeRobot dataset format using Parquet for tabular data with separate video files

    Hugging Face ↩
21. Datasheets for Datasets

    Datasheets for Datasets template covering 57 questions across dataset lifecycle

    arXiv ↩
22. truelabel data provenance glossary

    Truelabel provenance system linking annotations to collector hardware and conditions

    truelabel.ai ↩
23. C2PA Technical Specification

    C2PA content credentials for cryptographic provenance in media files

    C2PA ↩
24. docs.labelbox.com overview

    Labelbox platform for cloud-based annotation workflows

    docs.labelbox.com ↩
25. v7darwin.com data annotation

    V7 automation suite claims 90% pre-annotation accuracy after 2,000 training examples

    v7darwin.com ↩
26. MCAP guides

    MCAP format for multi-sensor data with nanosecond timestamps

    MCAP ↩
27. Reading from a ROS bag file

    ROS bag files for multi-sensor robotics data storage

    docs.ros.org ↩
28. kognic.com platform

    Kognic platform for automotive and robotics annotation with sensor fusion

    kognic.com ↩
29. Creative Commons Attribution-NonCommercial 4.0 International deed

    CC BY-NC 4.0 restricts commercial use of datasets

    creativecommons.org ↩
30. EPIC-KITCHENS-100 annotations license

    EPIC-KITCHENS-100 custom non-commercial license restrictions

    GitHub ↩
31. GDPR Article 7 — Conditions for consent

    GDPR Article 7 requires explicit consent for personal data collection

    GDPR-Info.eu ↩
32. truelabel physical AI data marketplace bounty intake

    Truelabel marketplace connects buyers to 12,000+ collectors for physical AI datasets

    truelabel.ai ↩
33. RLDS GitHub repository

    RLDS GitHub repository with format conversion scripts

    GitHub ↩
34. LeRobot GitHub repository

    LeRobot GitHub repository with data loaders and annotation tools

    GitHub ↩

FAQ