8 Types of Annotation for Robotics and Physical-AI Data

Why robotics annotation is a different problem

When a robot fails — reaches for the wrong object, clips a shelf, follows the words but misses the intent — the first instinct is to inspect the model. Often the real defect sits upstream in the training set: data collected in one context, labeled in another, and consumed as if those differences did not matter. A warehouse arm needs different labels than a mobile robot in a hospital corridor, and a vision-language-action (VLA) model trained on teleoperation traces needs richer supervision than a generic image detector.

That is why the consumer-style discussion of annotation types is not enough here. Robotics data is not a pile of still images. It is synchronized RGB, depth, LiDAR point clouds, proprioception, IMU, audio, and language tied to a shared timeline — which is exactly why labeling for robotics differs from standard image annotation, as Encord's overview of robotics data labeling describes. Teams also need labels that respect RLDS, MCAP, and per-trajectory provenance so the dataset stays useful after procurement, ingestion, and training. The eight types below are the ones robotics engineers and data buyers actually have to specify, review, and QA.

Spatial labels: boxes, segmentation, keypoints, and pose

Four annotation families answer the question "where is it, and how do I act on it?"

Bounding boxes and 3D object detection anchor an object in space, not just on the image plane. A rectangle around a carton is fine for a retail detector but useless for a robot that has to approach, avoid collisions, and decide whether the object is reachable from the current wrist pose. Boxes still earn their place for first-pass perception, triaging cluttered scenes, and sourcing large varied datasets — especially when paired with occlusion flags and visibility metadata. The rule that saves rework: define the box around the physical object extent the robot cares about, not the prettiest silhouette in the frame, and state whether annotators box visible extent, amodal extent, or both.

Semantic segmentation adds per-pixel meaning; panoptic annotation adds instance identity on top, so the system can separate one bin from the next or track the specific plate it selected rather than the class "plate." Dense masks are expensive to produce and review, so they earn their cost only when success depends on contact geometry, free space, or material boundaries. Define classes around what the robot must do — traversable surface, manipulable object, fixed structure, deformable obstacle, transient hazard — not by appearance, because a transparent divider, a hanging cable, and a painted floor stripe look similar but create very different planning constraints.

Keypoints and pose close the gap between "object detected" and "task executed." A box can be accurate while the geometry is wrong — the gripper closes on a connector and still misses the port. Sparse keypoints mark the hinge axis of pliers, the centerline of a drawer handle, or the fingertips of a teleoperator hand; full 6D pose defines the rigid transform for grasp planning, insertion, and placement. Pose projects fail at the interface with robotics infrastructure, so the spec — not the label name — decides usability: coordinate frame, rotation convention, calibration version, timestamp source, and occlusion policy all have to be written down.

Geometry labels: dense depth and surface normals

Depth and surface-normal labels matter when the task depends on contact, clearance, slope, or support — the layer that turns "object detected" into "reachable, graspable, traversable, or placeable." A warehouse robot reaching for a shrink-wrapped case on a reflective pallet can have a clear RGB frame and still fail because the surface angle was wrong and measured depth broke at the plastic wrap.

This label type also exposes a basic sourcing trap: two vendors can both deliver a "depth map," but one provides metric depth in the camera frame while another provides disparity-like values with undocumented post-processing. Those are not interchangeable. Write down whether depth is z-distance in meters, radial distance, disparity, or a rendered depth image, and state the invalid-pixel policy for glass, mirrors, water, motion blur, and sensor shadows. For normals, define the coordinate frame, vector convention, and whether labels are per-pixel unit vectors or piecewise planar regions.

QA has to test where robots actually fail — thin edges, reflective metal, transparent bins, black plastic, fork tines, cable bundles — because a colored depth map can look clean and still be unusable for control. A practical pass reprojects depth onto RGB to check edge alignment, compares adjacent frames for flicker, audits glossy and dark surfaces separately, and confirms uncertain pixels stay marked uncertain rather than being silently filled.

Time and motion: action, trajectory, temporal, and optical flow

Robots operate in time, and three annotation families capture what static labels cannot.

Action and trajectory annotation connects observation to control. When a manipulator bumps a drawer handle, backs off, and opens it on the second try, labeling the whole episode as one "open drawer" action throws away the hesitation, recovery, and contact sequence that matter for imitation learning and VLA training. Action labels cover task verbs (reach, align, grasp, lift, insert, retract, handoff, abort); trajectory labels cover end-effector pose, joint states, gripper state, contact onset, and teleoperation inputs. Keep the action set small enough that two trained annotators agree, and tie every boundary to something observable — first stable contact, gripper closure past a threshold, object lift-off — so "grasp starts at first closing motion after the target enters the gripper envelope" replaces vague wording that produces noisy labels.

Temporal event and state-change annotation marks what changed, when, and how long the new condition held. It separates two things teams wrongly merge: an event is a bounded change (contact, slip onset, drop, emergency stop); a state is a condition that persists (object in gripper, drawer ajar, human in safety zone). Policies often learn from state occupancy while failure analysis depends on precise event timing, so they need different rules — and a conflict rule for when sensors disagree or the view is occluded. If two reviewers cannot resolve a boundary dispute from the written spec, the taxonomy is not ready for sourcing.

Optical flow and motion annotation helps a robot anticipate movement instead of classifying a frame after the fact — crossing pedestrians, opening gates, a hand approaching versus retreating. Combined with depth it becomes scene flow. Model baselines like RAFT or GMFlow can draft it, but robotics teams still need review rules for occlusion, disocclusion, motion blur, rolling shutter, and low-texture regions, plus validity maps that preserve low-confidence pixels rather than painting over them.

Natural language and spatial reasoning for VLA data

Language annotation is what turns a vision-language-action sample into something trainable, rather than a generic captioning example. Given "take the tote beside the blue pallet and place it under the conveyor sensor," a loose caption like "robot moves a box" loses everything execution depends on: which tote, which reference object, which spatial relation, and what final state should be true.

The useful unit is rarely free text alone. It is free text plus grounded entities, spatial relations, temporal qualifiers, and links to sensor or action records in RLDS or MCAP. "Pick the red mug" should point to an object instance; "approach from the left" needs a declared reference frame (camera-left, robot-left, or table-left); "place it inside the sink" should resolve to a container relation checkable in the final frames. The expensive ambiguity is cross-modal: two annotators can both write valid English while disagreeing on whose left is meant, and that error compounds across egocentric video, fixed third-person cameras, and teleoperation logs.

Language QA for robotics should not stop at grammar. Review whether a model, operator, or downstream engineer could execute the instruction from the annotation alone: can every referred object be uniquely identified, is each spatial relation resolvable from the stated frame, and do final-state phrases match the visible outcome? A strong sample packet includes crowded shelves, mirrored scenes, repeated objects, and human-robot handoffs — the cases that expose whether a supplier produces grounded language or polished but vague description.

• Reference frame policy: robot-centric, camera-centric, world-centric, or task-centric — declared, not assumed.
• Grounding format: whether text spans map to boxes, masks, tracks, or persistent object IDs.
• Spatial ontology: a fixed relation set (left of, inside, under, facing, inserted into) with edge cases.
• Provenance for multilingual data: keep original instruction, translation, and grounding IDs tied together so QA can tell translation errors from scene ambiguity.

The 8 annotation types at a glance

Use this to match a label family to your binding constraint. Complexity and cost rise roughly from top to bottom, but the right choice is driven by what the robot has to decide, not by what is cheapest to produce.

From spec to scale: sourcing a dataset that matches your robot

Knowing the annotation types you need is only the start. Most robotics teams do not fail because they chose the wrong label family — they fail because they sourced data that did not match the robot, the environment, or the training format. A clean box dataset from the wrong camera placement will not rescue a manipulation model, and a beautifully segmented scene with missing trajectory provenance will not help in an RLDS or MCAP pipeline.

The market backdrop reinforces the point. Straits Research projects the data annotation tools market to grow from USD 3.14 billion in 2026 to USD 29.82 billion by 2034 at a 32.5% CAGR, with manual workflows still holding 53.40% share in 2025 while automatic techniques advance at a 23.97% CAGR to 2031, per its data annotation tools market report. For robotics that does not mean "automate everything" — manual review still matters where semantics, contact intent, or phase boundaries are subtle. Credence Research adds an operational signal: image annotation held 35.74% market share in 2025 and video 26%, while 3D and point-cloud workflows are projected to grow at a 22.45% CAGR to 2031, and standardized outputs like RLDS and MCAP correlate with better downstream integration, as noted in its market analysis of annotation tools. Format conformity and provenance discipline save real time after delivery.

The practical procurement pattern is straightforward: write one spec tied to robot, task, and environment; define the ontology, metadata, coordinate frames, and delivery schema; then require a sample packet before scale and review it with the people who own perception, controls, and data infrastructure. If the sample satisfies only one of those groups, it is not ready. A marketplace reduces fragmentation — instead of managing separate capture vendors, annotators, and licensing threads, teams route one specification through a system that returns rights-cleared samples, standardized metadata, and delivery-ready files across egocentric, exocentric, teleoperation, and cinematic captures, standardized into RLDS, LeRobot, MCAP, or custom schemas. The most durable habit is simple: do not buy "annotated data," buy a spec-compliant dataset with reviewable provenance, sample-backed QA, and labels that match how the robot acts in the world.

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

FAQ