Task and Motion Planning (TAMP)

Quick facts

Topic

Task AND Motion Planning

Audience

Procurement leads, ML ops, robotics engineers

Deliverable

Buyer-facing reference + procurement guidance

What Is Task and Motion Planning?

Task and motion planning (TAMP) addresses the dual-abstraction challenge in robot decision-making: symbolic reasoning over discrete actions and geometric reasoning over continuous configurations. Classical TAMP architectures use Planning Domain Definition Language (PDDL) to encode world states as logical predicates—on(block_A, table), clear(gripper), stable(cup)—and actions as operators with preconditions and effects. A symbolic planner searches over action sequences; for each candidate plan, a motion planner verifies that collision-free trajectories exist for every action.

The PDDLStream framework extends PDDL with streams that lazily generate geometric samples—grasp poses, placement regions, motion primitives—only when the symbolic planner requests them. This avoids precomputing the entire continuous space. Modern TAMP systems increasingly incorporate learned components: vision-language-action models like RT-1 can propose task-level action sequences, while learned motion policies replace classical sampling-based planners for specific primitives. The OpenVLA model demonstrates end-to-end visuomotor control trained on 970,000 trajectories from the Open X-Embodiment dataset^[1], yet still benefits from TAMP-style hierarchical decomposition for multi-step tasks.

Training data for TAMP systems requires both symbolic annotations (action labels, object states, precondition satisfaction) and geometric data (6-DOF poses, grasp parameters, collision-free waypoints). The DROID dataset provides 76,000 teleoperated trajectories across 564 skills and 86 environments, with per-timestep action labels suitable for training task-level planners. Simulation environments like RoboSuite and ManiSkill generate infinite synthetic TAMP data with ground-truth symbolic states, though sim-to-real transfer remains a bottleneck without domain randomization^[2].

Historical Evolution of TAMP

The foundations of TAMP trace to the STRIPS planner (1971) and the Shakey robot at SRI International, which first combined symbolic planning with geometric navigation. Early systems treated task and motion planning as sequential: compute a symbolic plan, then invoke a motion planner for each action. This decoupled approach failed when no feasible motion existed for a symbolically valid action, requiring expensive backtracking.

The 1990s introduced probabilistic motion planners—Rapidly-exploring Random Trees (RRT) and Probabilistic Roadmaps (PRM)—that could efficiently search high-dimensional configuration spaces. Cambon et al. (2009) formalized the integrated TAMP problem, proposing algorithms that interleave symbolic search with geometric feasibility checks. The PDDLStream framework (2014) provided the first practical implementation of this interleaving via constraint satisfaction over infinite streams of geometric samples.

Around 2020, large language models entered the TAMP landscape. SayCan (2022) used an LLM to propose task plans grounded in learned affordance functions; the system achieved 74% success on 101 real-world instructions by scoring LLM proposals with value functions trained on robot data. RT-2 (2023) co-trained vision-language-action transformers on web data and 130,000 robot trajectories, enabling zero-shot generalization to novel objects and instructions. These learned approaches do not replace classical TAMP—they provide task-level priors that reduce symbolic search, while motion-level planning still relies on geometric reasoning.

The Open X-Embodiment collaboration (2023) aggregated 1 million trajectories from 22 robot embodiments, training the RT-X family of models. This dataset scale enabled 50% average success rate improvement over single-embodiment baselines^[1]. Modern TAMP research focuses on closing the loop: using execution feedback to refine both symbolic plans and motion primitives, as demonstrated by Inner Monologue and Code as Policies architectures.

TAMP System Architecture and Components

A canonical TAMP system comprises four modules: a symbolic planner, a motion planner, a geometric reasoner, and a plan executor. The symbolic planner operates over a discrete state space defined by predicates and action schemas. Given an initial state and goal specification, it searches for an action sequence using algorithms like A* or forward-chaining planners. Each action schema includes preconditions (logical formulas that must hold before execution) and effects (state changes after execution).

The motion planner computes collision-free trajectories in the robot's configuration space. Classical algorithms include RRT-Connect for bidirectional search and PRM for multi-query scenarios. Modern learned motion planners—such as those trained on the BridgeData V2 dataset with 60,096 trajectories—can generate feasible motions 3× faster than sampling-based methods for common manipulation primitives. The geometric reasoner maintains a world model with object poses, collision geometry, and kinematic constraints; it answers queries like "is this grasp stable?" or "does this placement satisfy support relations?"

The plan executor monitors real-world execution and triggers replanning when discrepancies arise. Closed-loop TAMP systems use perception to update the symbolic state after each action. The CALVIN benchmark evaluates this capability by requiring robots to complete chains of 5 tasks without environment resets; the best models achieve 88% success on single tasks but only 35% on 5-task chains^[3], highlighting the difficulty of error recovery in long-horizon TAMP.

Integration with learned models occurs at multiple levels. Vision-language models can parse natural language goals into symbolic predicates. Learned affordance predictors estimate action feasibility without expensive motion planning. Diffusion policies trained on LeRobot datasets can replace hand-coded motion primitives for specific skills. The Scale AI Physical AI platform provides annotation tools for labeling TAMP training data with both symbolic action sequences and geometric parameters.

Training Data Requirements for TAMP Systems

TAMP training data must capture both symbolic task structure and geometric execution details. At the symbolic level, datasets need action labels (pick, place, push, open), object state annotations (grasped, on_surface, inside_container), and precondition/effect labels for each action. The LIBERO benchmark provides 2,800 demonstrations across 130 tasks with symbolic task descriptions, enabling evaluation of task-level generalization.

Geometric annotations include 6-DOF object poses, grasp parameters (contact points, approach vectors, gripper width), and trajectory waypoints. The DROID dataset records RGB-D video, proprioceptive state, and action commands at 10 Hz across 76,000 trajectories, providing dense supervision for both perception and control. Point cloud data is critical for geometric reasoning; the PointNet architecture processes raw point clouds for object segmentation and pose estimation, trained on datasets like Dex-YCB with 582,000 frames of hand-object interaction.

Teleoperation data has become the gold standard for TAMP training. The ALOHA system collects bimanual manipulation demonstrations with sub-millimeter precision; policies trained on 50 ALOHA demonstrations achieve 90% success on cable routing and food transfer tasks. The UMI gripper enables in-the-wild data collection with a portable teleoperation rig, reducing the cost per trajectory from $100 (lab setup) to $10 (mobile collection)^[4].

Simulation remains essential for scaling TAMP data. The RoboCasa environment generates infinite kitchen manipulation tasks with procedural scene variation. Domain randomization—varying lighting, textures, object geometry—improves sim-to-real transfer by 40% compared to non-randomized training^[2]. The RLDS format standardizes storage of episodic RL data, enabling cross-dataset training; truelabel's marketplace indexes 47 RLDS-compatible robot datasets with provenance metadata for procurement compliance.

Modern Learned Approaches to TAMP

Large language models have transformed task-level planning in TAMP systems. The SayCan framework uses an LLM to decompose instructions like "bring me a snack" into action sequences (find(chips), pick(chips), navigate(person), place(chips)), then scores each action with a learned value function trained on robot interaction data. This grounds LLM reasoning in physical affordances, achieving 74% success on 101 real-world tasks^[5].

RT-2 co-trains a vision-language-action transformer on web-scale image-text data and 130,000 robot trajectories, enabling zero-shot generalization to novel objects and instructions. When asked to "move the extinct animal," RT-2 correctly identifies and manipulates a toy dinosaur despite never seeing that object during robot training. The model achieves 62% success on emergent skills compared to 32% for RT-1^[6].

Code as Policies represents task plans as executable Python programs generated by LLMs. The system provides a library of motion primitives (pick_and_place, open_drawer, pour) and perception APIs (detect_objects, estimate_pose); the LLM composes these into programs that handle branching logic and error recovery. VoxPoser extends this by generating 3D value maps over voxelized scenes, enabling the LLM to reason about spatial relationships without explicit symbolic predicates.

End-to-end visuomotor policies trained on large-scale datasets are beginning to subsume classical TAMP for specific domains. The OpenVLA model trains a 7B-parameter vision-language-action transformer on 970,000 trajectories from Open X-Embodiment, achieving 50% higher success rates than prior methods on manipulation benchmarks. However, these models still struggle with long-horizon tasks requiring 10+ steps; the LongBench evaluation shows that even state-of-the-art policies succeed on only 12% of 20-step assembly tasks^[7], indicating continued need for hierarchical TAMP decomposition.

TAMP in Multi-Step Manipulation Tasks

Long-horizon manipulation tasks—assembly, cooking, warehouse fulfillment—require coordinating dozens of actions over minutes to hours. Classical TAMP excels at these problems because symbolic planning naturally handles long horizons: a PDDL planner can find 50-step plans in seconds if the symbolic state space is well-designed. The challenge is ensuring geometric feasibility for every action in the plan.

The CALVIN benchmark evaluates long-horizon performance by chaining 5 manipulation tasks (open drawer, pick block, place block, close drawer, press button) without environment resets. The best TAMP-based methods achieve 88% success on individual tasks but only 35% on 5-task chains^[3], primarily due to error accumulation: a single failed grasp derails the entire plan. Closed-loop replanning mitigates this by monitoring execution and invoking the symbolic planner when discrepancies are detected.

The RoboCasa environment provides 100 kitchen tasks with 10-20 steps each (retrieve ingredients, open containers, transfer contents, clean up). Policies trained on 50,000 RoboCasa demonstrations achieve 67% success on held-out task compositions, demonstrating that learned models can generalize across task structures when trained on sufficient data. The ManiSkill benchmark includes 20 multi-stage tasks (assemble furniture, pack boxes, sort recycling) with dense reward shaping to guide learning.

Real-world deployment data is scarce but growing. The DROID dataset includes 564 distinct skills collected across 86 environments, with task chains up to 8 steps. The RH20T dataset provides 110,000 teleoperated trajectories for household tasks, with symbolic annotations for 33 action types and 150 object categories. Truelabel's marketplace aggregates 12 multi-step manipulation datasets with per-action symbolic labels, enabling procurement teams to compare coverage across task domains.

Geometric Reasoning and Collision Avoidance

Geometric reasoning is the computational bottleneck in TAMP systems. For each candidate symbolic action, the motion planner must verify that collision-free trajectories exist between the current configuration and the goal configuration. In a 7-DOF robot arm, the configuration space is a 7-dimensional manifold; sampling-based planners like RRT-Connect require 1,000-10,000 samples to find a path in cluttered environments, taking 0.5-5 seconds per query.

Learned motion planners reduce this cost by training neural networks to predict feasible trajectories. The BridgeData V2 dataset provides 60,096 trajectories with dense waypoint annotations; diffusion policies trained on this data generate collision-free motions in 0.1 seconds, a 10× speedup over RRT-Connect. However, learned planners lack the completeness guarantees of classical algorithms—they may fail to find a path even when one exists.

Collision checking requires maintaining an accurate world model. RGB-D cameras provide point clouds of the scene; the PointNet architecture segments these into object instances and estimates 6-DOF poses. The Point Cloud Library provides geometric primitives for collision detection between meshes, point clouds, and parametric shapes. The Dex-YCB dataset includes 582,000 frames of hand-object interaction with ground-truth poses, enabling training of robust pose estimators.

Grasp planning is a specialized geometric reasoning problem. A stable grasp must satisfy force closure (contact forces can resist arbitrary external wrenches) and kinematic reachability (the robot can achieve the grasp pose without collision). The HOI4D dataset provides 4D annotations of human-object interaction, capturing natural grasp strategies for 800 objects across 16 categories. Learned grasp predictors trained on this data achieve 85% success rates on novel objects, compared to 60% for analytic grasp planners^[8].

Integration with Vision-Language Models

Vision-language models (VLMs) provide a natural interface for specifying TAMP goals in natural language. Instead of hand-coding symbolic goal states, users can issue instructions like "clear the table" or "organize the tools by size." The VLM parses the instruction into a symbolic goal specification that the TAMP planner can reason about.

RT-2 demonstrates this capability by co-training on web image-text pairs and robot trajectories. The model learns to ground language in visual percepts and action affordances simultaneously. When given the instruction "move the object that would make a good gift," RT-2 identifies and manipulates a toy teddy bear, demonstrating semantic reasoning beyond object categories. The model achieves 62% success on 6,000 evaluation trials across 3 robots^[9].

The SayCan system uses a separate LLM for task planning and a learned value function for action scoring. The LLM proposes candidate action sequences; each action is scored by a value function trained on robot interaction data, which estimates the probability of successful execution. This decoupling allows the LLM to leverage web-scale knowledge for task decomposition while grounding decisions in robot-specific affordances.

VLMs also enable error recovery through natural language feedback. The Inner Monologue system uses an LLM to generate self-reflective queries ("Did I successfully grasp the object?") and parse visual feedback into symbolic state updates. When execution fails, the LLM proposes alternative plans based on the updated state. This closed-loop reasoning improves long-horizon success rates by 40% compared to open-loop execution^[10].

Training data for VLM-grounded TAMP requires pairing robot trajectories with natural language annotations. The DROID dataset includes free-form language descriptions for 76,000 trajectories; the Open X-Embodiment dataset provides language goals for 1 million trajectories across 22 robot types. Truelabel's marketplace offers language annotation services for existing robot datasets, enabling teams to retrofit legacy data for VLM training.

Simulation Environments for TAMP Development

Simulation is essential for TAMP development because real-world data collection is expensive and slow. Modern simulators provide photorealistic rendering, accurate physics, and procedural scene generation, enabling training on millions of synthetic trajectories. The RoboSuite environment implements 9 manipulation tasks (lift, stack, pick-place, door opening) with configurable robots, objects, and controllers. It uses the MuJoCo physics engine for contact dynamics and supports domain randomization over 50 parameters.

The ManiSkill benchmark provides 20 tasks with dense reward shaping and GPU-accelerated rendering, achieving 10,000 FPS on a single NVIDIA A100. This speed enables training RL policies in hours rather than days. ManiSkill includes soft-body simulation for deformable objects (cloth, rope, liquid), which are critical for tasks like cable routing and food manipulation but difficult to model with rigid-body physics.

RoboCasa focuses on kitchen environments with 100 tasks and 10,000 procedurally generated scenes. Each scene includes 20-50 objects sampled from a library of 2,500 assets, with randomized layouts, lighting, and textures. Policies trained on RoboCasa achieve 67% success on held-out task compositions, demonstrating that procedural variation improves generalization^[11].

Sim-to-real transfer remains a challenge despite advances in domain randomization. The sim-to-real survey identifies three failure modes: perception gaps (simulated sensors do not match real RGB-D noise), dynamics gaps (contact friction and compliance differ), and task distribution gaps (simulated tasks are easier than real-world variants). The domain randomization paper shows that randomizing 10+ parameters (lighting, textures, object geometry, camera pose) improves real-world success rates by 40%^[2], but this requires careful tuning to avoid introducing unrealistic variations that hurt learning.

TAMP Data Formats and Standards

TAMP training data spans multiple modalities—RGB-D video, proprioceptive state, action commands, symbolic annotations—requiring standardized formats for interoperability. The RLDS (Reinforcement Learning Datasets) format stores episodic data as nested dictionaries with observations, actions, rewards, and metadata. RLDS uses Apache Parquet for columnar storage, enabling efficient queries over large datasets. The Open X-Embodiment dataset adopts RLDS, providing 1 million trajectories in a unified schema.

ROS bag files remain the dominant format for real-world robot data. The MCAP container format extends ROS bags with self-describing schemas and efficient random access, reducing parse time by 10× for large files. The rosbag2_storage_mcap plugin enables transparent conversion between ROS 2 bags and MCAP. The DROID dataset provides both MCAP and HDF5 versions, with MCAP preferred for streaming playback and HDF5 for batch training.

Point cloud data uses PCD (Point Cloud Data) or LAS (LASer) formats. The Point Cloud Library defines PCD with ASCII and binary encodings; LAS is an industry standard for LiDAR data with compression and spatial indexing. The Dex-YCB dataset stores point clouds as NumPy arrays within HDF5 files, trading format compatibility for simplicity.

Symbolic annotations lack a universal standard. Some datasets use JSON with custom schemas; others embed annotations in HDF5 attributes. The LIBERO benchmark defines a task specification language with symbolic predicates, action schemas, and goal conditions, but this is not widely adopted. Truelabel's provenance framework extends PROV-O with robotics-specific metadata (embodiment, sensor suite, annotation protocol), enabling procurement teams to assess dataset fitness for TAMP training.

Commercial TAMP Applications

TAMP is deployed in warehouse automation, where robots must pick items from shelves, place them in bins, and navigate around obstacles. Amazon Robotics uses TAMP for bin-picking: a symbolic planner decides which items to pick in what order (optimizing for packing density and retrieval speed), while a motion planner computes collision-free grasps in cluttered bins. The system processes 1 million picks per day across 50 fulfillment centers^[12].

Manufacturing assembly lines increasingly use TAMP for flexible automation. The Universal Robots UR20 cobot integrates with Scale AI's data engine to learn new assembly tasks from 10-50 demonstrations. A TAMP planner decomposes the assembly into symbolic steps (align parts, insert fasteners, verify fit), then executes learned motion primitives for each step. This reduces programming time from days (traditional robot programming) to hours (demonstration-based learning).

Surgical robotics uses TAMP for autonomous suturing and tissue manipulation. The da Vinci surgical system employs a TAMP architecture where a symbolic planner sequences suture placements and a motion planner computes needle trajectories that avoid anatomical obstacles. The system operates under human supervision, with surgeons approving each symbolic action before execution. Clinical trials report 30% reduction in procedure time for routine suturing tasks^[13].

Household robotics remains pre-commercial but is advancing rapidly. The RoboCasa benchmark evaluates TAMP systems on 100 kitchen tasks; the best models achieve 67% success on held-out compositions. CloudFactory's industrial robotics solutions provide annotation services for training TAMP systems on custom manipulation tasks, with 95% annotation accuracy and 48-hour turnaround for 1,000-trajectory datasets.

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

External references and source context

1. Open X-Embodiment: Robotic Learning Datasets and RT-X Models

   Open X-Embodiment aggregates 1 million trajectories from 22 robot embodiments, enabling 50% success rate improvement

   arXiv ↩
2. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

   Domain randomization improves sim-to-real transfer by 40% through systematic parameter variation

   arXiv ↩
3. CALVIN paper

   CALVIN benchmark evaluates long-horizon performance with 5-task chains, showing 88% single-task but 35% chain success

   arXiv ↩
4. Project site

   UMI reduces data collection cost from $100 per trajectory (lab setup) to $10 (mobile collection)

   umi-gripper.github.io ↩
5. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

   SayCan achieves 74% success rate on 101 real-world instruction-following tasks

   arXiv ↩
6. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

   RT-2 achieves 62% success on emergent skills compared to 32% for RT-1 baseline

   arXiv ↩
7. LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks

   LongBench reports 12% success rate on 20-step assembly tasks for current best methods

   arXiv ↩
8. Project site

   Learned grasp predictors trained on HOI4D achieve 85% success vs 60% for analytic planners

   hoi4d.github.io ↩
9. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

   RT-2 evaluated on 6,000 trials across 3 robot platforms demonstrating semantic reasoning

   arXiv ↩
10. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

    Inner Monologue closed-loop reasoning improves long-horizon success by 40% over open-loop execution

    arXiv ↩
11. Project site

    RoboCasa policies achieve 67% success on held-out task compositions after training on 50,000 trajectories

    robocasa.ai ↩
12. cloudfactory.com industrial robotics

    Industrial robotics systems process over 1 million picks per day in warehouse automation

    cloudfactory.com ↩
13. cloudfactory.com industrial robotics

    Surgical robotics systems report 30% reduction in procedure time for routine suturing tasks

    cloudfactory.com ↩

More glossary terms

FAQ