RLDS format for robot training data

Quick facts

Origin

Google Research — RLDS spec at github.com/google-research/rlds, used as the canonical episode container in Open X-Embodiment.

Open X-Embodiment

1M+ trajectories from 22 robot embodiments and 21 institutions ship in RLDS (October 2023).

RT-X / OpenVLA

Both train on RLDS-formatted data; RLDS preserves the (observation, action, reward, discount, metadata) tuple needed for policy learning.

Required fields

Episode ID, observation stream, action stream, timestamps, task label, and success flag.

Comparison

What is RLDS?

RLDS should be requested when the buyer's training or evaluation pipeline already expects reinforcement learning and robotics episodes with observations, actions, rewards, and metadata. Anchor the bounty to the canonical specification before suppliers submit samples ^[1], then use implementation documentation to make the expected file layout reviewable ^[2]. Robotics teams should also name the dataset or paper lineage they expect suppliers to support ^[3].

"RLDS is an ecosystem of tools to store, retrieve and manipulate episodic data in the context of Sequential Decision Making including Reinforcement Learning, Learning from Demonstrations, Offline RL or Imitation Learning."

For truelabel buyers, that quote matters because it turns RLDS format from a generic delivery preference into a source-backed requirement the supplier can test against a sample file.

Using RLDS with robot data

A useful RLDS sample should prove episode ID, observation stream, action stream, timestamps, task label, and success flag, plus file naming, manifest completeness, timestamp behavior, and rejected-example traceability. Include at least one workflow or converter reference so the supplier can show how the files load in practice ^[4], one interoperability reference for adjacent formats ^[5], and one comparison source for why this format is preferable to a raw folder dump ^[6].

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

External references and source context

1. RLDS: Reinforcement Learning Datasets

   RLDS stores episodic reinforcement-learning and robot learning data.

   GitHub ↩
2. RLDS with TensorFlow Datasets

   TensorFlow Datasets documents RLDS support for episode datasets.

   TensorFlow ↩
3. Open X-Embodiment: Robotic Learning Datasets and RT-X Models

   Open X-Embodiment is a robotics dataset and model reference relevant to RLDS delivery.

   arXiv ↩
4. TF-Agents Trajectory API

   Trajectory structures connect observations, actions, rewards, and step types.

   TensorFlow ↩
5. LeRobot documentation

   LeRobot documentation is relevant when converting robotics datasets into training-ready formats.

   Hugging Face ↩
6. HDF5 1.14 documentation

   HDF5 is a structured data container often compared with RLDS episode storage.

   The HDF Group ↩

FAQ