The LeRobot Dataset Format: Schema, v3.0 Migration, and Visualizer

Quick facts

Topic

LeRobot dataset format (v3.0 schema, migration, visualizer)

Audience

Robotics ML engineers, data engineers, ML ops

Deliverable

Practitioner reference on the LeRobotDataset format + custom-delivery path

LeRobotDataset: the format and the ecosystem, disambiguated

"LeRobot dataset" names two things at once, and conflating them is the most common source of confusion. The first is a file format — a specific on-disk layout for robot-learning trajectories. The second is the Hugging Face Hub ecosystem that distributes those files: you call `push_to_hub()` on a recorded dataset and stream it back later with a single constructor. The format is what this guide dissects; the ecosystem is why it matters, because a dataset published in LeRobot format is directly consumable by the training scripts shipped in Hugging Face's LeRobot library^[1] — ACT, Diffusion Policy, and the VLA fine-tuning recipes — with no custom loader.

LeRobot was introduced in the 2024 LeRobot paper^[2] as a PyTorch-native home for real-world robotics, deliberately lower-friction than the TensorFlow-centric RLDS^[3] standard that preceded it. Where RLDS wraps TFRecord shards for petabyte-scale distributed loading, LeRobot optimizes for a contributor pushing a dataset from a laptop and a researcher pulling three episodes to debug a policy. That design bias — ease of contribution over framework lock-in — is why the public LeRobot catalog spans arms, humanoids, mobile bases, and driving.

The current format is LeRobotDataset v3.0^[4], and it rests on three pillars. Tabular data — low-dimensional, high-frequency signals such as joint states, actions, and timestamps — lives in Apache Parquet^[5], memory-mapped or streamed through the Hugging Face `datasets` stack. Visual data — camera frames — is concatenated and encoded into MP4, sharded per camera so files stay a practical size. Metadata in JSON and Parquet records the feature schema (names, dtypes, shapes), the frame rate, per-feature normalization statistics, and the episode-segmentation offsets that reconstruct individual episodes from shared files. Keep those three pillars in mind; every design decision in v3.0 follows from them.

The v3.0 storage model: file-based, not episode-based

The defining principle of v3.0 is decoupling storage from the user API: data is stored as a few large files, while the Python API still exposes intuitive episode-level access. Concretely, tabular rows and video frames from many episodes are concatenated into shared Parquet and MP4 shards, and episode-specific views are reconstructed through metadata rather than file boundaries^[4]. This is the single biggest change from v2.1, which wrote one Parquet file and one MP4 per episode — an approach that buckled at scale, where a 50,000-episode dataset became 50,000 tiny files that throttled filesystem initialization and Hub sync.

The on-disk layout is small enough to memorize. Under `meta/`, `info.json` holds the canonical schema — feature names, shapes, dtypes, FPS, codebase version, and the path templates used to locate shards; `stats.json` holds global mean/std/min/max per feature for normalization and is exposed at runtime as `dataset.meta.stats`; `tasks.jsonl` maps natural-language task strings to integer IDs for task-conditioned policies; and `episodes/` stores per-episode records — lengths, task assignments, and byte/frame offsets — as chunked Parquet so the index itself scales to millions of episodes^[4]. Alongside `meta/`, the `data/` directory holds the frame-by-frame Parquet shards (each file typically many episodes) and `videos/` holds the per-camera MP4 shards (again, many episodes each).

Two consequences fall out of this for anyone evaluating a dataset. First, you cannot tell how many episodes a v3.0 dataset contains by counting files — you read `meta/info.json`. Second, the offset table in `meta/episodes/` is load-bearing: a corrupt or truncated episodes index breaks episode reconstruction even when the underlying Parquet and MP4 are intact, which is why the API forces a `finalize()` call before publishing.

Migrating v2.1 to v3.0 (and the finalize() gotcha)

If you hold a v2.1 dataset — the one-file-per-episode layout — you migrate it with the official converter, which aggregates the per-episode files into shards and writes the offset metadata. LeRobotDataset v3.0 ships in `lerobot >= 0.4.0`; until that stable release you install a pre-release build from source^[4]. The converter is `lerobot.scripts.convert_dataset_v21_to_v30`, invoked with a single `--repo-id` pointing at the Hub dataset. It does three things: aggregates `episode-0000.parquet`, `episode-0001.parquet`, and so on into `file-0000.parquet`; aggregates the matching `episode-0000.mp4`, `episode-0001.mp4`, and so on into `file-0000.mp4`; and rewrites the chunked Parquet under `meta/episodes/` with per-episode lengths, tasks, and the byte/frame offsets that make the concatenated files addressable again^[4].

The one gotcha that bites everyone recording a new v3.0 dataset — not migrating — is `finalize()`. v3.0 writes Parquet incrementally with buffered metadata, so you must call `dataset.finalize()` after your last `save_episode()` and before `push_to_hub()`. `finalize()` flushes buffered episode metadata, closes the Parquet writers so they write their footer, and validates the dataset; skip it and the Parquet footers never get written, leaving corrupt files that will not load^[4]. If you inherit a dataset that loads locally but fails on the Hub, an un-finalized upload is the first thing to check.

1. 01

   Install a v3-capable LeRobot

   LeRobotDataset v3.0 ships in lerobot >= 0.4.0; until that release, install a pre-release build from the LeRobot main branch as documented in the migration guide.

2. 02

   Run the converter

   Invoke python -m lerobot.scripts.convert_dataset_v21_to_v30 with --repo-id set to your Hub dataset id. It aggregates per-episode Parquet and MP4 into file-level shards and rewrites the episode-offset metadata.

3. 03

   Verify episode reconstruction

   Load the migrated dataset and index a few episodes; confirm episode lengths, task labels, and camera streams match the source. The meta/episodes offset table is what maps shared shards back to episodes.

4. 04

   For newly recorded sets, finalize before push

   When recording rather than migrating, call dataset.finalize() after the last save_episode() and before push_to_hub() so Parquet footers and buffered metadata are written; otherwise the files are corrupt.

Reading the data: features, tensors, and temporal windows

At training time you construct `LeRobotDataset(repo_id)` and index it like any PyTorch dataset; each sample is a dictionary of tensors keyed by feature name — `observation.state` for proprioception, `action` for the commanded action, `observation.images.<camera>` for each camera (e.g. `observation.images.front_left`), and `timestamp`^[4]. The object plugs straight into `torch.utils.data.DataLoader`, so the format never forces a bespoke collation path. Feature names are namespaced by convention, which is what lets a policy config address cameras and state channels by string rather than by positional index.

Two API features are worth internalizing because they shape how you consume the format. `delta_timestamps` requests a temporal window around each sampled frame: pass a dict mapping a feature to a list of second offsets (say `[-0.2, -0.1, 0.0]`) and indexing returns a stacked `[T, C, H, W]` tensor instead of a single frame — this is how you feed frame-stacked observations to Diffusion Policy or action-chunked models without hand-rolling a buffer^[4]. `StreamingLeRobotDataset` iterates a dataset directly from the Hub with no local download and no full load into memory, which is the payoff of the file-based v3.0 layout: a multi-terabyte corpus becomes trainable from a laptop or an ephemeral cloud node^[4]. Normalization draws on the `meta/stats.json` values exposed as `dataset.meta.stats`, so you standardize actions and states against dataset-global statistics rather than recomputing them per run.

Because the tabular and visual pillars are separate, image augmentation is a training-time concern, not a storage one: raw frames are stored untransformed and image transforms (color jitter, sharpness, or any `torchvision.transforms.v2` op) are applied on load, so you can retune augmentations without re-recording^[4].

Visualizing a LeRobot dataset

Before you train on a dataset — yours or a stranger's — look at it. LeRobot provides two visualizers. The online visualizer is a hosted Hugging Face Space: open the LeRobot dataset visualizer^[6], paste any Hub repo id, and browse episodes, camera streams, and action/state traces in the browser without downloading a byte. Its source is a self-hostable web app^[7] if you need it behind your own auth or pointed at a private dataset.

For local inspection during collection or debugging, the CLI tool `lerobot-dataset-viz` renders a single episode. Point it at a Hub dataset with `--repo-id` and `--episode-index`, or add `--root` and `--mode local` to read a dataset on disk; by default it opens a Rerun viewer^[8] showing synchronized camera streams, robot states, and actions^[8]. Add `--display-mode foxglove` to serve the episode over a WebSocket — connect the Foxglove app to `ws://127.0.0.1:8765` — as a seekable timeline you can scrub, play, and pause^[8]. In practice the visualizer is the fastest way to catch the failure modes that pass a schema check but ruin training: a camera that dropped frames mid-episode, an action channel pinned at zero, or a proprioception trace that flatlines when a joint encoder glitched.

Editing datasets: split, merge, filter, re-encode

A recorded dataset is rarely training-ready on the first pass, and the format is built to be edited without decoding back to raw frames. The `lerobot-edit-dataset` CLI is the single entry point^[8]. Its operations, selected with `--operation.type`, cover the real editing workflow: `delete_episodes` drops bad takes by index (optionally writing to a new `--new_repo_id` so the original is preserved); `split` divides a dataset into named subsets by fraction or by explicit episode indices; `merge` concatenates multiple datasets that share an identical feature schema; `remove_feature` strips a camera or channel you will not train on; and `info` prints episode count, frame count, and file size without touching the data^[8].

Two storage-oriented operations matter for delivery. `convert_image_to_video` re-encodes an image-based dataset into MP4-backed video, collapsing storage and speeding loads, with separate encoder settings for RGB and depth cameras — depth maps are detected automatically and routed to a depth encoder that persists its quantization parameters to metadata, so depth can be dequantized back to physical units on load^[8]. `reencode_videos` swaps codec or quality on an existing video dataset in place while preserving those depth-quantization parameters. Any operation takes `--push_to_hub true` to publish the result. This tooling is why LeRobot format works as a pipeline rather than just an archive: the same commands a lab uses to clean its captures are the ones a supplier uses to shape a delivery to a buyer's exact feature schema.

From public sets to custom capture

The strongest public datasets in LeRobot format are ports of Open X-Embodiment^[9] — the aggregation of over one million real-robot trajectories across many embodiments — surfaced as sets like `bridge_orig` (WidowX manipulation) and DROID (76,000 Franka teleoperation demonstrations across hundreds of scenes)^[10], the same pretraining pool that OpenVLA^[11] and RT-2-style models draw on. You can browse the ones profiled in the catalog on the LeRobot datasets hub, and interconvert with TensorFlow-native pipelines via the RLDS and LeRobot formats guide.

The limit of public sets is coverage: they fix an embodiment, a task distribution, and a license you may not be able to ship a product on. When the format is right but the data is not — you need a specific gripper, a task the public corpora skip, or commercial rights the research licenses withhold — the answer is custom capture delivered in the same format. Truelabel sources physical-AI data^[12] and delivers it as LeRobotDataset (or RLDS, or MCAP), rights-cleared, with per-trajectory provenance and a sample packet before scale, so it drops into the exact `LeRobotDataset(repo_id)` call your training loop already runs. You get the ecosystem's tooling — visualizer, edit CLI, streaming — on data scoped to your task rather than someone else's.

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

External references and source context

1. LeRobot GitHub repository

   LeRobot GitHub repository shipping the dataset class and the ACT, Diffusion Policy, and VLA training scripts

   GitHub ↩
2. LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch

   LeRobot paper introducing the PyTorch-native robotics dataset and model ecosystem

   arXiv ↩
3. RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning

   RLDS paper describing the TensorFlow-based trajectory format that preceded LeRobot

   arXiv ↩
4. LeRobot dataset documentation

   LeRobotDataset v3.0 documentation: three-pillar storage, directory layout, migration script, finalize() requirement, streaming and delta_timestamps API

   Hugging Face ↩
5. Apache Parquet file format

   Apache Parquet file-format specification for the columnar tabular storage used by LeRobotDataset

   Apache Parquet ↩
6. LeRobot dataset visualizer (Hugging Face Space)

   Hosted LeRobot dataset visualizer Hugging Face Space for browsing any Hub dataset online

   Hugging Face ↩
7. LeRobot dataset visualizer (self-hostable web app)

   Self-hostable LeRobot dataset visualizer web app, the source behind the hosted Space

   GitHub ↩
8. Using Dataset Tools (LeRobot)

   LeRobot dataset tools docs: lerobot-edit-dataset operations and the lerobot-dataset-viz local visualizer with Rerun and Foxglove display modes

   Hugging Face ↩
9. Open X-Embodiment: Robotic Learning Datasets and RT-X Models

   Open X-Embodiment paper aggregating over one million real-robot trajectories across many embodiments

   arXiv ↩
10. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

    DROID paper reporting 76,000 Franka teleoperation demonstrations across hundreds of scenes

    arXiv ↩
11. OpenVLA: An Open-Source Vision-Language-Action Model

    OpenVLA paper describing the open-source vision-language-action model pretrained on Open X-Embodiment data

    arXiv ↩
12. truelabel physical AI data marketplace bounty intake

    Truelabel physical AI data marketplace for custom rights-cleared capture and delivery

    truelabel.ai ↩

FAQ