VLA Benchmarks Explained: Octo, OpenVLA, ManiSkill, LIBERO & SIMPLER

Quick facts

Topic

VLA benchmark landscape (Octo, OpenVLA, ManiSkill, LIBERO, SIMPLER)

Audience

Robotics and ML engineers evaluating VLA policies

Deliverable

Benchmark map + protocol checklist + held-out eval sourcing path

The three benchmarks a VLA policy actually gets scored on

When someone reports that a vision-language-action model "hits X% success," the first question worth asking is: on what? A raw success rate is meaningless until you know the task suite, the embodiment, and the evaluation protocol behind it. For open policies like Octo and OpenVLA, three benchmarks show up again and again, and they measure genuinely different things.

ManiSkill is a GPU-parallelized simulator you can run thousands of environments on at once, so it is where people stress-test contact-rich control and generalization at scale^[1]. LIBERO is a fixed suite of 130 teleoperated tasks split so that each subset changes exactly one thing — where the objects are, which object it is, what the goal is, or how long the horizon runs^[2]. SIMPLER is the odd one out: it exists to answer whether a simulated score predicts what the robot will do on a real table^[3].

Most confusion about VLA benchmarks comes from treating these as interchangeable leaderboards. They are not. A policy can look strong in ManiSkill's parallel sim and mediocre on LIBERO's long-horizon split, and both numbers can be honest. The rest of this guide is about reading them correctly — and about the gap between any of them and your own deployment.

Why an Octo ManiSkill score and an Octo LIBERO score aren't the same currency

The instinct is to line benchmark numbers up side by side and pick the winner. Resist it. Each suite fixes a different variable and randomizes the rest, so identical-looking percentages describe different experiments.

Simulation benchmarks like ManiSkill can regenerate physics, lighting, and object placement on every episode, which makes them great for measuring robustness under distribution shift but disconnected from real sensor noise and real contact dynamics^[1]. LIBERO runs from human teleoperated demonstrations on a fixed task set, so its numbers speak to imitation quality and forgetting under continual learning rather than raw physical robustness^[2]. SIMPLER deliberately builds its scenes to mirror real evaluation setups so its scores can be correlated against physical trials^[3]. Three different questions, three different answers.

The practical rule: a benchmark number is a coordinate, not a rating. It only locates a policy relative to other policies measured with the exact same protocol, seeds, and episode count. Cross-suite comparisons — or worse, comparisons against a number pulled from a different paper's table — are how teams talk themselves into models that fail on their own hardware.

ManiSkill: simulation fast enough to be a benchmark

ManiSkill's whole argument is throughput. Built on the SAPIEN engine, it runs simulation and rendering on the GPU in parallel, collecting RGBD and segmentation data at more than 30,000 frames per second on a single RTX 4090^[1]. That speed is what turns simulation from a demo into a benchmark — you can run enough episodes across enough randomized conditions to get a success rate with a tight confidence interval instead of a lucky-run anecdote.

It also supports heterogeneous parallel environments, where every parallel instance can hold a different scene and object set^[1]. For a generalist policy that is the interesting stress test: not "can you pick this cube 100 times" but "can you pick across hundreds of object and layout variations sampled simultaneously." Tasks span table-top manipulation, drawing and cleaning, and dexterous work across several robot embodiments.

The catch is the one every simulator carries. ManiSkill tests policy behavior against a physics model, not against the world. Friction, deformation, sensor latency, and lighting on a real rig will not match the sim exactly, so a strong ManiSkill number is evidence a policy learned the task structure — not proof it will transfer. That is precisely the gap SIMPLER was built to quantify.

LIBERO: four suites, one variable each

LIBERO is a lifelong-learning benchmark: 130 manipulation tasks built from human teleoperated demonstrations, grouped into four task suites so you can isolate what actually breaks a policy^[2]. The design is the useful part. Three of the suites each perturb a single factor — the spatial arrangement of otherwise-identical scenes, the identity of the objects, or the declared goal of the task — while the fourth targets long-horizon behavior where a policy has to chain many steps without drifting^[4].

Because the variables are separated, a per-suite breakdown tells you where a model is fragile rather than just how fragile it is overall. A policy that holds up when you move objects around but collapses when the goal changes has a language-grounding problem, not a control problem. That diagnostic value is why LIBERO shows up as a fine-tuning target for VLAs — it reveals failure modes an aggregate score hides.

The lifelong framing matters too. LIBERO is meant to probe catastrophic forgetting and knowledge transfer as a policy learns tasks in sequence, so results depend heavily on task ordering and the training regime^[2]. Read a LIBERO table with the setup in view: a number from a from-scratch sequential-learning run is not the same as one from a policy fine-tuned on all suites at once.

SIMPLER: the sim-to-real correlation check

SIMPLER (Simulated Manipulation Policy Evaluation for Real Robots) answers the question the other two dodge: does the simulator agree with reality? It provides simulated environments purpose-built to mirror real evaluation setups on the Google Robot and WidowX/BridgeData platforms, then evaluates the same set of policies — RT-1, RT-1-X, RT-2-X, and both Octo-Base and Octo-Small — in sim and on hardware^[3].

The headline is not a single success rate. It is a correlation. SIMPLER reports how well simulated performance tracks real-world performance using Mean Maximum Rank Violation (does sim preserve the real-world ranking of policies?) and Pearson correlation (do the absolute numbers move together?)^[3]. A benchmark that scores benchmarks.

Why this belongs in a VLA evaluation kit: if you are going to make a purchasing or deployment decision from a simulated number, you need evidence that the sim is predictive for your kind of task. SIMPLER makes that evidence explicit for a specific set of policies and setups. Outside that validated envelope — a new embodiment, a new task family, real clutter the sim never modeled — the correlation is an open question, and the honest answer is to measure on real held-out data.

Octo vs OpenVLA: two architectures, two evaluation profiles

The two open policies most people benchmark come at the problem from opposite ends, and that shapes how their scores read.

Octo is small and fast. It is a transformer generalist policy with a diffusion action head that predicts action chunks, pretrained on 800,000 trajectories drawn from 25 Open X-Embodiment datasets, and it ships in a 27M-parameter Small and 93M-parameter Base variant that run on consumer GPUs^[5]. It accepts language or goal-image conditioning, which is unusual — most VLAs are language-only. The design bet is deployability: enough capability to fine-tune onto your robot, small enough to run in real time.

OpenVLA is large and language-first. It is a 7B-parameter model built on a Llama 2 7B backbone with fused DINOv2 and SigLIP vision encoders, and it represents actions as discretized action tokens — the continuous action dimensions are binned and emitted as tokens, not produced by a diffusion or continuous head^[6]. (If you have seen OpenVLA described with a diffusion action head, that is wrong; the base model uses discretized tokens.) Trained on 970,000 real-world demonstrations, it outperforms the closed RT-2-X (55B) by 16.5 percentage points of absolute task success across 29 tasks^[6]. The trade is cost: more parameters, heavier inference, stronger language grounding.

So the two land in different regions of every benchmark. Octo tends to look good where fast fine-tuning and action smoothness matter; OpenVLA tends to look good where language generalization and semantic breadth matter. Neither is "the better VLA" in the abstract — the right question is which profile matches your task and your latency budget, and the only way to settle it is to evaluate both on data that looks like yours. A fine-tuning pass on your embodiment is where these paper numbers either hold up or fall apart.

How to read a VLA benchmark table without fooling yourself

Benchmark tables are easy to misread in ways that cost real money once you commit hardware and a data budget. A short protocol before you trust any number:

1. 01

   Name the suite and the split

   "90% on LIBERO" is not a claim until you know which suite. LIBERO-Long behaves nothing like the single-variable spatial split. ManiSkill, LIBERO, and SIMPLER are not comparable to each other at all.

2. 02

   Check the embodiment

   A number on a WidowX or Google Robot setup tells you little about your Franka or your mobile base. Cross-embodiment transfer is the whole difficulty; assume it degrades until measured.

3. 03

   Count the episodes

   Success rates from 10-20 rollouts are noise. You want enough episodes per task for a stated confidence interval, and you want the seeds fixed so the run is reproducible.

4. 04

   Find the evaluation protocol

   Was it zero-shot, or fine-tuned on the target suite? Sequential lifelong learning, or trained on everything at once? The regime moves the number more than the architecture does.

5. 05

   Ask if sim was validated against real

   If the headline is a simulated score, look for a SIMPLER-style correlation result for that policy and setup. Without it, a sim number is a hypothesis about the real world, not a measurement of it.

The number benchmarks can't give you: your own held-out eval set

Every benchmark above is someone else's task distribution. ManiSkill's objects, LIBERO's kitchens, SIMPLER's validated setups — none of them are your bin, your lighting, your gripper, or your acceptance criteria. A model that tops all three can still miss your spec, because your spec was never in the test.

That is why the held-out evaluation set is the artifact that actually de-risks a deployment. Before you fund a large capture program off a VLA training data order, you want a small, rights-cleared set of trajectories on your embodiment and your tasks, deliberately held out of training, delivered in RLDS, LeRobot, or MCAP with per-trajectory provenance so the eval is auditable. You fine-tune on the training split, you score on the held-out split, and now the number means something — it was measured on the distribution you are actually shipping into.

On Truelabel you post a spec describing the embodiment, tasks, and the diversity axes you need held out, and matched capture partners return a sample packet with QA evidence before you commit to scale. It is the difference between quoting a paper and measuring your own robot.

Related pages

Use these to move from category-level context into specific task, dataset, format, and comparison detail.

External references and source context

1. Project site

   ManiSkill: SAPIEN-powered GPU-parallelized robot simulation collecting RGBD + segmentation at 30,000+ FPS on a single RTX 4090, with heterogeneous parallel environments

   maniskill.ai ↩
2. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning

   LIBERO: lifelong robot learning benchmark of 130 tasks across four suites built from human-teleoperated demonstrations

   arXiv ↩
3. Evaluating Real-World Robot Manipulation Policies in Simulation

   SIMPLER: simulated environments for evaluating real-robot manipulation policies (RT-1, RT-1-X, RT-2-X, Octo-Base, Octo-Small) with Mean Maximum Rank Violation and Pearson correlation against paired real-world results

   arXiv ↩
4. Dataset page

   LIBERO task-suite structure and downloadable teleoperated demonstration datasets

   libero-project.github.io ↩
5. Octo: An Open-Source Generalist Robot Policy

   Octo: transformer generalist policy, diffusion action head predicting action chunks, Octo-Small 27M / Octo-Base 93M, 800K trajectories from 25 Open X-Embodiment datasets, language and goal-image conditioning, RSS 2024

   arXiv ↩
6. OpenVLA: An Open-Source Vision-Language-Action Model

   OpenVLA: 7B-parameter open VLA on a Llama 2 7B backbone with fused DINOv2 + SigLIP vision encoders, discretized action tokens, trained on 970K real-world demonstrations, beats RT-2-X (55B) by 16.5% absolute success across 29 tasks

   arXiv ↩

FAQ