RoboDojo Benchmark Reveals a 12.8% Success Rate for Top Robot Models

How high is this mountain called General Robot?

Over the past year, VLA, robotic foundation models, and world models have emerged in succession.

Each demo is becoming smoother: stacking bowls, inserting tubes, organizing items, pouring water, tidying the desk—the robot finally seems to understand human instructions, comprehend the world, and start taking action.

But the question is: which of these models is truly stronger, and in what ways? Can they transition from simulation to the real world? How far are we from achieving a true general-purpose manipulation robot?

Now, a new "mountain climbing roadmap" has arrived.

The same team behind the RoboTwin series of benchmarks has introduced RoboDojo: a unified benchmark for simulating and evaluating real-world robotic operations.

Website: https://robodojo-benchmark.com/ arXiv: https://arxiv.org/abs/2607.04434 Leaderboard: https://robodojo-benchmark.com/LeaderBoard Benchmark code: https://github.com/RoboDojo-Benchmark/RoboDojo XPolicyLab code: https://github.com/XPolicyLab/XPolicyLab Community: https://robodojo-benchmark.com/community

It’s not just another benchmark—it’s more like setting a “Mount Everest” for embodied intelligence:

42 simulated tasks, 18 real robot tasks, and 30 leading robot strategies compete side by side, evaluating five key capabilities: generalization, memory, fine manipulation, long-horizon execution, and open-ended semantic understanding.

The result is straightforward and harsh:

The strongest general-purpose robot strategy currently has an average success rate of only 8.80% in simulations. In the real world, the best model achieves an average success rate of just 12.8%.

And what about human experts? In the simulation, it was 76.03%; in the real world, it was 100%.

Robot base models appear to have begun climbing the summit of embodied intelligence, but RoboDojo’s leaderboard shows most are still at the base camp acclimatizing to the altitude.

The challenge with RoboDojo lies not in simply accumulating tasks, but in breaking down robotic capabilities into a series of "climbing levels" that more closely resemble real-world scenarios.

In a simulated environment, RoboDojo has designed 42 tasks centered around five core competencies:

Generalization: Can the model adapt to new backgrounds, lighting conditions, objects, and complex, cluttered scenes?

Memory: Check whether the model can remember previously seen information and use it in subsequent actions.

Precision: See if the model can perform high-precision operations such as insertion, alignment, and exact contact.

Long-horizon: Can the model handle multi-step tasks with strong dependencies and accumulating errors?

Open, see if the model can understand unseen open semantic instructions and turn language goals into actions.

These tasks are not simple pick-and-place variants.

For example, in generalization tasks, up to 25 desktop items can be randomly selected, with changes to the background, lighting, object appearance, and layout;

In the memory task, the robot must remember objects that appeared and then disappeared on the conveyor belt, and then select the matching target from subsequent candidates.

In delicate operation tasks, robots must perform high-precision actions such as tube insertion, alignment, and placement—slight deviations can lead to failure.

Long-term tasks are more akin to real household chores: instead of performing a single action, the robot must sequentially complete multiple sub-steps—picking up, moving, transferring, aligning, and placing—each of which may introduce errors that accumulate over time.

But RoboDojo didn't stop at simulation.

What truly elevated this "embodied Everest" is that it moved evaluations onto real robots.

RoboDojo designed 18 real-world tasks covering the ARX X5, Piper, and Piper X dual-arm robotic platforms, with six tasks for each platform.

These tasks are not exact replicas of simulation tasks, but are specifically designed to evaluate the robot's deployment capabilities in the real physical world.

For example, the ARX X5 handles tasks such as stacking blocks, baking bread, preparing food, filling and emptying fruit containers, storing items in a safe, and plugging in tubes; the Piper handles tasks like stacking and covering blocks, filling pen holders, placing objects into baskets, plugging in chargers, stacking bowls, and uprighting bottles; the Piper X includes tasks such as object classification, disassembling LEGO bricks, hanging cups, packing items into a backpack, cleaning up blocks, and capping pens.

These tasks may sound routine, but they are not simple for robots.

In the real world, each step carries physical uncertainty: objects may slip, grippers may not hold securely, robotic arms may have slight delays, cameras may introduce noise, and the moment of contact may push the target off course.

More importantly, RoboDojo-RealEval standardizes real-machine evaluations: unifying hardware configurations, workspace layouts, lighting conditions, scene reset procedures, evaluation protocols, and deployment interfaces.

Before each test, evaluators will recreate the scenario according to a predefined layout; each trial is also scored by three reviewers using a double-blind process, assessing both the final success and the completion of intermediate steps.

In other words, RoboDojo’s physical component is not just about “recording a few demo videos,” but rather transforming real robot operations into a standardized, reproducible, comparable, and remotely accessible examination.

In other words, RoboDojo doesn’t just ask robots in simulation, “Can you solve the problem?”—it also tests them in the real world: Will the robot still perform stably if swapped for another? Will it shake upon physical contact? Can it correct itself if the object is slightly offset? Can it recover if an error occurs midway through a task? And can it keep climbing mountains after leaving the simulation training ground?

This is the true meaning of "Embodied Everest": it’s not about achieving peak performance in a single capability, but about ensuring neither simulation-based diagnosis nor real-world deployment falters.

The core of RoboDojo is its public leaderboard.

This is also what sets it apart from many evaluations that use their own models to test themselves:

RoboDojo is initiated and maintained by a consortium of academic institutions, with no commercial interests involved. The governance of the rankings is managed by the non-profit AI MMLab Club Foundation.

In other words, this "embodied Everest" is not a viewing platform built by a single company for itself, but rather a public climbing route open to the entire community.

On the simulation leaderboard, the team integrated and evaluated 30 representative robotic operation strategies, including Hy-Embodied-0.5-VLA, Spatial Forcing, π0.5, X-VLA, GR00T-N1.7, π0, and OpenVLA-OFT.

The top-ranked model is Hy-Embodied-0.5-VLA, with an average score of 13.07 and an average success rate of 8.80%.

Followed by models such as Spatial Forcing, π0.5, and X-VLA, overall performance remains in a very low range.

Even leading models do not excel in all five key capability dimensions.

Some models generalize better, others perform more stably in fine-grained tasks, and some can advance several steps further in long-range tasks—but once viewed on the full leaderboard, their weaknesses become very apparent.

A key message from RoboDojo is that today’s robotic models aren’t incapable of movement—they’re just unstable; they aren’t unable to perform tasks—they just struggle to complete them consistently.

Many strategies can complete some steps, but ultimately have a very low success rate.

For example, in long-range tasks, the robot may have already picked up the object and moved near the target, but fails at the final alignment, insertion, placement, or recovery stage.

This is also the key difference between embodied intelligence and purely linguistic or purely visual tasks: in the physical world, being slightly off means failure.

If simulation is still a "training ground," then real robots are the "real-world Everest."

In the real-world rankings, the top-performing model is π0.5, with an overall success rate of 12.8% and an average score of 22.9.

The top-tier models include InternVLA-A1, GalaxeaVLA, Xiaomi-Robotics-0, and X-VLA, but the overall success rate remains in the single digits to just a few percent.

This highlights a critical issue: performing well in simulations does not guarantee stability in the real world.

Real robots introduce additional challenges: camera noise, calibration errors, robotic arm latency, unstable contact, motion jitter, safety boundaries, and small deviations in object initial positions. These issues are often invisible in demo videos but are prominently exposed during standardized evaluations.

The meaning of RoboDojo lies here: it doesn't just ask, "Did the robot succeed?" but rather asks:

Can this strategy pass a comprehensive evaluation in simulation while also standing up to real-world challenges?

The results reveal a realistic assessment by RoboDojo: the current growth in capabilities of foundational robot models is uneven.

Some models better identify targets, some execute actions more smoothly, and some advance more steps in long-term tasks.

But a truly general-purpose bot cannot be strong only in one specific capability dimension.

It must be understandable and memorable; it must be well-planned and precisely executed; it must handle familiar tasks while also comprehending open-ended instructions; it must run successfully in simulation and execute stably on real robotic arms.

However, RoboDojo's experimental results show that today's models still have significant shortcomings in these dimensions.

The most typical example is the Open task, where even the strongest models achieve a success rate of only about 1.67%.

This means that current robot base models are still significantly far from truly “understanding human language and reliably getting things done.”

They can mimic familiar tasks, but the chain of semantic understanding, visual localization, skill selection, and action execution remains fragile when faced with new goals, new semantics, or new combinations.

This is precisely the challenge of embodied Everest: it’s not about excelling at a single skill, but about ensuring no capability falls short.

RoboDojo also has two other important components.

One is heterogeneous parallel simulation.

Traditional simulation parallelization typically involves replicating the same scenario with only the initial positions changed; RoboDojo supports running different tasks, objects, and layouts simultaneously, significantly improving evaluation efficiency.

The other is XPolicyLab.

It serves as the unified access layer behind RoboDojo, specifically designed to address a practical challenge in evaluating robot strategies: different models often have varying data formats, preprocessing pipelines, training scripts, action representations, and deployment environments, making the engineering cost of fairly comparing them on a single leaderboard extremely high.

What XPolicyLab does is standardize these external processes.

It provides unified data transformation, training templates, deployment processes, and evaluation scripts, while preserving the original model architectures and implementations of each strategy.

With this setup, different robot strategies can run on both the RoboDojo simulation environment and the RoboDojo-RealEval real-world platform simply by connecting to the unified observation-action interface.

In this paper, the team has integrated 30 representative robotic operation models through XPolicyLab.

For researchers, this means the model can be “integrated once, evaluated everywhere”: first rapidly iterate and diagnose performance gaps in simulation, then deploy to real robots for standardized testing.

Therefore, RoboDojo is not just a static benchmark in academic papers, but a continuously updated embodied intelligence arena.

Models can continuously rank on the leaderboard, tasks can be continuously expanded, and real robot evaluations can be remotely accessed.

This is important for the field of robotic foundation models.

On the path to general-purpose robots, everyone needs not only larger models and more impressive demos, but also a consistent "altitude gauge" to measure progress.

In the past, the robotics field was often demo-driven.

A model that can perform several impressive tasks can easily create the illusion that general-purpose robots are just around the corner.

But RoboDojo’s conclusion is more measured: while the current model is indeed improving, it is still far from achieving reliable, generalizable, and deployable general robotic manipulation.

This is not bad news.

On the contrary, RoboDojo makes it clear: who can generalize, who forgets, who trembles in motion, who only completes half the task, who falls behind in the real world, and who can climb the leaderboard.

Embodied intelligence is no longer just about competing through promotional videos—it’s now about delivering real results on standardized tracks.

This "embodied Everest" has been erected. Now, it’s up to who will reach the summit first.

Chen Tianxing is a direct Ph.D. student at the MMLab of the University of Hong Kong, supervised by Professor Luo Ping.

Has published over ten papers at top-tier conferences such as ICML, CVPR, ICLR, and RSS, and has won multiple best paper awards at conference workshops, as well as first and second places in several academic competitions at top conferences.

Recognized as one of AI25 (Under 25 AI Innovators) by Sequoia Capital China and MIT Technology Review China, awarded the Special Prize (highest student honor) at Shenzhen University, and named one of the 99 Outstanding Undergraduates by the China Computer Federation (CCF).

The first author of RoboTwin 2.0 and founder of Lumina, a leading embodied open-source community, with open-source projects collectively earning nearly 20,000 stars on GitHub.

Chen Yue, a master's student at Peking University, primarily researches 3D visual representation and robotic simulation.

More than ten high-level papers have been published in CCF Class A and CAAI Class A journals, with several results presented as Oral or Spotlight presentations. Related work has received Best Paper awards at international conferences such as CVPR and IROS. The author has been honored with the National Scholarship and the Peking University Outstanding Student Award.

The RoboDojo team will continue to release evaluations on dexterous manipulation, mobile manipulation, haptic manipulation, and humanoid full-body operations—stay tuned for more updates.

This article is republished with permission from QbitAI; the views expressed are solely those of the original author.

This article is from the WeChat public account "Quantum Bit," authored by Yunzhong.