Benchmarking Real-World Robotic Manipulation through Stochastic Challenges

Elias Gläßer, Gülhan Şikaroğlu, Pavlo Kupyn, Marc Pollefeys, Zuria Bauer, Hermann BlumarXiv 2026 · arXiv

TL;DR

This paper introduces stochastic challenges, a new benchmarking paradigm that evaluates robotic manipulation by standardizing task distributions rather than fixed test sets. By combining seeded scene randomization with a rigorous statistical framework, the authors enable reproducible, cross-lab comparisons of heterogeneous robots (like the Boston Dynamics Spot and Hello Robot Stretch 3) without requiring identical hardware or physical travel.

What Problem Does This Paper Address?

Evaluating how well a robot manipulates objects in the real world is notoriously difficult. Unlike computer vision, where you can test an algorithm on a static dataset, manipulation is interactive—the robot’s actions change the environment.

Current solutions have major drawbacks:

  • Simulation is fast but suffers from the "sim-to-real" gap.
  • Centralized competitions (like RoboCup) require expensive shipping of robots to one location.
  • Fixed benchmarks (like the YCB object set) often use the same static setups, which can lead to "overfitting," where researchers tune their systems to succeed on specific hand-picked scenes rather than general tasks.
  • Distributed benchmarks often force everyone to use the same robot hardware, limiting innovation in robot design.

The authors aim to create a "hidden test set" for the physical world: a way for any lab to generate a unique but standardized challenge that is statistically comparable to challenges faced by other robots elsewhere.

Figure 1: Stochastic Challenges target several desired aspects of robotic benchmarking: (i) system- level innovation, i.e. not restricting evaluation to specific hardware; (ii) local evaluation, avoiding the effort to transport systems and reducing the chance of errors when trying to reproduce methods; (iii) reduced overfitting, testing true generalization instead of pre-known tasks.

Methods

The core of the paper is the stochastic challenges framework. Instead of defining a single setup, it defines a distribution of setups.

  1. Seeded Randomization: A central randomizer uses a "seed" (a number) to generate a specific configuration specification. This spec tells a researcher exactly how to set up a scene (e.g., which shelf gets a door, which gets a drawer, and where the objects go). Because it’s seeded, any lab can perfectly reconstruct the exact same "random" scene.
  2. Variance Decomposition: The authors provide a mathematical framework to handle two types of uncertainty:
  3. Trial Noise: The robot might succeed on a task once and fail the next time due to sensor noise or physics.
  4. Configuration Difficulty: Some random scenes are just harder than others.
  5. Statistical Rigor: They derive formulas to determine exactly how many different scenes ($K$) and how many repeated trials ($J$) are needed to prove that Robot A is actually better than Robot B with statistical confidence (e.g., $p < 0.05$).
Figure 2: Concept overview. Seeded randomization generates reproducible KALLAX manipulation scenes; robots across labs (Stretch 3 and Spot) execute sampled configurations, with outcomes sta- tistically aggregated for robust comparison.

Datasets

The authors introduce the KALLAX Benchmark, built around the ubiquitous IKEA KALLAX shelf system.

  • The Setup: A 3x4 grid of cubbies that can be fitted with 82 different KALLAX-compatible inserts (doors, drawers, open boxes).
  • The Tasks: Opening doors/drawers, picking objects (using 79 different YCB objects), and placing/moving items.
  • The Platform: A web interface that generates configurations from seeds, provides assembly instructions, and records results to ensure transparency and prevent selective reporting of "only the good runs."

Key Insights and Ideas

  • The "Floor" of Uncertainty: A key insight is that no matter how many times you repeat a single task, your statistical uncertainty is eventually limited by the number of different scenes you test. To get more precise results, you must sample more seeds, not just more trials on the same seed.
  • Paired vs. Unpaired Evaluation: Testing two robots on the exact same seeds (paired) is much more statistically efficient but risks overfitting. Testing on different seeds (unpaired) better measures true generalization. The authors suggest a mixed design (sharing some seeds but not all) as a practical middle ground.
  • Cross-Embodiment Comparison: Because the benchmark standardizes the environment and the task distribution rather than the robot, it allows for the first time a rigorous comparison between a wheeled mobile manipulator (Stretch 3) and a quadruped with an arm (Spot) across different countries.

Key Results

The researchers put their framework to the test by comparing a Hello Robot Stretch 3 (at the University of Bonn) and a Boston Dynamics Spot (at ETH Zürich).

  • Performance Gap: In the door-opening task, Spot achieved a success rate of ~71% vs. Stretch’s ~18%. In drawer-opening, Spot achieved ~88% vs. Stretch’s ~64%.
  • Statistical Significance: Using their framework, they confirmed these gaps were "distinguishable." For drawers, the observed gap of 0.244 was greater than the calculated "detectable margin" of 0.232, meaning the result was statistically significant even with a relatively small number of seeds.
  • Efficiency Gains: They showed that by sharing just 50% of the seeds between labs (the mixed design), they could reduce the detectable margin by about 20-25%, making the benchmark much more efficient.
Figure 3: Experimental setup. Our KALLAX benchmark draws sample configs from the same distribution, which are then recreated in two different labs to evaluate the two different robotic systems.

Limitations and Future Work

  • Sample Complexity: Real-world testing is slow. The authors acknowledge that a "considerable number of trials" is still required to reach high statistical confidence, which is a hurdle for smaller labs.
  • Human-in-the-loop: Currently, humans must manually set up the shelves and label successes. The authors suggest that future versions could use Vision-Language Models (VLMs) to automatically verify task success from video.
  • Reconstruction Variability: While the seeds standardize the plan, different humans might still set up the "same" scene slightly differently (e.g., a door hinge being slightly tighter). Quantifying this "reconstruction noise" is a goal for future studies.

Summary updated

Abstract

A challenging bottleneck in manipulation research is reproducibility and comparability of methods in real-world environments. Since manipulation is an interactive task, it cannot be evaluated on a static dataset of observations. Benchmarking in simulation is a common remedy, but cannot cover the whole complexity of real-world tests. This paper proposes a new paradigm of stochastic challenges. Like the well-known YCB objects, it is based on reproducible real-world scenes and therefore allows system-level innovation and comparability between different platforms. Due to their stochastic nature, our challenges however, mimic the rigor and surprise of hidden test sets. To this end, we show how different robots in different labs can be compared based on their success rate over different random task samples, if these samples come from the same distribution. We propose stochastic challenges as a general paradigm for robot benchmarking and implement this paradigm in a novel benchmark for mobile manipulation, on which we compare two robots tested in two different labs.
BibTeX
@misc{glaer2026benchmarking,
  author = {Elias Gläßer and Gülhan Şikaroğlu and Pavlo Kupyn and Marc Pollefeys and Zuria Bauer and Hermann Blum},
  title = {{Benchmarking Real-World Robotic Manipulation through Stochastic Challenges}},
  year = {2026},
  eprint = {2611.12804},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12804},
}
Figure