Is Your Imitation Learning Policy Better than Mine? Policy Comparison with Near-Optimal Stopping

David Snyder, Asher James Hancock, Apurva Badithela, Emma Dixon, Patrick Miller, Rares Andrei Ambrus, Anirudha Majumdar, Masha Itkina, Haruki Nishimura

TL;DR

Evaluating and comparing robot manipulation policies is notoriously time-consuming and expensive. STEP (Sequential Testing for Efficient Policy Comparison) is a novel statistical framework that allows researchers to safely stop evaluation trials early as soon as enough evidence is gathered to declare a winner. By optimizing stopping boundaries offline, STEP reduces the number of required evaluation trials by up to 32% compared to existing baselines while strictly preventing the statistical sin of "p-hacking."

What Problem Does This Paper Address?

In robotics and imitation learning, determining whether a new policy ($\pi_1$) outperforms a baseline ($\pi_0$) typically requires running dozens of tedious physical or simulated trials. Because continuous reward functions are hard to define, researchers usually rely on binary success/failure metrics.

Standard practice uses batch testing (e.g., running exactly 50 trials and calculating a p-value). However, batch testing is highly inflexible:

  1. Wasted Effort: If a new policy is obviously better, you still have to finish all 50 trials.
  2. The P-Hacking Trap: If the results are neck-and-neck after 50 trials, you cannot simply run 20 more trials and re-test. Doing so dynamically alters the statistical properties of the test, inflating false-positive rates (Type-I error)—a phenomenon known as p-hacking.

The authors address this by introducing a mathematically rigorous sequential testing framework tailored for the small-sample-size regime common in robotics.

Methods

The authors propose STEP (Sequential Testing for Efficient Policy Comparison). Instead of waiting until the end of a fixed batch, STEP checks the data after every single trial to decide whether to Continue, Reject the Null ($\pi_1 > \pi_0$), or Accept the Null ($\pi_1 \le \pi_0$).

Fig. 1: Robot policy comparison problem under binary success/failure metrics. Novel policy π1 is compared against baseline π0 in a sequence of trials. Within a given evaluation budget, the evaluator seeks a statistically significant comparison in as few trials as possible. Due to the cost of hardware setup and calibration, as well as the limited inference speed of complex policies, these results generally arrive in sequence from a single (or few) hardware setups. Allowing the evaluator to adaptively and near-optimally tailor the number of trials based on the data observed so far — without compromising statistical assurances of the comparison — is a central contribution of this work.

Key technical components include:

  • Control-Theoretic State Representation: The evaluation state is modeled as a discrete integrator tracking the cumulative successes of both policies and the current trial number.
  • Composite-vs-Composite Testing: Unlike simple hypothesis tests, policy comparison deals with infinite possible true success rates. The authors frame this as a distributionally robust control problem over the entire set of null hypotheses.
  • Tractable Offline Optimization: To find the optimal stopping boundaries, the authors discretize the worst-case null hypotheses and solve a sequence of simple linear programs offline. This maximizes the statistical power and minimizes the expected sample size under a strict user-defined "risk budget" $f(n)$.
The policy comparison problem as a composite-vs-composite statistical test.

Datasets and Tasks

The framework was validated using a mix of extensive numerical simulations and real-world robotic tasks:

  • Numerical Simulation: 45 distinct policy pair combinations evaluated over 5,000 simulated trajectories to map out exact error rates and statistical power.
  • Real-World Hardware Tasks (Franka Emika Panda & WidowX 250S):
  • FoldRedTowel: Bimanual towel-folding task trained via diffusion policies (300 human demonstrations).
  • CleanUpSpill: Bimanual task involving uprighting a mug and wiping a spill (150 human demonstrations).
  • CarrotOnPlate: Evaluated using open-source Vision-Language-Action (VLA) models (Octo-Base and OpenVLA) within a toy kitchen environment based on the Bridge Data V2 dataset.
  • Simulated Benchmark (SimplerEnv): Evaluated Octo-Small vs. Octo-Base across three manipulation tasks (SpoonOnTowel, EggplantInBasket, StackCube).

Key Insights and Ideas

  • Finite-Sample Exactness: Prior sequential methods (like the Lai baseline) rely on the Central Limit Theorem (CLT), assuming empirical means are normally distributed. This assumption breaks down in robotics where sample sizes are small ($N \le 50$) or success rates are highly skewed (near 0 or 1). STEP solves the exact probability propagation, making it highly efficient in small-sample regimes.
  • Immunity to P-Hacking: STEP provides a "safety margin" for continuous testing. Evaluators can look at data sequentially and safely stop early or append more trials up to a maximum budget ($N_{max}$) without invalidating their statistical guarantees.
  • The Difficulty-Adaptive Property: STEP automatically adapts to the problem's difficulty. If one policy is vastly superior, the test terminates almost immediately. If they are closely matched, it safely utilizes the full budget or abstains from making an overconfident, incorrect decision.

Key Results

  • Trial Reduction: In numerical simulations, STEP reduced the required number of evaluation trials by up to 32% compared to state-of-the-art sequential baselines (like SAVI and Lai) while maintaining identical Type-I error control.
  • Multi-Task Efficiency: In the simulated SimplerEnv multi-task evaluation, STEP successfully certified that Octo-Small outperformed Octo-Base while saving the evaluator over 160 simulation rollouts compared to standard batch testing.
  • Hardware Validation: On the CleanUpSpill task, STEP correctly identified the superior policy configuration in just 8 to 13 trials, rendering the remaining 35+ planned trials completely unnecessary.

Limitations and Future Work

  • Binary Constraint: STEP relies heavily on the mathematical structure of Bernoulli distributions, meaning it currently only supports binary success/failure outcomes.
  • Risk Budget Tuning: The user must specify a risk accumulation rate $f(k)$. While a uniform budget yields excellent results, optimizing the shape of this budget function remains an open research question.
  • Future Extensions: The authors plan to extend STEP to handle multinoulli distributions (to accommodate partial-credit scoring schemas in long-horizon tasks) and more advanced multi-policy ranking setups.

Summary updated

Abstract

Imitation learning has enabled robots to perform complex, long-horizon tasks in challenging dexterous manipulation settings. As new methods are developed, they must be rigorously evaluated and compared against corresponding baselines through repeated evaluation trials. However, policy comparison is fundamentally constrained by a small feasible sample size (e.g., 10 or 50) due to significant human effort and limited inference throughput of policies. This paper proposes a novel statistical framework for rigorously comparing two policies in the small sample size regime. Prior work in statistical policy comparison relies on batch testing, which requires a fixed, pre-determined number of trials and lacks flexibility in adapting the sample size to the observed evaluation data. Furthermore, extending the test with additional trials risks inducing inadvertent p-hacking, undermining statistical assurances. In contrast, our proposed statistical test is sequential, allowing researchers to decide whether or not to run more trials based on intermediate results. This adaptively tailors the number of trials to the difficulty of the underlying comparison, saving significant time and effort without sacrificing probabilistic correctness. Extensive numerical simulation and real-world robot manipulation experiments show that our test achieves near-optimal stopping, letting researchers stop evaluation and make a decision in a near-minimal number of trials. Specifically, it reduces the number of evaluation trials by up to 32% as compared to state-of-the-art baselines, while preserving the probabilistic correctness and statistical power of the comparison. Moreover, our method is strongest in the most challenging comparison instances (requiring the most evaluation trials); in a multi-task comparison scenario, we save the evaluator more than 160 simulation rollouts.
BibTeX
@inproceedings{snyder2025is,
  author = {David Snyder and Asher James Hancock and Apurva Badithela and Emma Dixon and Patrick Miller and Rares Andrei Ambrus and Anirudha Majumdar and Masha Itkina and Haruki Nishimura},
  title = {{Is Your Imitation Learning Policy Better than Mine? Policy Comparison with Near-Optimal Stopping}},
  booktitle = {RobotEvaluation @ RSS},
  year = {2025},
  url = {https://openreview.net/forum?id=zCjkDbEzTW},
}
Figure