TL;DR
Evaluating contact-rich robotic manipulation at scale has traditionally been the playground of well-funded labs. This paper introduces a reproducible, serverless evaluation harness that runs 200-rollout physics simulations for just $3.50, revealing a critical "negative result": the popular technique of action chunking can completely collapse policy performance when training data is sparse.
What Problem Does This Paper Address?
In robotics, simulation is essential for safety and scale, but "high-fidelity" often translates to "high-cost." Evaluating how a neural network policy performs in a closed-loop physics environment requires significant GPU compute and complex infrastructure (networking, IAM permissions, container orchestration). This creates a "compute divide" where only top-tier labs can afford to run the thousands of rollouts needed for statistical significance.
Beyond cost, the authors highlight the "reproducibility crisis" in robotics. Even with open-source code, undocumented environment requirements—like specific EGL driver configurations or hardcoded file paths—often prevent external researchers from verifying results. This paper aims to democratize evaluation by providing a turn-key, affordable wrapper for existing benchmarks.
Methods
The authors built an evaluation harness wrapper around the lbm_eval benchmark, which uses the Drake physics engine.
The Infrastructure: To keep costs low and throughput high, they utilized a serverless "fan-out" architecture on Modal. Instead of one giant machine, the workload is split into eight parallel shards (containers), each running a subset of the 200 rollouts. Each container runs a headless rendering stack (Xvfb + EGL/OpenGL) and communicates with a standalone policy server via gRPC.
The Policies: They tested two main types of decoders:
- Behavior Cloning (BC): A standard regressor using Mean Squared Error.
- Diffusion Policy (DDPM): A generative approach that has recently set state-of-the-art records in manipulation.
They specifically experimented with action chunking—predicting a sequence of future actions ($H$) and executing a subset of them ($K$) before replanning—versus single-step prediction ($H=1, K=1$).
Datasets
The study focused on the pick_and_place_box task from the TRI (Toyota Research Institute) lbm_eval repository. This task requires a Franka FR3 robot arm to pick up a randomly placed box and move it to a goal region within 15 seconds.
The models were trained on a "sparse" dataset of only 99 usable episodes (approx. 4,200 keyframes). This low-data regime is common for researchers who don't have massive robot farms to collect thousands of demonstrations.
Key Insights and Ideas
- The "Trivial Success" Trap: The authors found that 2.5% of their "successes" were actually artifacts where the box started at the goal. They established a strict reporting protocol (Rule R1) to exclude any completion under one second, preventing "phantom" performance gains.
- Validation MSE is a Liar: In the sparse data regime, a lower validation error (MSE) did not correlate with better real-world performance. Two models with nearly identical MSEs had success rates of 2.6% vs. 11.8%.
- Chunking Collapse: While action chunking is a standard "best practice" in data-rich environments, it proved disastrous here. Every single chunked checkpoint produced zero genuine successes across 780 rollouts.
- Infrastructure as a Bottleneck: The authors noted that compute cost wasn't the primary hurdle; rather, it was "undocumented environment requirements" like EGL vendor files and interpreter path assumptions that consumed the most engineering time.
Key Results
The quantitative results highlight both the efficiency of the harness and the fragility of the models:
- Cost Efficiency: A full 200-rollout evaluation costs **$3.49 USD** and takes about **38 minutes** of wall-clock time. The entire study, including training and multiple replications, cost roughly $34.
- Single-Step Success: The best single-step BC model achieved an 11.8% success rate.
- Chunking Failure: All chunked models (both BC and Diffusion) scored 0% success. Even when the authors forced the chunked model to replan every single step (setting $K=1$), it still failed, suggesting the issue lies in the training formulation itself rather than the open-loop execution of the chunk.
Limitations and Future Work
The authors are careful to note that their negative result on action chunking is specific to this "sparse data regime" (approx. 100 demonstrations) and might not generalize to larger datasets.
Future work includes:
- Expanding the harness to support multi-task benchmarks.
- Integrating compliant contact simulation to generate synthetic training data, which might help bridge the gap to higher success rates.
- Testing whether different model scales or architectures can mitigate the chunking collapse in low-data settings.
By releasing their harness on GitHub, the authors hope to enable researchers outside of major labs to iterate rapidly on physics-based policies without breaking the bank.
Summary updated