Closed-Loop Physics-Based Manipulation Evaluation at a Commodity Price: What 4.4 L4 GPU hours at $3.50 Buys You

Rohun Kshirsagar, Diana RypkemaarXiv 2026 · arXiv

TL;DR

Evaluating contact-rich robotic manipulation at scale has traditionally been the playground of well-funded labs. This paper introduces a reproducible, serverless evaluation harness that runs 200-rollout physics simulations for just $3.50, revealing a critical "negative result": the popular technique of action chunking can completely collapse policy performance when training data is sparse.

What Problem Does This Paper Address?

In robotics, simulation is essential for safety and scale, but "high-fidelity" often translates to "high-cost." Evaluating how a neural network policy performs in a closed-loop physics environment requires significant GPU compute and complex infrastructure (networking, IAM permissions, container orchestration). This creates a "compute divide" where only top-tier labs can afford to run the thousands of rollouts needed for statistical significance.

Beyond cost, the authors highlight the "reproducibility crisis" in robotics. Even with open-source code, undocumented environment requirements—like specific EGL driver configurations or hardcoded file paths—often prevent external researchers from verifying results. This paper aims to democratize evaluation by providing a turn-key, affordable wrapper for existing benchmarks.

Methods

The authors built an evaluation harness wrapper around the lbm_eval benchmark, which uses the Drake physics engine.

The Infrastructure: To keep costs low and throughput high, they utilized a serverless "fan-out" architecture on Modal. Instead of one giant machine, the workload is split into eight parallel shards (containers), each running a subset of the 200 rollouts. Each container runs a headless rendering stack (Xvfb + EGL/OpenGL) and communicates with a standalone policy server via gRPC.

The runtime architecture uses a cloud-resident dispatcher to fan out evaluations across eight parallel GPU containers.

The Policies: They tested two main types of decoders:

  1. Behavior Cloning (BC): A standard regressor using Mean Squared Error.
  2. Diffusion Policy (DDPM): A generative approach that has recently set state-of-the-art records in manipulation.

They specifically experimented with action chunking—predicting a sequence of future actions ($H$) and executing a subset of them ($K$) before replanning—versus single-step prediction ($H=1, K=1$).

Datasets

The study focused on the pick_and_place_box task from the TRI (Toyota Research Institute) lbm_eval repository. This task requires a Franka FR3 robot arm to pick up a randomly placed box and move it to a goal region within 15 seconds.

The models were trained on a "sparse" dataset of only 99 usable episodes (approx. 4,200 keyframes). This low-data regime is common for researchers who don't have massive robot farms to collect thousands of demonstrations.

Key Insights and Ideas

  • The "Trivial Success" Trap: The authors found that 2.5% of their "successes" were actually artifacts where the box started at the goal. They established a strict reporting protocol (Rule R1) to exclude any completion under one second, preventing "phantom" performance gains.
  • Validation MSE is a Liar: In the sparse data regime, a lower validation error (MSE) did not correlate with better real-world performance. Two models with nearly identical MSEs had success rates of 2.6% vs. 11.8%.
  • Chunking Collapse: While action chunking is a standard "best practice" in data-rich environments, it proved disastrous here. Every single chunked checkpoint produced zero genuine successes across 780 rollouts.
  • Infrastructure as a Bottleneck: The authors noted that compute cost wasn't the primary hurdle; rather, it was "undocumented environment requirements" like EGL vendor files and interpreter path assumptions that consumed the most engineering time.

Key Results

The quantitative results highlight both the efficiency of the harness and the fragility of the models:

  • Cost Efficiency: A full 200-rollout evaluation costs **$3.49 USD** and takes about **38 minutes** of wall-clock time. The entire study, including training and multiple replications, cost roughly $34.
  • Single-Step Success: The best single-step BC model achieved an 11.8% success rate.
  • Chunking Failure: All chunked models (both BC and Diffusion) scored 0% success. Even when the authors forced the chunked model to replan every single step (setting $K=1$), it still failed, suggesting the issue lies in the training formulation itself rather than the open-loop execution of the chunk.
Visual verification of rollouts: Scene 125 shows a success, while Scene 3 shows a failure where the arm overshoots. Scene 6 is a "trivial success" excluded by the R1 rule.

Limitations and Future Work

The authors are careful to note that their negative result on action chunking is specific to this "sparse data regime" (approx. 100 demonstrations) and might not generalize to larger datasets.

Future work includes:

  • Expanding the harness to support multi-task benchmarks.
  • Integrating compliant contact simulation to generate synthetic training data, which might help bridge the gap to higher success rates.
  • Testing whether different model scales or architectures can mitigate the chunking collapse in low-data settings.

By releasing their harness on GitHub, the authors hope to enable researchers outside of major labs to iterate rapidly on physics-based policies without breaking the bank.

Summary updated

Abstract

Running evaluations of contact-rich robotic manipulation at statistical scale has largely been possible only at well-resourced labs, highlighting the need for affordable and efficient options for broader audiences. We present a reproducible evaluation harness wrapper based on a previous study at 200-rollout scale on serverless commodity GPUs, at a cost of approximately $3.50 USD and 38 minutes of wall-clock per replication (4.4 L4-GPU-hours across eight parallel shards). Using the harness, we report a partially controlled negative result on action chunking with sparse data: three independently trained chunked checkpoints produce zero genuine successes across 780 rollouts, while three single-step checkpoints range from 2.6% to 11.8% success rate. A per-step-replanning control partially isolates chunk prediction, rather than open-loop chunk execution as the mechanism, with the failure localized to the training formulation and multi-step prediction. This work further democratizes physics-based policy development and evaluation through our reusable harness.
BibTeX
@misc{kshirsagar2026closed,
  author = {Rohun Kshirsagar and Diana Rypkema},
  title = {{Closed-Loop Physics-Based Manipulation Evaluation at a Commodity Price: What 4.4 L4 GPU hours at $3.50 Buys You}},
  year = {2026},
  eprint = {2611.07441},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07441},
}
Figure