TL;DR
Researchers have long been puzzled by "mysteries" in real-world behavioral cloning (BC), such as why overfitting often helps performance or why closed-loop control frequently fails. This paper introduces OCBench, a GPU-accelerated benchmark that uses scripted policies designed to mimic the structural properties of human data, finally allowing these counterintuitive phenomena to be studied scientifically in a controlled, simulated environment.
What Problem Does This Paper Address?
Behavioral cloning—training a robot to mimic a demonstrator—is deceptively simple in theory but behaves strangely in practice. When trained on large-scale human data, BC exhibits several "mysteries":
- Beneficial Overfitting: Performance often improves even as the model overfits and validation loss increases.
- Closed-Loop Failure: Policies that react to every frame often fail, while "open-loop" action chunking (predicting a sequence of future actions) works much better.
- Massive Model Requirements: Real-world tasks seem to require models with hundreds of millions of parameters, far exceeding what typical simulations suggest.
Until now, studying these issues was nearly impossible. Real-world experiments are too expensive and non-reproducible, while existing simulations use "too perfect" scripted policies that don't trigger these failure modes. The authors bridge this gap by creating a benchmark that reproduces these real-world headaches in simulation.
Methods
The core innovation lies in how OCBench generates its data. Instead of using standard rule-based or RL-trained scripts, the authors developed scripted policies using randomized piecewise $C^1$-continuous Hermite splines.
This approach mimics three key properties of human demonstrations:
- Narrow Distribution: Unlike Gaussian-noise-augmented trajectories, these are smooth and purposeful.
- Non-Markovianity: The trajectories are highly temporally correlated.
- Semantic Diversity: The scripts randomize grasping angles, motion speeds, and high-level strategies.
To make scientific study feasible, the entire suite is implemented in MJWarp, a GPU-accelerated version of MuJoCo. This allows for "streaming-data training," where an agent can be trained on a continuous, effectively unlimited flow of new demonstrations at up to 200,000 environment steps per second.
Datasets
The paper introduces OCBench, which consists of 28 tasks across five environment types:
- Block: Standard manipulation (picking, placing, stacking).
- Chamber: Complex multi-object interactions involving drawers, windows, and buttons.
- Switch: A "Lights Out" puzzle requiring combinatorial reasoning.
- Hanoi: High-precision manipulation based on the Tower of Hanoi.
- Bowling: A task requiring an understanding of complex object dynamics.
The authors also provide "lite" versions of these tasks (10Hz control vs. 50Hz) to enable faster iteration for researchers with fewer compute resources.
Key Insights and Ideas
-
Human-like Statistics Matter: Previous benchmarks produced action distributions that were either overly uniform or saturated at the limits (-1 or +1). OCBench produces action statistics that closely mirror human data. OCBench’s scripted policy has more human-like action statistics than previous scripted policies.
-
The "Size-Reversal" Phenomenon: In a truly bizarre finding, the authors reproduced a real-world observation where a smaller dataset can sometimes lead to better performance than a larger one, even when both are sampled from the same distribution.
-
Validation Loss is a Liar: The paper confirms that in BC, validation flow loss is a poor predictor of success. Performance often continues to climb long after validation loss has bottomed out and started to explode. Performance can increase as the agent overfits more.
-
Causal Confusion in History: While adding history (past states) to a policy should provide more information, it often hurts performance in BC. The authors suggest this is due to "causal confusion," where the model learns to simply repeat the last action rather than understanding the task.
Key Results
- Reproduction of Real-World Failures: OCBench successfully reproduced the failure of closed-loop control. While open-loop policies achieved high success rates, standard closed-loop policies almost completely failed on tasks like
block-single. - Scaling Laws: The authors showed that OCBench tasks benefit from massive models. Performance continued to improve up to 540-million parameter residual MLPs, a scaling property rarely seen in other state-based simulations but common in real-world robotics.
- Feature Sensitivity: Even with "unlimited" data, the scale of input features (e.g., scaling XYZ coordinates by 10x) drastically changed performance, despite the information content remaining identical. This highlights that test-time generalization is heavily influenced by feature emphasis. The size-reversal phenomenon where larger datasets can hurt performance.
Limitations and Future Work
While OCBench is a significant step forward, the authors acknowledge that their scripted policies are still just scripts—they do not perfectly capture every nuance of human behavior. They suggest that while simulation is a great "playground" for rapid testing, final validation must still happen on real hardware.
Future work includes using OCBench to study Reinforcement Learning (RL), specifically how different demonstration speeds and subtask labels can help train more robust hierarchical controllers.
Summary updated