TL;DR
Simulators (proxy domains) are essential for training and evaluating embodied agents like robots, but quantifying their actual usefulness is notoriously difficult. This paper introduces two novel metrics—Proxy Relative Predictivity Value (PRPV) and Proxy Learning Value (PLV)—to independently measure a simulator's ability to accurately rank agent performance and its ability to reduce real-world training data requirements.
What Problem Does This Paper Address?
In robotics, training and evaluating agents directly on real hardware (the "target domain") is expensive, slow, and potentially dangerous. To get around this, researchers rely heavily on simulators (the "proxy domain"). However, because of the "reality gap," policies that work in simulation often fail in the real world.
While much research focuses on crossing this gap (e.g., via domain randomization), there is surprisingly little work on explicitly quantifying the gap at the simulator level. Naively, one might assume a simulator is useful if it perfectly replicates real-world observations (pixel-for-pixel or physics-for-physics). The authors argue this is flawed: visual or physical fidelity doesn't matter if it doesn't impact the specific task. The paper addresses the need for principled, task-conditioned metrics to evaluate exactly how useful a proxy domain is for developing real-world agents.
Methods
The authors argue that a proxy domain serves two distinct, often conflated purposes: acting as a Predictor (evaluating agents) and acting as a Teacher (training agents). They propose separate metrics for each:
-
Proxy Relative Predictivity Value (PRPV) - The Proxy as a Predictor: Instead of measuring if a simulator perfectly predicts an agent's exact real-world score, PRPV measures if the simulator preserves the ranking (partial ordering) of different agents. If Agent A outperforms Agent B in simulation, a highly predictive proxy ensures Agent A will also outperform Agent B in the real world. It is calculated using the 1-norm distance between matrices representing the partial orders of agents in both domains.
-
Proxy Learning Value (PLV) - The Proxy as a Teacher: A simulator's value for learning is defined by how much it reduces the need for real-world data. PLV measures the difference in the number of real-world trials (or data samples) required to reach a specific performance threshold when an agent is trained entirely in the real world versus pre-trained in the simulator and fine-tuned in the real world.
The authors also demonstrate how these metrics can be used as objective functions to optimize simulator parameters (like camera angle, latency, or blur) to make the proxy more useful.
Datasets
The experiments are conducted using the Duckietown platform, which features both a physical robotics environment and a highly customizable simulator.
- Evaluation Data: 10 agent submissions from the AI Driving Olympics (AIDO) lane-following challenge.
- Training Data: An imitation learning dataset consisting of 9,000 annotated real-world data points, alongside simulated datasets generated with varying camera pitch angles.
Key Insights and Ideas
- Fidelity $\neq$ Usefulness: A simulator does not need to be a perfect digital twin to be highly useful. It only needs to capture the dynamics and visuals that actually matter for the specific task the agent is trying to solve.
- Predictivity and Teachability are Orthogonal: A simulator might be terrible at predicting how well an agent will perform in the real world (e.g., because the simulated camera angle is entirely wrong), but it can still be an excellent teacher that teaches the agent fundamental features, drastically reducing the real-world data needed to adapt.
- Rankings Matter More Than Raw Scores: When evaluating agents, absolute performance scores in simulation rarely match reality. By focusing on rank inversion (PRPV), researchers can confidently use simulators for model selection.
Key Results
- Optimizing the Simulator (PRPV): The authors used PRPV to evaluate different simulator parameters (wheel trim, camera blur, latency). PRPV successfully identified the true physical parameters of the real robot (e.g., a wheel trim of 0.0) simply by observing which simulator settings best preserved the real-world rankings of the 10 AIDO agents. Naive metrics (like raw observation discrepancy) failed to do this clearly.
- Massive Sample Efficiency Gains (PLV): Training an imitation learning agent entirely on the real robot required 9,000 samples to reach convergence.
- Pre-training the agent on a highly accurate simulator (19° camera angle) reduced the real-world sample requirement to just 250 (a 97% reduction).
- Surprisingly, pre-training on a heavily distorted simulator (e.g., a 3° or 35° camera angle) still reduced the real-world sample requirement to 1,000 and 750 samples, respectively (an 88-92% reduction). This proves that even "bad" simulators hold immense teaching value.
Limitations and Future Work
The primary limitation acknowledged by the authors is the computational cost of evaluating the Proxy Learning Value (PLV). Because PLV requires fully training an agent in the proxy and then fine-tuning it in the target domain to measure sample reduction, using it as a loss function to optimize simulator parameters is highly expensive.
Future work will focus on more rigorous optimization techniques to find optimal simulator parameters efficiently. The authors suggest that leveraging differentiable simulation environments or using Bayesian Optimization could make this search much more tractable.
Summary updated