On Assessing the Usefulness of Proxy Domains for Developing and Evaluating Embodied Agents

Anthony Courchesne, Andrea Censi, Liam Paull

Abstract

In many situations it is either impossible or impractical to develop and evaluate agents entirely on the target domain on which they will be deployed. This is particularly true in robotics, where doing experiments on hardware is much more arduous than in simulation. This has become arguably more so in the case of learning-based agents. To this end, considerable recent effort has been devoted to developing increasingly realistic and higher fidelity simulators. However, we lack any principled way to evaluate how good a "proxy domain" is, specifically in terms of how useful it is in helping us achieve our end objective of building an agent that performs well in the target domain. In this work, we investigate methods to address this need. We begin by clearly separating two uses of proxy domains that are often conflated: 1) their ability to be a faithful predictor of agent performance and 2) their ability to be a useful tool for learning. In this paper, we attempt to clarify the role of proxy domains and establish new proxy usefulness (PU) metrics to compare the usefulness of different proxy domains. We propose the relative predictive PU to assess the predictive ability of a proxy domain and the learning PU to quantify the usefulness of a proxy as a tool to generate learning data. Furthermore, we argue that the value of a proxy is conditioned on the task that it is being used to help solve. We demonstrate how these new metrics can be used to optimize parameters of the proxy domain for which obtaining ground truth via system identification is not trivial.

The Challenge of the Reality Gap

In robotics, training and evaluating agents directly on real hardware (the "target domain") is expensive, slow, and potentially dangerous. To circumvent these issues, researchers rely heavily on simulators (the "proxy domain"). However, because of the "reality gap," policies that work in simulation often fail in the real world.

The relationship between a proxy domain (simulation) and a target domain (real world) for agent development.

While much research focuses on crossing this gap (e.g., via domain randomization), there is surprisingly little work on explicitly quantifying the gap at the simulator level. Naively, one might assume a simulator is useful if it perfectly replicates real-world observations (pixel-for-pixel or physics-for-physics). This paper argues that visual or physical fidelity doesn't matter if it doesn't impact the specific task. Instead, we need principled, task-conditioned metrics to evaluate exactly how useful a proxy domain is for developing real-world agents.

Redefining Usefulness: Predictivity and Teachability

A proxy domain serves two distinct, often conflated purposes: acting as a Predictor (evaluating agents) and acting as a Teacher (training agents). This research proposes separate metrics for each:

  1. Proxy Relative Predictivity Value (PRPV) - The Proxy as a Predictor: Instead of measuring if a simulator perfectly predicts an agent's exact real-world score, PRPV measures if the simulator preserves the ranking (partial ordering) of different agents. If Agent A outperforms Agent B in simulation, a highly predictive proxy ensures Agent A will also outperform Agent B in the real world.
  2. Proxy Learning Value (PLV) - The Proxy as a Teacher: A simulator's value for learning is defined by how much it reduces the need for real-world data. PLV measures the difference in the number of real-world trials required to reach a specific performance threshold when an agent is trained entirely in the real world versus pre-trained in the simulator and fine-tuned in the real world.

Experimental Validation in Duckietown

The experiments were conducted using the Duckietown platform, featuring both physical robots and a customizable simulator. The evaluation used 10 agent submissions from the AI Driving Olympics (AIDO) lane-following challenge. Training data included an imitation learning dataset of 9,000 annotated real-world points, alongside simulated datasets generated with varying parameters.

Key Results and Insights

Fidelity $\neq$ Usefulness

A simulator does not need to be a perfect digital twin to be highly useful. It only needs to capture the dynamics and visuals that matter for the specific task. Furthermore, predictivity and teachability are orthogonal; a simulator might be poor at predicting performance but still serve as an excellent teacher, reducing real-world data needs by teaching fundamental features.

Optimizing the Simulator

PRPV can be used as an objective function to optimize simulator parameters (like wheel trim, camera blur, or latency). PRPV successfully identified true physical parameters of the real robot simply by observing which simulator settings best preserved the real-world rankings of the 10 AIDO agents.

PRPV values across various simulator parameters, demonstrating how the metric can identify optimal settings that match real-world dynamics.

Sample Efficiency Gains

Training an imitation learning agent entirely on the real robot required 9,000 samples. Pre-training on a highly accurate simulator reduced this to just 250 samples (a 97% reduction). Remarkably, even pre-training on heavily distorted simulators (e.g., incorrect camera angles) still reduced the real-world sample requirement significantly (88-92% reduction), proving that even "imperfect" simulators hold immense teaching value.

Conclusion

By shifting the focus from raw fidelity to task-conditioned usefulness, researchers can more effectively develop and select simulators. While calculating PLV is computationally expensive, these metrics provide a rigorous framework for bridging the reality gap in embodied AI development.

Summary updated

Abstract

In many situations it is either impossible or impractical to develop and evaluate agents entirely on the target domain on which they will be deployed. This is particularly true in robotics, where doing experiments on hardware is much more arduous than in simulation. This has become arguably more so in the case of learning-based agents. To this end, considerable recent effort has been devoted to developing increasingly realistic and higher fidelity simulators. However, we lack any principled way to evaluate how good a "proxy domain" is, specifically in terms of how useful it is in helping us achieve our end objective of building an agent that performs well in the target domain. In this work, we investigate methods to address this need. We begin by clearly separating two uses of proxy domains that are often conflated: 1) their ability to be a faithful predictor of agent performance and 2) their ability to be a useful tool for learning. In this paper, we attempt to clarify the role of proxy domains and establish new proxy usefulness (PU) metrics to compare the usefulness of different proxy domains. We propose the relative predictive PU to assess the predictive ability of a proxy domain and the learning PU to quantify the usefulness of a proxy as a tool to generate learning data. Furthermore, we argue that the value of a proxy is conditioned on the task that it is being used to help solve. We demonstrate how these new metrics can be used to optimize parameters of the proxy domain for which obtaining ground truth via system identification is not trivial.
BibTeX
@inproceedings{courchesne2021assessing,
  author = {Anthony Courchesne and Andrea Censi and Liam Paull},
  title = {{On Assessing the Usefulness of Proxy Domains for Developing and Evaluating Embodied Agents}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=nC9j0D7qAB},
}
Figure