On Assessing the Usefulness of Proxy Domains for Developing and Evaluating Embodied Agents

Anthony Courchesne, Andrea Censi, Liam Paull

TL;DR

Simulators (proxy domains) are essential for training and evaluating embodied agents like robots, but quantifying their actual usefulness is notoriously difficult. This paper introduces two novel metrics—Proxy Relative Predictivity Value (PRPV) and Proxy Learning Value (PLV)—to independently measure a simulator's ability to accurately rank agent performance and its ability to reduce real-world training data requirements.

What Problem Does This Paper Address?

In robotics, training and evaluating agents directly on real hardware (the "target domain") is expensive, slow, and potentially dangerous. To get around this, researchers rely heavily on simulators (the "proxy domain"). However, because of the "reality gap," policies that work in simulation often fail in the real world.

While much research focuses on crossing this gap (e.g., via domain randomization), there is surprisingly little work on explicitly quantifying the gap at the simulator level. Naively, one might assume a simulator is useful if it perfectly replicates real-world observations (pixel-for-pixel or physics-for-physics). The authors argue this is flawed: visual or physical fidelity doesn't matter if it doesn't impact the specific task. The paper addresses the need for principled, task-conditioned metrics to evaluate exactly how useful a proxy domain is for developing real-world agents.

Methods

The authors argue that a proxy domain serves two distinct, often conflated purposes: acting as a Predictor (evaluating agents) and acting as a Teacher (training agents). They propose separate metrics for each:

  1. Proxy Relative Predictivity Value (PRPV) - The Proxy as a Predictor: Instead of measuring if a simulator perfectly predicts an agent's exact real-world score, PRPV measures if the simulator preserves the ranking (partial ordering) of different agents. If Agent A outperforms Agent B in simulation, a highly predictive proxy ensures Agent A will also outperform Agent B in the real world. It is calculated using the 1-norm distance between matrices representing the partial orders of agents in both domains.

  2. Proxy Learning Value (PLV) - The Proxy as a Teacher: A simulator's value for learning is defined by how much it reduces the need for real-world data. PLV measures the difference in the number of real-world trials (or data samples) required to reach a specific performance threshold when an agent is trained entirely in the real world versus pre-trained in the simulator and fine-tuned in the real world.

The authors also demonstrate how these metrics can be used as objective functions to optimize simulator parameters (like camera angle, latency, or blur) to make the proxy more useful.

Datasets

The experiments are conducted using the Duckietown platform, which features both a physical robotics environment and a highly customizable simulator.

  • Evaluation Data: 10 agent submissions from the AI Driving Olympics (AIDO) lane-following challenge.
  • Training Data: An imitation learning dataset consisting of 9,000 annotated real-world data points, alongside simulated datasets generated with varying camera pitch angles.

Key Insights and Ideas

  • Fidelity $\neq$ Usefulness: A simulator does not need to be a perfect digital twin to be highly useful. It only needs to capture the dynamics and visuals that actually matter for the specific task the agent is trying to solve.
  • Predictivity and Teachability are Orthogonal: A simulator might be terrible at predicting how well an agent will perform in the real world (e.g., because the simulated camera angle is entirely wrong), but it can still be an excellent teacher that teaches the agent fundamental features, drastically reducing the real-world data needed to adapt.
  • Rankings Matter More Than Raw Scores: When evaluating agents, absolute performance scores in simulation rarely match reality. By focusing on rank inversion (PRPV), researchers can confidently use simulators for model selection.

Key Results

  • Optimizing the Simulator (PRPV): The authors used PRPV to evaluate different simulator parameters (wheel trim, camera blur, latency). PRPV successfully identified the true physical parameters of the real robot (e.g., a wheel trim of 0.0) simply by observing which simulator settings best preserved the real-world rankings of the 10 AIDO agents. Naive metrics (like raw observation discrepancy) failed to do this clearly.
  • Massive Sample Efficiency Gains (PLV): Training an imitation learning agent entirely on the real robot required 9,000 samples to reach convergence.
  • Pre-training the agent on a highly accurate simulator (19° camera angle) reduced the real-world sample requirement to just 250 (a 97% reduction).
  • Surprisingly, pre-training on a heavily distorted simulator (e.g., a 3° or 35° camera angle) still reduced the real-world sample requirement to 1,000 and 750 samples, respectively (an 88-92% reduction). This proves that even "bad" simulators hold immense teaching value.

Limitations and Future Work

The primary limitation acknowledged by the authors is the computational cost of evaluating the Proxy Learning Value (PLV). Because PLV requires fully training an agent in the proxy and then fine-tuning it in the target domain to measure sample reduction, using it as a loss function to optimize simulator parameters is highly expensive.

Future work will focus on more rigorous optimization techniques to find optimal simulator parameters efficiently. The authors suggest that leveraging differentiable simulation environments or using Bayesian Optimization could make this search much more tractable.

Summary updated

Abstract

In many situations it is either impossible or impractical to develop and evaluate agents entirely on the target domain on which they will be deployed. This is particularly true in robotics, where doing experiments on hardware is much more arduous than in simulation. This has become arguably more so in the case of learning-based agents. To this end, considerable recent effort has been devoted to developing increasingly realistic and higher fidelity simulators. However, we lack any principled way to evaluate how good a "proxy domain" is, specifically in terms of how useful it is in helping us achieve our end objective of building an agent that performs well in the target domain. In this work, we investigate methods to address this need. We begin by clearly separating two uses of proxy domains that are often conflated: 1) their ability to be a faithful predictor of agent performance and 2) their ability to be a useful tool for learning. In this paper, we attempt to clarify the role of proxy domains and establish new proxy usefulness (PU) metrics to compare the usefulness of different proxy domains. We propose the relative predictive PU to assess the predictive ability of a proxy domain and the learning PU to quantify the usefulness of a proxy as a tool to generate learning data. Furthermore, we argue that the value of a proxy is conditioned on the task that it is being used to help solve. We demonstrate how these new metrics can be used to optimize parameters of the proxy domain for which obtaining ground truth via system identification is not trivial.
BibTeX
@inproceedings{courchesne2021assessing,
  author = {Anthony Courchesne and Andrea Censi and Liam Paull},
  title = {{On Assessing the Usefulness of Proxy Domains for Developing and Evaluating Embodied Agents}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=nC9j0D7qAB},
}

Loading related content...

Figure