Getting By Goal Misgeneralization With a Little Help From a Mentor

Tu Trinh, Mohamad H. Danesh, Nguyen X. Khanh, Benjamin PlautarXiv 2024 · DOI · arXiv

Introduction

Reinforcement Learning (RL) agents often suffer from "goal misgeneralization," a phenomenon where an agent learns a proxy goal that correlates with the true reward during training but fails to generalize when deployed in a shifted environment. For instance, in a platformer game where a coin is always located at the far right, an agent might learn to "always run right" rather than "collect the coin." When the coin is moved to the middle of the level, the agent continues to run right, ignoring the reward entirely.

Figure 1: Examples of the CoinRun environment used to test goal misgeneralization. The agent must navigate platforms to collect the coin, which may appear in different locations in the test distribution.

This study investigates whether agents can overcome this failure by proactively requesting assistance from a mentor (an expert agent) when they encounter unfamiliar situations.

Methodology

The researchers evaluated a "Weak Agent" trained on a fixed-goal distribution and an "Expert Agent" trained on a randomized distribution to serve as a mentor. Several "Ask-For-Help" (AFH) strategies were tested:

  1. Action-Based Methods: Uncertainty is measured via policy outputs, such as high entropy or low maximum probability for the chosen action.
  2. Observation-Based Methods: Anomaly detection (Deep-SVDD) is applied to raw pixels or the agent's internal latent features to identify out-of-distribution states.
  3. Random Baseline: The agent requests help with a fixed probability at each step.
  4. Skyline Meta-Policies: An upper-bound reference where a policy learns the optimal switching points between the weak and expert agents.

The experiments were conducted using OpenAI’s CoinRun environment, specifically comparing standard training (coin at far right) against the procgenAISC test set (randomized coin locations).

Key Results and Representation Blindness

The primary finding is that while asking for help consistently improves performance, current RL agents suffer from a profound "representation blindness." Even when using sophisticated anomaly detection on internal features, agents often fail to recognize that a misplaced coin is an anomaly. This occurs because the training process (e.g., PPO) encourages the agent to ignore any features not strictly necessary for the reward in the training distribution. If "running right" always leads to the coin, the agent's internal representation may literally discard the visual information of the coin itself.

Figure 2: Performance vs. Ask-For-Help Percentage (AFHP). All help-seeking methods improve the average normalized reward compared to the weak agent baseline, though they remain below the expert's performance.

As shown in the performance analysis, action-based methods tend to be "reactive"---they only signal uncertainty once the agent has already missed the goal and reached a dead end. Surprisingly, the random baseline often outperformed these metrics because it was more likely to trigger help early in a level, allowing the expert to steer the agent toward the coin before it was bypassed.

Conclusion

The study highlights a significant gap between current uncertainty-based help-seeking and the theoretical "Skyline" performance. To build truly robust agents, future research must address "skinny" representations that ignore reward-irrelevant features. Potential solutions include training algorithms that encourage more comprehensive environment modeling and developing online learning mechanisms that allow agents to incorporate mentor feedback into their own policies permanently.

Summary updated

Abstract

While reinforcement learning (RL) agents often perform well during training, they can struggle with distribution shift in real-world deployments. One particularly severe risk of distribution shift is goal misgeneralization, where the agent learns a proxy goal that coincides with the true goal during training but not during deployment. In this paper, we explore whether allowing an agent to ask for help from a supervisor in unfamiliar situations can mitigate this issue. We focus on agents trained with PPO in the CoinRun environment, a setting known to exhibit goal misgeneralization. We evaluate multiple methods for determining when the agent should request help and find that asking for help consistently improves performance. However, we also find that methods based on the agent's internal state fail to proactively request help, instead waiting until mistakes have already occurred. Further investigation suggests that the agent's internal state does not represent the coin at all, highlighting the importance of learning nuanced representations, the risks of ignoring everything not immediately relevant to reward, and the necessity of developing ask-for-help strategies tailored to the agent's training algorithm.
BibTeX
@misc{trinh2024getting,
  author = {Tu Trinh and Mohamad H. Danesh and Nguyen X. Khanh and Benjamin Plaut},
  title = {{Getting By Goal Misgeneralization With a Little Help From a Mentor}},
  year = {2024},
  eprint = {2410.21052},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.21052},
}
Figure