Disentangling Recognition and Decision Regrets in Image-Based Reinforcement Learning

Alihan Hüyük, Arndt Ryo Koblitz, Atefeh Mohajeri, Matthew Andrews

Abstract

When image-based reinforcement learning (RL) agents fail to generalize to new environments, it is often unclear whether the failure stems from the representation learning (the "eyes") or the policy optimization (the "brain"). This paper introduces Recognition Regret ($R^{rec}$) and Decision Regret ($R^{dec}$), two metrics designed to disentangle these sources of error. By analyzing the gap between an agent's performance and the theoretical optimum, we can diagnose whether an agent is distracted by noise (over-specific) or relying on shortcuts (under-specific). We validate these metrics across grid-world and Atari environments, demonstrating their utility in identifying the root causes of observational overfitting.

Introduction

In image-based RL, agents process high-dimensional visual inputs through a two-stage pipeline: an encoder transforms pixels into low-dimensional features, and a policy head selects actions based on these features. A persistent challenge in this domain is observational overfitting, where an agent performs well during training but fails in test environments with minor visual variations, such as background color changes.

Historically, it has been difficult to determine if the encoder failed to extract relevant information or if the policy head failed to interpret that information correctly. This paper addresses this "credit assignment" problem for generalization failures by decomposing total regret into components attributable to each stage of the agent's architecture.

Methodology: Recognition and Decision Regret

The proposed framework splits the total Regret—the difference between an agent's performance and the optimal performance—into two distinct metrics:

  1. Recognition Regret ($R^{rec}$): This quantifies the limitation imposed by the encoder's representation. It is defined as the regret an optimal policy would still incur if forced to use the agent's current features. High $R^{rec}$ indicates the encoder is ignoring critical information or relying on spurious "shortcuts" that do not generalize.
  2. Decision Regret ($R^{dec}$): This measures the performance "wasted" by the policy head. It is the difference between the performance of an optimal policy using the agent's features and the agent's actual performance. High $R^{dec}$ often occurs when representations are "over-specific," containing irrelevant noise that confuses the policy.
The agent architecture consists of an encoder mapping observations to representations, followed by an output head that selects actions.

To evaluate generalization, the authors define Generalization Regret ($GR$). By re-training only the policy head in a test environment while keeping the encoder fixed, researchers can estimate how much of the failure is due to the "eyes" ($GR^{rec}$) versus the "brain" ($GR^{dec}$).

Experimental Validation

The metrics were evaluated using three environments: a toy 3-state MDP, Minigrid (a maze task), and Atari Pong. In Minigrid and Pong, the authors introduced controlled perturbations to validate the sensitivity of the metrics:

  • Random Actions: Increasing the probability of random actions should primarily increase $R^{dec}$.
  • Observation Masking: Masking critical parts of the visual input should primarily increase $R^{rec}$.

Key Results

The experiments confirmed that the metrics behave as expected. As shown in the validation plots, $R^{dec}$ tracks closely with random action probability, while $R^{rec}$ tracks with observation masking.

Validation of regret metrics in Minigrid and Pong. $R^{dec}$ (green) increases with random action probability, while $R^{rec}$ (orange) increases with observation masking.

In Minigrid, an "Identity" agent (which sees everything) exhibited high Decision Regret ($GR^{dec} = 0.647$) due to its inability to ignore irrelevant colors. Conversely, a "HideDoor" agent (forced to rely on color shortcuts) showed high Recognition Regret ($GR^{rec} = 0.607$) because its encoder failed to capture the door's location. Similar patterns were observed in Pong, where agents relying on a "frame counter" shortcut failed due to high $R^{rec}$ when the game's starting state was randomized.

Discussion and Taxonomy of Overfitting

The paper introduces a useful taxonomy for diagnosing RL agents:

  • Over-specific agents see too much detail. They encode irrelevant features (e.g., background textures), leading to high Decision Regret as the policy head struggles to generalize across noisy inputs.
  • Under-specific agents see too little. They rely on "cheats" (e.g., timestamps or specific colors) and ignore core mechanics, leading to high Recognition Regret when those cheats are removed.

The authors also demonstrate that representation size matters; reducing latent channels can lower $R^{dec}$ by filtering noise but may inadvertently raise $R^{rec}$ by forcing the agent toward shortcuts.

Conclusion

This framework provides a principled way to debug image-based RL agents. While estimating these regrets can be computationally intensive—requiring policy optimization in test environments—the insights gained allow for more targeted improvements. Future work may involve using these metrics to design new loss functions that encourage robust, generalizable representations directly during training.

Summary updated

BibTeX
@inproceedings{huyuk2025disentangling,
  author = {Alihan Hüyük and Arndt Ryo Koblitz and Atefeh Mohajeri and Matthew Andrews},
  title = {{Disentangling Recognition and Decision Regrets in Image-Based Reinforcement Learning}},
  booktitle = {RLC},
  year = {2025},
  url = {https://openreview.net/forum?id=cGwaIEMkdt},
}
Figure