TL;DR
When image-based reinforcement learning (RL) agents fail to generalize to new visual environments, it is often unclear whether the failure stems from flawed feature extraction (the visual encoder) or poor action selection (the policy head). This paper introduces recognition regret and decision regret to untangle these two error sources, formally characterizing observational overfitting into two distinct failure modes: under-specific representations (relying on spurious shortcuts) and over-specific representations (retaining task-irrelevant visual clutter).
What Problem Does This Paper Address?
Modern image-based RL architectures typically split an agent into two stages:
- A recognition policy (encoder): compresses high-dimensional pixel inputs into lower-dimensional latent states.
- A decision policy (output head): selects actions based on those latent states.
When an agent fails on unseen test environments—a problem known as observational overfitting—standard metrics like generalization error only provide a single aggregate performance drop. They cannot explain why the agent failed. Did the encoder throw away critical state details and latch onto spurious shortcuts that only worked during training? Or did the encoder faithfully capture the true state along with irrelevant cosmetic variations, causing the decision head to treat familiar states as completely novel?
Because neural network encoders and policy heads are trained jointly end-to-end, diagnosing which module caused the failure has historically been elusive.
Methods
The authors decompose standard performance regret into two complementary terms:
$$\text{Regret} = R^{\text{rec}} + R^{\text{dec}}$$
- Recognition Regret ($R^{\text{rec}}$): The gap between the environment's optimal value and the best performance achievable by any decision policy paired with the given encoder $\rho_0$. It measures how much the encoder acts as an information bottleneck.
- Decision Regret ($R^{\text{dec}}$): The gap between the best performance possible under encoder $\rho_0$ and the performance actually attained by the agent's specific decision policy $\pi_0$.
Measuring Generalization Regret ($\widehat{GR}$)
To diagnose test-environment failures without knowing theoretical optimal bounds, the authors introduce empirical test-time decompositions:
- They evaluate the trained agent $(\rho_{\text{train}}, \pi_{\text{train}})$ on the test environment.
- They then freeze the trained encoder $\rho_{\text{train}}$ and train a fresh decision policy directly in the test environment to convergence ($\pi_{\text{test}}^*$).
- $\widehat{GR}^{\text{dec}}$ is the performance gained simply by retraining the decision policy on the frozen encoder.
- $\widehat{GR}^{\text{rec}}$ is the remaining performance gap that can only be resolved by retraining the encoder itself.
From this decomposition, the authors define two distinct modes of observational overfitting:
- Under-specific representations (High $\widehat{GR}^{\text{rec}}$, Low $\widehat{GR}^{\text{dec}}$): The encoder captured "too few" features—often spurious correlations. Retraining the decision head in the test environment cannot recover performance because the necessary state information was discarded by the encoder.
- Over-specific representations (Low $\widehat{GR}^{\text{rec}}$, High $\widehat{GR}^{\text{dec}}$): The encoder captured "too many" features—including task-irrelevant noise. Because the true state information is still present in the latents, retraining the decision head immediately restores optimal performance.
Datasets and Environments
The authors evaluate these metrics on two vision-based benchmarks with synthetic modifications to isolate spurious and irrelevant cues:
- Minigrid Mazes: A 3×3 grid world where an agent must grab a key and unlock a door located at either the North, Center, or South position.
- Spurious correlation: In training, door position strictly correlates with key/door color (North = Red, Center = Green, South = Blue).
- Test distribution: All doors and keys are Gray (an unseen color), breaking the shortcut.
- Deterministic Atari Pong (
ALE/Pong-v5): - Spurious feature: A frame counter (progress bar) along the screen header, which is completely predictive of the ball trajectory during training, but rendered obsolete at test time by introducing random initial null actions.
- Irrelevant features (distractions): Synthetic visual patterns generated by passing true game coordinates through randomly initialized transposed convolutions and concatenating them onto the observation.
Key Insights and Ideas
- Over-specificity is an encoder underfitting problem; under-specificity is an encoder overfitting problem. In over-specificity, the encoder fails to compress away observation-level noise. In under-specificity, the encoder overfits to training shortcuts and fails to capture the underlying environment state.
- Diagnostic clarity for architecture tuning: Simply reducing latent channel capacity to prevent over-specificity can backfire. In Minigrid, reducing the CNN latent channels successfully reduced decision regret ($\widehat{GR}^{\text{dec}}$), but once the bottleneck became too narrow ($< 16$ channels), the encoder abandoned structural spatial cues and relied entirely on color shortcuts, causing recognition regret ($\widehat{GR}^{\text{rec}}$) to skyrocket.
Key Results
- Minigrid Mazes (Normalized Regret $\in [0, 1]$):
- Identity (Unfiltered CNN): Incurred near-zero training regret ($R = 0.002$) but suffered severe generalization regret ($\widehat{GR} = 0.911$). This was dominated by decision regret ($\widehat{GR}^{\text{dec}} = 0.647$) over recognition regret ($\widehat{GR}^{\text{rec}} = 0.264$), identifying over-specificity due to color sensitivity.
- HideDoor (Door obscured, forcing reliance on color): Generalization regret ($\widehat{GR} = 0.837$) was dominated by recognition regret ($\widehat{GR}^{\text{rec}} = 0.607$), verifying under-specificity.
- Pong Generalization:
- NoCounter Agent (exposed to visual distractions): Failed on new distractions ($\widehat{GR} = 0.642$) almost entirely via decision regret ($\widehat{GR}^{\text{dec}} = 0.641$ vs. $\widehat{GR}^{\text{rec}} = 0.001$), cleanly exposing over-specificity.
- JustCounter Agent (relied on the frame counter): Failed on randomized start states ($\widehat{GR} = 0.954$) with high recognition regret ($\widehat{GR}^{\text{rec}} = 0.737$ vs. $\widehat{GR}^{\text{dec}} = 0.217$), demonstrating under-specificity.
Limitations and Future Work
- Computational Cost of Head Retraining: Estimating $\widehat{GR}^{\text{rec}}$ and $\widehat{GR}^{\text{dec}}$ requires optimizing an auxiliary decision head directly in the target test environment. While effective for diagnostic validation on evaluation splits, it cannot be run in strict zero-shot deployment settings without test-time reward signals.
- Synthetic vs. Complex Natural Clutter: Experiments relied on controlled, semi-synthetic visual distractions (frame counters, explicit color shifts). Validating these behaviors on photorealistic, high-dimensional simulators (e.g., Procgen, CARLA) remains open.
- Automated Remediation: The framework diagnoses the nature of overfitting, but automating the response—such as dynamically applying observation-invariance regularizers for high decision regret versus representation-learning auxiliary losses for high recognition regret—is left for future work.
Summary updated