Abstract
Learning to control an agent from data collected offline in a rich pixel-based visual observation space is vital for real-world applications of reinforcement learning (RL). A major challenge in this setting is the presence of input information that is hard to model and irrelevant to controlling the agent. This problem has been approached by the theoretical RL community through the lens of exogenous information, i.e., any control-irrelevant information contained in observations. For example, a robot navigating in busy streets needs to ignore irrelevant information, such as other people walking in the background, textures of objects, or birds in the sky. In this paper, we focus on the setting with visually detailed exogenous information and introduce new offline RL benchmarks offering the ability to study this problem. We find that contemporary representation learning techniques can fail on datasets where the noise is a complex and time-dependent process, which is prevalent in practical applications. To address these, we propose to use multi-step inverse models, which have seen a great deal of interest in the RL theory community, to learn Agent-Controller Representations for Offline-RL (ACRO). Despite being simple and requiring no reward, we show theoretically and empirically that the representation created by this objective greatly outperforms baselines.
The Challenge of Exogenous Information
In real-world robotics or autonomous driving, visual data is inherently messy. An agent's camera captures both endogenous information—elements the agent can control or that directly affect its rewards, such as its own robotic arm—and exogenous information, which is irrelevant "noise" like moving clouds or background traffic.
In Offline RL, where learning occurs from a fixed dataset without further exploration, this noise presents a significant hurdle. Most representation learning techniques attempt to model the entire observation frame. If the background noise is complex or temporally correlated (e.g., logically moving objects), the agent may exhaust its representational capacity modeling these irrelevant dynamics instead of the controllable state. This distraction isn't just inefficient; the authors demonstrate that including exogenous noise can violate Bellman completeness, causing RL algorithms to diverge because the value functions cannot be accurately represented within the noise-polluted latent space.
ACRO: Agent-Centric Representations for Offline-RL
To address these challenges, we introduce ACRO, a method designed to filter out exogenous noise by focusing strictly on controllability. The core of ACRO is a multi-step inverse model.
Instead of predicting the next frame (which would require modeling the noise), the model takes two observations—one from the current time ($x_t$) and one from $k$ steps in the future ($x_{t+k}$)—and predicts the initial action ($a_t$) that was taken. This approach works for several reasons:
- Invariance: Since exogenous noise is independent of the agent's actions, it provides no predictive value for the action $a_t$, leading the encoder to naturally ignore it.
- Multi-step Advantage: Single-step models can be "lazy," relying on immediate visual cues (like brake lights) rather than the underlying state. A multi-step model forces the representation to capture the agent's actual state to bridge the temporal gap.
- Reward-Free Learning: ACRO identifies controllable features without needing reward signals, making it an ideal candidate for pre-training on diverse datasets.
Benchmarking Robustness
We introduced a suite of new benchmarks to test robustness to increasingly difficult noise:
- EASY-EXO: Standard D4RL tasks with static background distractors.
- MEDIUM-EXO: Adds STL-10 images to frame edges or a fixed background video.
- HARD-EXO: Features complex noise, such as background videos that change per episode or a "Multi-Agent" grid where the agent must distinguish its own actions from eight other autonomous entities.
Key Results and Insights
ACRO consistently outperformed baselines like DRIML, CURL, and DrQv2 across all tested environments. In the "HARD-EXO" changing video task, most baselines saw their performance collapse, while ACRO remained robust. Notably, in the 9-agent grid task, ACRO successfully identified which agent it actually controlled, whereas other methods struggled to distinguish the "self" from the "others."
Beyond performance metrics, decoder reconstructions reveal that ACRO-trained encoders produce blurry, indistinct backgrounds while keeping the agent's body pose sharp. This demonstrates that the model has mathematically learned to "forget" irrelevant parts of the world.
Conclusion
ACRO provides a principled, simple, and effective way to learn representations for offline RL in visually rich environments. By leveraging the concept of controllability through multi-step inverse models, it filters out complex exogenous noise that typically leads to Bellman incompleteness. Future work will explore integrating reward information to prioritize specific controllable elements and testing the transferability of ACRO representations across different noise distributions.
Summary updated