TL;DR
Visual offline reinforcement learning (RL) struggles when observations contain complex, task-irrelevant visual distractors ("exogenous noise"), such as background videos or other moving agents. This paper introduces ACRO (Agent-Controller Representations for Offline-RL), a reward-free representation learning objective based on multi-step inverse action prediction ($x_t, x_{t+k} \to a_t$). ACRO provably ignores exogenous distractors while retaining all controllable state information necessary to preserve Bellman completeness for downstream offline RL algorithms.
What Problem Does This Paper Address?
Deploying RL in the real world often requires learning purely from static, offline visual datasets (e.g., camera feeds on autonomous vehicles or robot manipulators). A major bottleneck is exogenous information: visual input that changes over time but cannot be controlled by the agent and is irrelevant to the reward (such as passing cars, pedestrians on sidewalks, or moving foliage).
Standard representation learning methods in RL fall into several traps when confronted with complex exogenous distractors:
- Generative/Reconstruction models (e.g., autoencoders) waste capacity modeling irrelevant background noise.
- Contrastive methods (e.g., CURL, DRIML, HOMER) can inadvertently pull exogenous distractors into the latent state or break under temporally correlated noise.
- Single-step inverse dynamics models ($x_t, x_{t+1} \to a_t$) filter out exogenous noise, but frequently fail to capture the full controllable state due to short-horizon shortcuts or an inability to capture long-range dependencies.
Methods
ACRO learns a continuous state representation $\phi(x)$ using an offline dataset of observation and action sequences $\mathcal{D} = \{(x_1, a_1, \dots, x_N, a_N)\}$ in a two-stage pipeline:
- Reward-Free Pretraining with a Multi-Step Inverse Model: ACRO trains an encoder $\phi$ to predict the first action $a_t$ taken between the current observation $x_t$ and a future observation $x_{t+k}$, where $k$ is sampled uniformly from $\{1, \dots, K\}$: $$\max_{\phi} \mathbb{E}_{t, k} \left[ \log P\left(a_t \mid \phi(x_t), \phi(x_{t+k})\right) \right]$$
- Why multi-step? Unlike 1-step models, conditioning on distant future frames forces the representation to capture global, long-range controllable state dynamics without collapsing states.
- Why predict only the first action? Predicting an entire action sequence ($a_t, \dots, a_{t+k}$) is computationally heavier and becomes noisy under environment stochasticity. The authors prove that under deterministic latent dynamics, predicting the first action across varying $k$ is asymptotically equivalent to predicting action sequences.
- Downstream Offline Policy Optimization: The encoder $\phi$ is frozen, mapping visual frames $x_t$ to latent features $s_t = \phi(x_t)$. An off-the-shelf offline RL algorithm (specifically, TD3 + BC with image crop augmentations) is then trained on top of these representations.
Unlike prior theoretical work (e.g., AC-State by Lamb et al., 2022) which forced discrete bottleneck quantization to construct small tabular MDPs, ACRO retains continuous representations and scales directly to deep continuous-control offline RL.
Datasets
The authors evaluate ACRO and introduce new benchmark datasets spanning three tiers of distractor complexity across DeepMind Control tasks (Cheetah-Run, Walker-Walk, Humanoid-Walk) and Atari:
- EASY-EXO: Standard visual D4RL (
v-d4rl) without distractors, as well as static uncorrelated background distractors. - MEDIUM-EXO: Downsampled STL-10 images fixed onto the corner or side of the observation (changing per episode), or a fixed background video.
- HARD-EXO (Newly Introduced):
- Correlated Background Images: Static STL-10 images that stay fixed during an episode but change across episodes.
- Changing Background Videos: Highly dynamic background video distractors that switch every episode.
- Background Agents (Multi-Agent Grid): The observation displays a $3 \times 3$ grid of 9 agents, but the offline actions only control one of them (the other 8 execute random actions). The algorithm must figure out which agent is controllable purely from the data.
- Atari MEDIUM-EXO: Offline DQN-Replay datasets for Pong, Breakout, Q*bert, and Seaquest with CIFAR-10 distractors evaluated using Decision Transformers.
Key Insights and Ideas
- Exogenous Noise Can Violate Bellman Completeness: A standard assumption for offline RL convergence is Bellman completeness (the function class is closed under the Bellman operator). The authors prove that while the minimal agent-controller representation $\phi^\star$ is Bellman complete, a representation that includes exogenous noise (even a strictly more informative "refinement") can violate Bellman completeness. Therefore, keeping task-irrelevant features in the representation can directly harm value estimation.
- Natural Distractor Invariance: By the factorized transition assumption of Exogenous Block MDPs, the distribution $P_\pi(a_t \mid x_t, x_{t+k})$ is conditionally independent of exogenous variables given the endogenous latent states. Thus, an optimal multi-step inverse model does not need to encode exogenous noise.
- Why Not Behavioral Cloning (BC)? While a BC objective ($x_t \to a_t$) also ignores exogenous noise, it collapses when the offline dataset is collected by non-expert or random policies. ACRO condition on future states ($x_{t+k}$), preserving controllable transition structure even under random data.
Key Results
- Superior Robustness Under Hard Distractors: In HARD-EXO settings (such as changing background videos and the 9-agent grid), conventional methods (DRIML, HOMER, CURL, DrQv2, and 1-step inverse dynamics) collapsed or suffered severe degradation. ACRO maintained high performance across expert, medium-expert, and medium datasets (see Figure 6 and Figure 17).
- Outperforming Baselines Without Distractors: Even on standard distractor-free
v-d4rlbenchmarks (Table 2), ACRO achieved higher average normalized returns (Cheetah: 528.7 vs. next best 518.1; Walker Expert: 924.5 vs. 888.6; Humanoid Medium-Replay: 197.8 vs. 121.0), demonstrating that multi-step inverse modeling serves as a strong general-purpose visual representation objective. - Atari Decision Transformer: Pretraining the visual encoder with ACRO outperformed 1-step inverse models and DRIML across Atari games (e.g., Seaquest: $972.4 \pm 136.1$ vs. $274.0 \pm 29.6$ for 1-step and $94.4 \pm 4.6$ for DRIML).
- Visual Reconstructions: Decoders trained to reconstruct images from ACRO representations cleanly reconstructed the agent's joints while blurring out or omitting background videos and distractors (Figure 7).
Limitations and Future Work
- Exo-Free Data Collection Assumption: Theory assumes the behavior policy generating the offline dataset is not itself conditioned on exogenous noise. If the data-collecting agent systematically reacts to distractors, multi-step inverse models could encode them.
- Block-MDP vs. POMDPs: The setup assumes rich Markovian observations where latent states can be decoded from current frame stacks, leaving partially observable MDPs (POMDPs) outside the theoretical guarantees.
- Object Manipulation / Uncontrollable Relevant Factors: In environments where passive or non-controllable objects are crucial for task success (e.g., a ball rolling toward a goal), purely action-predictive objectives might discard object dynamics if not reflected in the policy's actions during the trajectory.
- Extension to Online RL: Extending ACRO to online exploration—where representations and exploration strategies are tightly coupled—remains an open research direction.
Summary updated