Environment Agnostic Representation for Visual Reinforcement learning

Hyesong Choi, Hunsang Lee, Seongwon Jeong, Dongbo MinICCV 2023 · OpenReview

Abstract

Generalization capability of vision-based deep reinforcement learning (RL) is indispensable to deal with dynamic environment changes that exist in visual observations. The high-dimensional space of the visual input, however, imposes challenges in adapting an agent to unseen environments. In this work, we propose Environment Agnostic Reinforcement learning (EAR), which is a compact framework for domain generalization of the visual deep RL. Environment-agnostic features (EAFs) are extracted by leveraging three novel objectives based on feature factorization, reconstruction, and episode-aware state shifting, so that policy learning is accomplished only with vital features. EAR is a simple single-stage method with a low model complexity and a fast inference time, ensuring a high reproducibility, while attaining state-of-the-art performance in the DeepMind Control Suite and DrawerWorld benchmarks.

Introduction

Deep Reinforcement Learning (RL) has demonstrated remarkable success in controlled simulations. However, vision-based agents often struggle when deployed in real-world or slightly altered virtual environments. This lack of generalization stems from agents overfitting to task-irrelevant visual details, such as background textures or lighting, rather than learning the fundamental physics of the task.

Existing approaches typically rely on Domain Randomization (training on vast variations) or Domain Adaptation (fine-tuning at test time). The authors propose a Domain Generalization approach called Environment Agnostic Representation (EAR). The goal is to train an agent once to inherently ignore irrelevant visual noise and focus solely on "vital" features required for task completion.

The EAR Framework

The EAR framework decomposes visual observations into two distinct representations: Environment-Agnostic Features (EAFs), which capture task-essential information like robot joint positions, and Environment-Specific Features (ESFs), which capture background details like floor color.

The EAR framework architecture for extracting environment-agnostic features.

To achieve this separation without manual supervision, EAR employs three primary objectives:

  1. Feature Factorization ($L_{sep}$): This loss function encourages the EAF and ESF to be mathematically orthogonal, ensuring they represent distinct information.
  2. Reconstruction Constraint ($L_{recon}$): This ensures that the original visual observation can be reconstructed by combining EAF and ESF, preventing the loss of critical information during factorization.
  3. Episode-aware State Shifting (ESS): This novel component utilizes an Action Conditioned Prediction (ACP) module. It posits that if the current EAF and the agent's action are known, the subsequent EAF should be predictable. Since background noise (ESF) does not change in response to the agent's actions, this constraint forces the EAF to capture only the dynamic, task-related elements.

Unlike multi-stage methods, EAR is a streamlined, single-stage framework that trains end-to-end.

Key Results and Insights

EAR was evaluated on rigorous benchmarks including the DeepMind Control (DMC) Suite, the DMC Generalization Benchmark (with randomized colors/videos), the Distracting Control Suite (camera pose changes), and DrawerWorld (robotic manipulation with realistic textures).

Performance of EAR compared to baselines across different tasks and distraction intensities.

Performance and Efficiency

EAR outperformed existing state-of-the-art methods by up to 50.1% on DMC tasks and 97.7% on DrawerWorld. Notably, EAR achieves these results with significantly lower model complexity—roughly half the parameters of previous leaders like VAI (2.55M vs 4.88M). At inference time, EAR is faster than methods requiring test-time adaptation (e.g., PAD), making it more suitable for real-time applications.

Dynamics as a Filter

The core insight of EAR is using the agent's own actions as a filter. If a visual feature does not change predictably when the agent moves, it is treated as environment noise. Furthermore, the researchers confirmed that explicitly excluding ESFs from the policy network is crucial; providing the policy with background information inevitably leads to overfitting and poor generalization.

Conclusion

EAR provides a compact and effective solution for domain generalization in visual RL. By leveraging action-conditioned predictions and feature factorization, it successfully isolates task-relevant information from environmental noise. While EAR shows exceptional performance in simulation, future work will focus on real-world deployment where complex, independent environmental dynamics (e.g., moving people in the background) may challenge the current ACP module.

Summary updated

Abstract

Generalization capability of vision-based deep reinforcement learning (RL) is indispensable to deal with dynamic environment changes that exist in visual observations. The high-dimensional space of the visual input, however, imposes challenges in adapting an agent to unseen environments. In this work, we propose Environment Agnostic Reinforcement learning (EAR), which is a compact framework for domain generalization of the visual deep RL. Environmentagnostic features (EAFs) are extracted by leveraging three novel objectives based on feature factorization, reconstruction, and episode-aware state shifting, so that policy learning is accomplished only with vital features. EAR is a simple single-stage method with a low model complexity and a fast inference time, ensuring a high reproducibility, while attaining state-of-the-art performance in the DeepMind Control Suite and DrawerWorld benchmarks. Code is available at: https://github.com/doihye/EAR.
BibTeX
@inproceedings{choi2023environment,
  author = {Hyesong Choi and Hunsang Lee and Seongwon Jeong and Dongbo Min},
  title = {{Environment Agnostic Representation for Visual Reinforcement learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=c69l1ipZMY},
}
Figure