Abstract
Vision-based Reinforcement Learning (RL) agents often struggle to distinguish between static appearance and dynamic motion, leading to poor sample efficiency and limited generalization. We present Simoun, a framework that explicitly decouples these information streams using a dual-path architecture. By incorporating a Structural Interactive Module (SIM) and a consistency-guided curiosity reward, Simoun achieves state-of-the-art performance in complex environments such as autonomous driving (CARLA) and continuous control (DMControl).
Introduction
In vision-based RL, agents must learn optimal behaviors directly from raw pixel observations. These pixels contain a complex mixture of appearance (object identity and spatial location) and motion (temporal dynamics). Traditional methods often rely on stacking multiple frames or simple frame differencing, which can lead to "information bias"—where models become overly reliant on static backgrounds or fail to reconcile specific objects with their movements.
This lack of clear representation results in low sample efficiency and poor robustness to environmental changes, such as shifts in background color or lighting. Simoun addresses these challenges by synergizing interactive motion and appearance understanding.
Methodology
The Simoun architecture is built upon three primary innovations designed to improve feature representation and exploration:
1. Dual-Path Architecture
Unlike single-encoder models, Simoun employs two distinct paths to process visual input:
- Appearance Path: Processes individual frames to capture spatial structures and object details.
- Motion Path: Processes "residual frames" (the difference between consecutive observations) to explicitly model temporal dynamics.
2. Structural Interactive Module (SIM)
To move beyond simple feature concatenation, the SIM utilizes an attention-like mechanism that allows the two paths to interact. This module correlates motion features with appearance features, enabling the agent to understand which specific objects are associated with which movements.
3. Consistency-guided Curiosity Module
To facilitate learning in sparse-reward environments, Simoun introduces a novel intrinsic reward based on consistency. It measures how well the predicted motion features explain the observed changes in appearance features over time. High inconsistency triggers a "curiosity" reward, encouraging the agent to explore states where its internal dynamics model is uncertain.
Experiments and Results
Simoun was evaluated on the DeepMind Control Suite (DMControl) and the CARLA autonomous driving simulator.
Key Insights
Analysis of the "Units Encoding Factor" demonstrates that standard methods are heavily biased toward static features. Simoun’s dual-path approach successfully shifts the model's focus toward dynamic elements, which are typically more critical for task performance. The interactive nature of the SIM module ensures that the synergy between appearance and motion is captured structurally.
Performance Metrics
- Efficiency: On DMControl, Simoun outperformed state-of-the-art methods like CURL and DrQ, particularly at the 100k step mark, indicating superior learning speed.
- Autonomous Driving: In CARLA, Simoun achieved an average distance of 281m, significantly outperforming the next best baseline, DeepMDP (170m), while also exhibiting safer driving behavior with reduced crash intensity.
- Generalization: Simoun maintained high performance in environments with distracting video backgrounds, where traditional methods failed, proving its ability to decouple task-relevant motion from visual noise.
Conclusion
Simoun demonstrates that explicit decoupling and interactive fusion of motion and appearance are vital for robust vision-based RL. While the current implementation uses simple residual frames for motion extraction, future work could explore more advanced temporal modeling like optical flow. Additionally, optimizing the computational overhead of the dual-path architecture remains an objective for real-time deployment on edge devices.
Summary updated