Simoun: Synergizing Interactive Motion-appearance Understanding for Vision-based Reinforcement Learning

Yangru Huang, Peixi Peng, Yifan Zhao, Yunpeng Zhai, Haoran Xu, Yonghong Tian

Abstract

Vision-based Reinforcement Learning (RL) agents often struggle to distinguish between static appearance and dynamic motion, leading to poor sample efficiency and limited generalization. We present Simoun, a framework that explicitly decouples these information streams using a dual-path architecture. By incorporating a Structural Interactive Module (SIM) and a consistency-guided curiosity reward, Simoun achieves state-of-the-art performance in complex environments such as autonomous driving (CARLA) and continuous control (DMControl).

Introduction

In vision-based RL, agents must learn optimal behaviors directly from raw pixel observations. These pixels contain a complex mixture of appearance (object identity and spatial location) and motion (temporal dynamics). Traditional methods often rely on stacking multiple frames or simple frame differencing, which can lead to "information bias"—where models become overly reliant on static backgrounds or fail to reconcile specific objects with their movements.

Comparison of observation processing methods in vision-based RL, highlighting Simoun's interactive dual-path approach.

This lack of clear representation results in low sample efficiency and poor robustness to environmental changes, such as shifts in background color or lighting. Simoun addresses these challenges by synergizing interactive motion and appearance understanding.

Methodology

The Simoun architecture is built upon three primary innovations designed to improve feature representation and exploration:

1. Dual-Path Architecture

Unlike single-encoder models, Simoun employs two distinct paths to process visual input:

  • Appearance Path: Processes individual frames to capture spatial structures and object details.
  • Motion Path: Processes "residual frames" (the difference between consecutive observations) to explicitly model temporal dynamics.

2. Structural Interactive Module (SIM)

To move beyond simple feature concatenation, the SIM utilizes an attention-like mechanism that allows the two paths to interact. This module correlates motion features with appearance features, enabling the agent to understand which specific objects are associated with which movements.

The overall architecture of Simoun, illustrating the dual-path flow and the integration of the Structural Interactive Module.

3. Consistency-guided Curiosity Module

To facilitate learning in sparse-reward environments, Simoun introduces a novel intrinsic reward based on consistency. It measures how well the predicted motion features explain the observed changes in appearance features over time. High inconsistency triggers a "curiosity" reward, encouraging the agent to explore states where its internal dynamics model is uncertain.

Experiments and Results

Simoun was evaluated on the DeepMind Control Suite (DMControl) and the CARLA autonomous driving simulator.

Key Insights

Analysis of the "Units Encoding Factor" demonstrates that standard methods are heavily biased toward static features. Simoun’s dual-path approach successfully shifts the model's focus toward dynamic elements, which are typically more critical for task performance. The interactive nature of the SIM module ensures that the synergy between appearance and motion is captured structurally.

Performance Metrics

  • Efficiency: On DMControl, Simoun outperformed state-of-the-art methods like CURL and DrQ, particularly at the 100k step mark, indicating superior learning speed.
  • Autonomous Driving: In CARLA, Simoun achieved an average distance of 281m, significantly outperforming the next best baseline, DeepMDP (170m), while also exhibiting safer driving behavior with reduced crash intensity.
  • Generalization: Simoun maintained high performance in environments with distracting video backgrounds, where traditional methods failed, proving its ability to decouple task-relevant motion from visual noise.

Conclusion

Simoun demonstrates that explicit decoupling and interactive fusion of motion and appearance are vital for robust vision-based RL. While the current implementation uses simple residual frames for motion extraction, future work could explore more advanced temporal modeling like optical flow. Additionally, optimizing the computational overhead of the dual-path architecture remains an objective for real-time deployment on edge devices.

Summary updated

Abstract

Efficient motion and appearance modeling are critical for vision-based Reinforcement Learning (RL). However, existing methods struggle to reconcile motion and appearance information within the state representations learned from a single observation encoder. To address the problem, we present Synergizing Interactive Motion-appearance Understanding (Simoun), a unified framework for vision-based RL Given consecutive observation frames, Simoun deliberately and interactively learns both motion and appearance features through a dual-path network architecture. The learning process collaborates with a structural interactive module, which explores the latent motion-appearance structures from the two network paths to leverage their complementarity. To promote sample efficiency, we further design a consistency-guided curiosity module to encourage the exploration of under-learned observations. During training, the curiosity module provides intrinsic rewards according to the consistency of environmental temporal dynamics, which are deduced from both motion and appearance network paths. Experiments conducted on Deep-Mind control suite and CARLA automatic driving benchmarks demonstrate the effectiveness of Simoun, where it performs favorably against state-of-the-art methods.
BibTeX
@inproceedings{huang2023simoun,
  author = {Yangru Huang and Peixi Peng and Yifan Zhao and Yunpeng Zhai and Haoran Xu and Yonghong Tian},
  title = {{Simoun: Synergizing Interactive Motion-appearance Understanding for Vision-based Reinforcement Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00023},
}
Figure