Multi-View Deep Attention Network for Reinforcement Learning (Student Abstract)

Yueyue Hu, Shiliang Sun, Xin Xu, Jing ZhaoAAAI 2020 · DOI

Standard reinforcement learning (RL) agents often struggle because they rely on a single, limited representation of their environment. This research introduces the Multi-View Deep Attention Network (MvDAN), a multi-view approach that generates multiple policies from different data representations and uses an attention mechanism to intelligently combine them. This results in more stable and higher-performing agents on complex Atari benchmarks.

The Challenge: Representation Bottlenecks

In Deep Reinforcement Learning (DRL), an agent’s performance is heavily dependent on its ability to represent observations, such as game frames. Typically, a single deep network serves as a visual feature extractor. However, a single "view" of the data might overlook critical statistical properties or be overly sensitive to noise.

While multi-view learning—learning from multiple distinct feature sets—has seen success in traditional machine learning, its application to RL has been limited. The primary obstacle is view disagreement: if two different representations of the same state suggest conflicting actions, the agent may become unstable or fail to converge. MvDAN addresses this by bridging multi-view learning and RL to create a more comprehensive strategy.

The MvDAN Architecture

The authors propose a three-part architecture to handle multi-view integration effectively:

  1. Multi-view Policy Generation: The architecture employs $N$ different Q-networks to extract $N$ distinct visual representations from the same observation, producing a set of view-specific policies.
  2. Attention-Based Policy Integration: Since not all views are equally useful at all times, an attention module calculates weights for each view based on the relative importance of its action-value function ($Q$).
  3. Multi-view Generalized Policy Improvement (MvGPI): To mitigate view disagreement, a consistency penalty is introduced. This regularization term forces the different view-specific action-value functions to align their decisions, ensuring the agent operates in a "multi-view aligned space."
Architecture of the Multi-View Deep Attention Network (MvDAN)

Experimental Evaluation

MvDAN was evaluated on eight Atari 2600 video games, including Pong, Ms. Pac-Man, Seaquest, and Qbert*. The researchers compared MvDAN against the standard Deep Q-Network (DQN) baseline, using two manually generated view representations for the experiments.

Key Results and Insights

  • Superior Performance and Stability: MvDAN significantly outperformed DQN on most benchmark games. For instance, in Ms. Pac-Man, MvDAN achieved a score of 75.4 compared to DQN’s 56.6. In Qbert*, the standard deviation of the reward was nearly halved, suggesting that the multi-view approach effectively mitigates training volatility.
  • Faster Convergence: Training curves indicate that MvDAN reaches higher reward thresholds in fewer epochs than the baseline. By analyzing the problem from multiple "angles" simultaneously, the agent learns the underlying value function more efficiently.
  • Complementarity: The attention mechanism allows the agent to prioritize the representation that is most confident or relevant in a specific game state, turning potential competition between views into a complementary advantage.
Training performance comparison between MvDAN and DQN across four Atari games

Limitations and Future Outlook

While MvDAN shows promise, several areas remain for future exploration. The current implementation relies on manually generated views; future work could investigate methods to automatically learn these representations. Additionally, maintaining multiple policy generators increases computational overhead, and the scalability of adding more than two views requires further study to determine if there are diminishing returns in performance.

Summary updated

BibTeX
@inproceedings{hu2020multi,
  author = {Yueyue Hu and Shiliang Sun and Xin Xu and Jing Zhao},
  title = {{Multi-View Deep Attention Network for Reinforcement Learning (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7177},
}
Figure