Determining the Possibility of Transfer Learning in Deep Reinforcement Learning Using Grad-CAM (Student Abstract)

Ho-Taek Joo, Kyung-Joong Kim

Introduction

Training Deep Reinforcement Learning (DRL) agents from scratch is a computationally expensive process, often requiring millions of frames to master a single task. Transfer Learning (TL) offers a potential solution by leveraging knowledge from a pre-trained "source" model to accelerate learning on a "target" task. However, the effectiveness of TL is unpredictable; if the tasks are insufficiently related, "negative transfer" can occur, where the pre-trained model performs worse than a baseline. This research introduces DPTL (Determining the Possibility of Transfer Learning), a novel method to predict transferability a priori using visual attention maps.

Methodology: The DPTL Framework

The proposed DPTL scheme estimates the potential for transfer learning by evaluating how well a source model's visual features align with the objects in a target environment. The process utilizes two primary tools:

  1. Grad-CAM (Gradient-weighted Class Activation Mapping): This technique generates heatmaps from the last convolutional layer of the DRL agent, highlighting the regions of the input image that most influence the model's action selection.
  2. SSIM (Structural Similarity Index Measure): To quantify the relevance of these heatmaps, they are compared against a "ground truth" mask of the target game. This mask is generated by applying a locally adaptive threshold to target game screenshots to isolate key game objects.
The DPTL workflow: A source model processes target game frames to generate Grad-CAM heatmaps, which are then compared to thresholded object masks to calculate a transferability score.

By calculating the SSIM between the Grad-CAM output and the object masks across 1,000 screenshots, researchers can derive a transferability score without performing any gradient updates on the target task.

Experimental Results

The DPTL method was evaluated using a source model trained on DemonAttack and tested against four target Atari games: Phoenix, Pong, SpaceInvaders, and Seaquest. The SSIM scores successfully predicted the ranking of transfer learning performance, with higher visual alignment correlating perfectly with faster learning and higher rewards.

Target Game SSIM Score Transfer Rank
Phoenix 0.62 1
Pong 0.46 2
SpaceInvaders 0.32 3
Seaquest 0.17 4

In games with high SSIM scores, such as Phoenix, the transfer model demonstrated significant performance gains, reaching higher reward plateaus much faster than an "origin model" trained from scratch.

Reward mean over epochs for Phoenix, demonstrating the superior performance of the transfer model predicted by the high SSIM score.

Discussion and Conclusion

The success of DPTL suggests that the convolutional layers of DRL agents act as generalized object detectors. When these visual "intuitions" align with the layout of a new game, transfer learning is likely to be effective. This zero-shot prediction method allows researchers to rank potential target games and save significant compute time. Future work will focus on automating ground truth generation for more complex environments and investigating functional transferability beyond purely visual alignment.

Summary updated

Abstract

Humans are usually good at guessing whether the two games are similar to each other and easily estimate how much time to master new games based on the similarity. Although Deep Reinforcement Learning (DRL) has been successful in various domains, it takes much training time to get a successful controller for a single game. Therefore, there has been much demand for the use of transfer learning to speed up reinforcement learning across multiple tasks. If we can automatically determine the possibility of transfer learning in DRL domain before training, it could efficiently transfer knowledge across multiple games. In this work, we propose a simple testing method, Determining the Possibility of Transfer Learning (DPTL), to determine the transferability of models based on Grad-CAM visualization of the CNN layer from the source model. Experimental results on Atari games show that the transferability measure is successfully suggesting the possibility of transfer learning.
BibTeX
@inproceedings{joo2020determining,
  author = {Ho-Taek Joo and Kyung-Joong Kim},
  title = {{Determining the Possibility of Transfer Learning in Deep Reinforcement Learning Using Grad-CAM (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7188},
}
Figure