Introduction
Training Deep Reinforcement Learning (DRL) agents from scratch is a computationally expensive process, often requiring millions of frames to master a single task. Transfer Learning (TL) offers a potential solution by leveraging knowledge from a pre-trained "source" model to accelerate learning on a "target" task. However, the effectiveness of TL is unpredictable; if the tasks are insufficiently related, "negative transfer" can occur, where the pre-trained model performs worse than a baseline. This research introduces DPTL (Determining the Possibility of Transfer Learning), a novel method to predict transferability a priori using visual attention maps.
Methodology: The DPTL Framework
The proposed DPTL scheme estimates the potential for transfer learning by evaluating how well a source model's visual features align with the objects in a target environment. The process utilizes two primary tools:
- Grad-CAM (Gradient-weighted Class Activation Mapping): This technique generates heatmaps from the last convolutional layer of the DRL agent, highlighting the regions of the input image that most influence the model's action selection.
- SSIM (Structural Similarity Index Measure): To quantify the relevance of these heatmaps, they are compared against a "ground truth" mask of the target game. This mask is generated by applying a locally adaptive threshold to target game screenshots to isolate key game objects.
By calculating the SSIM between the Grad-CAM output and the object masks across 1,000 screenshots, researchers can derive a transferability score without performing any gradient updates on the target task.
Experimental Results
The DPTL method was evaluated using a source model trained on DemonAttack and tested against four target Atari games: Phoenix, Pong, SpaceInvaders, and Seaquest. The SSIM scores successfully predicted the ranking of transfer learning performance, with higher visual alignment correlating perfectly with faster learning and higher rewards.
| Target Game | SSIM Score | Transfer Rank |
|---|---|---|
| Phoenix | 0.62 | 1 |
| Pong | 0.46 | 2 |
| SpaceInvaders | 0.32 | 3 |
| Seaquest | 0.17 | 4 |
In games with high SSIM scores, such as Phoenix, the transfer model demonstrated significant performance gains, reaching higher reward plateaus much faster than an "origin model" trained from scratch.
Discussion and Conclusion
The success of DPTL suggests that the convolutional layers of DRL agents act as generalized object detectors. When these visual "intuitions" align with the layout of a new game, transfer learning is likely to be effective. This zero-shot prediction method allows researchers to rank potential target games and save significant compute time. Future work will focus on automating ground truth generation for more complex environments and investigating functional transferability beyond purely visual alignment.
Summary updated