Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search

Yonggan Fu, Zhongzhi Yu, Yongan Zhang, Yingyan Lin

Abstract

Deep Reinforcement Learning (DRL) has shown remarkable success in complex tasks, yet its high computational complexity often hinders deployment on resource-constrained devices. This paper introduces Auto-Agent-Distiller (A2D), the first Neural Architecture Search (NAS) framework specifically designed to automate the creation of efficient DRL agents. By identifying an "optimal model size" for DRL tasks and introducing a novel distillation mechanism to stabilize the search process, A2D discovers task-specific architectures that outperform manually designed models while significantly reducing computational overhead.

The "Bigger is Better" Fallacy in DRL

In many deep learning domains, such as computer vision, increasing model capacity typically leads to improved performance. However, in DRL, the relationship between model size and performance is more complex. The authors identify a "Goldilocks zone"—an optimal model size for every task. If a model is too complex, the training difficulty increases to the point where performance actually degrades. This phenomenon suggests that simply scaling up standard architectures is inefficient for reinforcement learning applications.

Figure 1: Test scores across different Atari games for various ResNet depths. The results show that increasing model complexity (e.g., moving from ResNet-20 to ResNet-74) often leads to diminishing or even negative returns, highlighting the need for task-specific architecture optimization.

Auto-Agent-Distiller (A2D) Framework

Standard Neural Architecture Search (NAS) often fails in the context of DRL due to the inherent instability and high variance of the training process. To address this, A2D incorporates several key innovations:

  • AC-based Distillation: A2D utilizes a pre-trained "teacher" agent to guide the search. It distills knowledge from both the Actor (policy) and the Critic (value function). Distilling the Critic provides a stable semantic signal that reduces gradient variance during the search, preventing the "supernet" from failing to converge.
  • One-Level Optimization: Unlike common bi-level NAS approaches that alternate between updating weights and architecture parameters, A2D employs a one-level optimization. This provides more stable and unbiased gradient estimations in noisy DRL environments.
  • Differentiable Search Space: Using Gumbel-Softmax sampling, A2D makes the architectural search space differentiable, allowing for efficient end-to-end optimization of the agent's structure.

Experimental Results and Efficiency

The A2D framework was evaluated on a suite of Atari 2600 games. The results consistently demonstrate that A2D-searched models achieve superior performance compared to both standard baselines and manually scaled ResNet architectures.

Figure 2: Performance comparison between A2D-One-level, A2D-Bi-level, and Vanilla-DNAS. The A2D-One-level approach significantly outperforms the baselines, demonstrating the effectiveness of the proposed distillation and optimization strategy in finding high-performing agents.

Beyond raw performance, A2D excels in efficiency. The searched models achieved comparable or better test scores than manual baselines while requiring 1.76x to 3.20x fewer FLOPs. The framework automatically adapts to task difficulty, selecting smaller models for simpler games (e.g., Bowling) and more complex architectures for challenging ones (e.g., Alien), proving it can truly find the optimal efficiency-performance trade-off.

Conclusion

Auto-Agent-Distiller represents a significant step toward deploying powerful DRL agents on edge devices. By automating the search for task-specific, efficient architectures and stabilizing the process through actor-critic distillation, A2D bridges the gap between DRL research and real-world application. Future work will explore expanding the search space and applying these techniques to a broader range of DRL algorithms and high-stakes robotics applications.

Summary updated

Abstract

AlphaGo's astonishing performance has ignited an explosive interest in developing deep reinforcement learning (DRL) for numerous real-world applications, such as intelligent robotics. However, the often prohibitive complexity of DRL stands at the odds with the required real-time control and constrained resources in many DRL applications, limiting the great potential of DRL powered intelligent devices. While substantial efforts have been devoted to compressing other deep learning models, existing works barely touch the surface of compressing DRL. In this work, we first identify that there exists an optimal model size of DRL that can maximize both the test scores and efficiency, motivating the need for task-specific DRL agents. We therefore propose an Auto-Agent-Distiller (A2D) framework, which to our best knowledge is the first neural architecture search (NAS) applied to DRL to automatically search for the optimal DRL agents for various tasks that optimize both the test scores and efficiency. Specifically, we demonstrate that vanilla NAS can easily fail in searching for the optimal agents, due to its resulting high variance in DRL training stability, and then develop a novel distillation mechanism to distill the knowledge from both the teacher agent's actor and critic to stabilize the searching process and improve the searched agents' optimality. Extensive experiments and ablation studies consistently validate our findings and the advantages and general applicability of our A2D, outperforming manually designed DRL in both the test scores and efficiency. All the codes will be released upon acceptance.
BibTeX
@misc{fu2020auto,
  author = {Yonggan Fu and Zhongzhi Yu and Yongan Zhang and Yingyan Lin},
  title = {{Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search}},
  year = {2020},
  eprint = {2012.13091},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.13091},
}
Figure