Abstract
Deep Reinforcement Learning (DRL) has shown remarkable success in complex tasks, yet its high computational complexity often hinders deployment on resource-constrained devices. This paper introduces Auto-Agent-Distiller (A2D), the first Neural Architecture Search (NAS) framework specifically designed to automate the creation of efficient DRL agents. By identifying an "optimal model size" for DRL tasks and introducing a novel distillation mechanism to stabilize the search process, A2D discovers task-specific architectures that outperform manually designed models while significantly reducing computational overhead.
The "Bigger is Better" Fallacy in DRL
In many deep learning domains, such as computer vision, increasing model capacity typically leads to improved performance. However, in DRL, the relationship between model size and performance is more complex. The authors identify a "Goldilocks zone"—an optimal model size for every task. If a model is too complex, the training difficulty increases to the point where performance actually degrades. This phenomenon suggests that simply scaling up standard architectures is inefficient for reinforcement learning applications.
Auto-Agent-Distiller (A2D) Framework
Standard Neural Architecture Search (NAS) often fails in the context of DRL due to the inherent instability and high variance of the training process. To address this, A2D incorporates several key innovations:
- AC-based Distillation: A2D utilizes a pre-trained "teacher" agent to guide the search. It distills knowledge from both the Actor (policy) and the Critic (value function). Distilling the Critic provides a stable semantic signal that reduces gradient variance during the search, preventing the "supernet" from failing to converge.
- One-Level Optimization: Unlike common bi-level NAS approaches that alternate between updating weights and architecture parameters, A2D employs a one-level optimization. This provides more stable and unbiased gradient estimations in noisy DRL environments.
- Differentiable Search Space: Using Gumbel-Softmax sampling, A2D makes the architectural search space differentiable, allowing for efficient end-to-end optimization of the agent's structure.
Experimental Results and Efficiency
The A2D framework was evaluated on a suite of Atari 2600 games. The results consistently demonstrate that A2D-searched models achieve superior performance compared to both standard baselines and manually scaled ResNet architectures.
Beyond raw performance, A2D excels in efficiency. The searched models achieved comparable or better test scores than manual baselines while requiring 1.76x to 3.20x fewer FLOPs. The framework automatically adapts to task difficulty, selecting smaller models for simpler games (e.g., Bowling) and more complex architectures for challenging ones (e.g., Alien), proving it can truly find the optimal efficiency-performance trade-off.
Conclusion
Auto-Agent-Distiller represents a significant step toward deploying powerful DRL agents on edge devices. By automating the search for task-specific, efficient architectures and stabilizing the process through actor-critic distillation, A2D bridges the gap between DRL research and real-world application. Future work will explore expanding the search space and applying these techniques to a broader range of DRL algorithms and high-stakes robotics applications.
Summary updated