Reducing the Deployment-Time Inference Control Costs of Deep Reinforcement Learning Agents via an Asymmetric Architecture

Chin-Jui Chang, Yu-Wei Chu, Chao-Hsien Ting, Hao-Kang Liu, Zhang-Wei Hong, Chun-Yi Lee

Abstract

Deep Reinforcement Learning (DRL) has enabled complex robotic control, yet the deployment of high-capacity Deep Neural Networks (DNNs) often incurs a significant "computational tax" on battery-powered systems. We present an asymmetric hierarchical framework that mitigates these costs by dynamically switching between a large, high-performance policy and a small, energy-efficient one. By penalizing the use of the larger network during training, the agent learns to reserve expensive computations for critical maneuvers while employing the cheaper model for simpler states. Our approach reduces inference costs (FLOPs) by up to 82% across standard benchmarks with negligible performance loss.

Introduction

In robotics, energy is a finite and critical resource. While large DNNs are necessary to handle complex state-action mappings, their high computational demand quickly drains batteries. Conversely, small DNNs offer efficiency but often lack the capacity to master difficult tasks. Traditional knowledge distillation attempts to compress expertise into a single small network but frequently fails to capture the full nuance required for complex scenarios. This paper proposes a dynamic alternative: an asymmetric architecture that selects the appropriate "brain" based on the immediate task complexity.

Methodology

The proposed system leverages Hierarchical Reinforcement Learning (HRL) to manage two sub-policies with differing computational footprints.

  1. Master Policy ($\pi_\Omega$): A lightweight DQN that observes the environment and selects a sub-policy to execute for a fixed duration of $n_\omega$ timesteps.
  2. Sub-Policies ($\pi_\omega$):
    • Large DNN ($\pi_{\omega_{large}}$): Designed for high-capacity control in complex scenarios.
  3. Small DNN ($\pi_{\omega_{small}}$): Optimized for low-cost execution during steady-state or simple maneuvers.
  4. Cost-Aware Training: To encourage efficiency, the master policy is trained with a modified reward function: $r_{\Omega_t} = \sum r_i - \lambda n_\omega c_\omega$, where $c_\omega$ represents the computational cost (FLOPs) and $\lambda$ is a scaling factor. This "computational tax" forces the agent to justify the use of the large network through significant performance gains.
  5. Shared Experience Replay: Both sub-policies, implemented via Soft Actor-Critic (SAC), share a common replay buffer. This allows the small network to benefit from high-quality transitions generated by the large network, improving data efficiency.
Asymmetric hierarchical architecture with cost-aware training.

Experimental Evaluation

The framework was evaluated on standard robotic control benchmarks, including OpenAI Gym (Swimmer-v3, Ant-v3) and DeepMind Control Suite (walker-stand, finger-spin), using the MuJoCo physics engine.

Key Results

The asymmetric architecture achieved a "best of both worlds" outcome:

  • Significant Cost Reduction: Total FLOP reductions ranged from 37.5% to 82.3% compared to a baseline using only the large network.
  • Performance Retention: In most tasks, the agent maintained scores comparable to the large-only baseline. For instance, in the walker-stand task, the agent used the large network only 5.7% of the time—primarily to stand up—and switched to the small network for balance maintenance, yielding an 82% reduction in computation.

Complexity is Transient

A core insight of this work is that task difficulty is not constant. As shown in the Swimmer task, the agent utilizes the large network to execute powerful strokes and then switches to the small network to maintain posture during the glide phase.

Dynamic policy switching in the Swimmer-v3 environment.

Discussion and Conclusion

The asymmetric approach outperforms traditional distillation by maintaining the "expert" large network on standby for edge cases. However, the system's efficiency is sensitive to the cost penalty $\lambda$ and the decision interval $n_\omega$. Future work will explore adaptive schemes for $n_\omega$ and automated architecture search to further optimize the balance between performance and power consumption. By treating computation as a cost to be minimized, this framework provides a practical path for deploying sophisticated DRL agents on resource-constrained mobile robots.

Summary updated

BibTeX
@inproceedings{chang2021reducing,
  author = {Chin-Jui Chang and Yu-Wei Chu and Chao-Hsien Ting and Hao-Kang Liu and Zhang-Wei Hong and Chun-Yi Lee},
  title = {{Reducing the Deployment-Time Inference Control Costs of Deep Reinforcement Learning Agents via an Asymmetric Architecture}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=nR-RdRjmh7},
}
Figure