Safety Aware Neural Pruning for Deep Reinforcement Learning (Student Abstract)

Briti Gangopadhyay, Pallab Dasgupta, Soumyajit DeyAAAI 2023 · DOI

Abstract

Deep Reinforcement Learning (DRL) models often require significant computational resources, making them difficult to deploy on embedded systems. While standard pruning techniques reduce model size by maintaining average reward, they frequently compromise safety in edge-case scenarios. This paper introduces a safety-aware pruning method that identifies "recovery states"—points where a pruned model fails but the original succeeds—and selectively reactivates critical neural connections. This approach restores safety while maintaining high levels of model sparsity.

Introduction

Deep Reinforcement Learning (DRL) has shown remarkable success in complex control tasks, such as autonomous driving and robotics. However, the resulting neural networks are typically over-parameterized, posing challenges for deployment on low-power embedded systems with strict memory and latency constraints. Pruning is a common solution to reduce model size, but standard methods prioritize overall performance (average reward) over safety. In safety-critical applications, a pruned network might achieve a high score while failing in rare but catastrophic edge cases because the weights responsible for safety maneuvers were deemed "unimportant" for the average reward. This research bridges the gap between model efficiency and rigorous safety requirements.

Methodology

The proposed method utilizes a multi-step iterative refinement process to ensure safety without sacrificing the benefits of a sparse network.

Overview of the safety-aware pruning and iterative refinement workflow.

The process begins with one-shot pruning, where $k\%$ of weights with the lowest activations are removed while ensuring a minimum reward threshold is met. Next, safety testing via Bayesian Optimization (BO) is employed to efficiently search for "counterexamples"—specific scenarios where the pruned model violates safety constraints.

For every identified failure, the system backtracks through the trajectory to find the Recovery State. This is the final point where the original, unpruned network could have successfully navigated the situation, but the pruned network failed. By calculating the Mean Square Error (MSE) between the actions of the original and pruned networks at these states, the authors use the Jacobian matrix of this loss to identify the pruned weights most responsible for the divergence. These top $p\%$ of weights are then reactivated to their original values. This cycle repeats until all discovered safety violations are resolved.

Experimental Results

The methodology was evaluated on three standard OpenAI Gym environments: Cartpole, Lunar-Lander, and Bipedal Walker.

Comparison of weight distributions between the original policy and the refined pruned policy across three layers.

The results demonstrate that high sparsity can be achieved while maintaining safety:

  • Cartpole: Achieved 94.6% sparsity while correcting 37 distinct safety counterexamples.
  • Lunar-Lander: Achieved 39.48% sparsity.
  • Bipedal Walker: Achieved 42.35% sparsity.

The refined networks successfully eliminated the specific safety infractions uncovered during testing while maintaining their required reward thresholds.

Discussion and Conclusion

A primary insight of this work is that reward is often a blunt instrument for evaluating safety; safety requires analyzing the "tails" of the performance distribution rather than just the mean. The "Recovery State" concept provides a surgical way to pinpoint and fix logic divergences in pruned models.

While effective, the current method relies on testing-based safety via Bayesian Optimization, which does not provide formal mathematical guarantees. Future work aims to integrate formal verification to prove that compressed networks will never violate safety boundaries. Additionally, the lower sparsity achieved in more complex tasks like Lunar-Lander suggests a "safety tax" where more complex behaviors require preserving a larger portion of the original network's weights.

Summary updated

BibTeX
@inproceedings{gangopadhyay2023safety,
  author = {Briti Gangopadhyay and Pallab Dasgupta and Soumyajit Dey},
  title = {{Safety Aware Neural Pruning for Deep Reinforcement Learning (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.26966},
}
Figure