Maintaining Plasticity in Reinforcement Learning: A Cost-Aware Framework for Aerial Robot Control in Non-stationary Environments

Ali Tahir Karasahin, Ziniu Wu, Basaran Bahadir KocerarXiv 2025 · DOI · arXiv

Introduction

Reinforcement Learning (RL) has demonstrated remarkable success in training agents for complex tasks in static environments. However, a significant challenge remains: maintaining "plasticity"—the ongoing ability to learn and adapt—in non-stationary environments where external conditions shift over time. In aerial robotics, such as quadrotor control, these shifts often manifest as unpredictable wind gusts. Standard algorithms like Proximal Policy Optimization (PPO) frequently suffer from "policy collapse" during long-term training in these conditions, where the neural network loses its capacity to adapt, leading to catastrophic failure.

The Challenge of Plasticity Loss

The authors identify a critical phenomenon known as the loss of plasticity. During extended training, neural networks can develop "dormant units"—neurons that cease to contribute to the output. As the percentage of these inactive units grows, the network becomes increasingly brittle. In the context of a drone hovering task, a policy that was previously stable may suddenly fail as environmental parameters change, because the underlying network can no longer update its weights effectively to compensate for new disturbances.

The RECOM Framework

To address this, the researchers introduced the Retrospective Cost Mechanism (RECOM), a bio-inspired framework designed to regulate the learning process dynamically. RECOM is inspired by the human brain's ability to balance dopaminergic reward signals (encouraging success) with error-processing systems (signaling caution).

The RECOM framework architecture, illustrating the balance between reward and loss signals to dynamically adjust the RL agent's learning rate.

RECOM operates by calculating a "retrospective cost" over a sliding window of time, combining the negative of the rewards with the actual training loss. Unlike standard RL setups that use fixed or decaying learning rates, RECOM utilizes the cost gradient—the change in cost between windows—to update the learning rate ($L_r$) in real-time. This creates a negative feedback control loop: if the cost rises (indicating a potential loss of performance or stability), the learning rate is adjusted to facilitate re-learning. When combined with L2 regularization, this approach prevents network weights from becoming stagnant or exploding.

Experimental Evaluation

The framework was tested using the RotorPy simulator, a high-fidelity environment for multirotor dynamics. The task required a quadrotor to maintain a stable hover while facing variable wind disturbances (1.5 m/s to 3.0 m/s) that shifted every 2 million timesteps. The total training duration spanned 20 million timesteps to observe long-term stability and plasticity.

Results and Discussion

The experiments compared standard PPO, PPO with L2 regularization, and the proposed RECOM + L2 framework. The results highlighted several key advantages of the RECOM approach:

  • Prevention of Policy Collapse: As shown in the performance metrics, standard PPO experienced a dramatic crash after approximately 10 million timesteps. In contrast, RECOM maintained stable rewards throughout the entire 20 million timestep duration.
  • Success Rates: In high-wind validation tests (3 m/s), RECOM achieved a 90% success rate, significantly outperforming standard PPO, which plummeted to a 30% success rate.
  • Neural Health: RECOM effectively mitigated the growth of dormant units, resulting in 11.29% fewer inactive neurons compared to L2 regularization alone. This confirms that the mechanism successfully preserves the network's capacity for ongoing adaptation.
Comparison of reward per episode across 20 million timesteps. Standard PPO (orange) suffers from policy collapse, while RECOM (blue) maintains stable performance.

Conclusion

RECOM provides a robust solution for maintaining plasticity in RL agents operating in unpredictable environments. By explicitly linking reward maximization and loss minimization through a dynamic learning rate, the framework prevents the neural network from becoming brittle. While currently validated in high-fidelity simulations for hovering tasks, future work will focus on applying RECOM to more dynamic maneuvers and transitioning the technology to physical hardware for real-world outdoor flight.

Summary updated

BibTeX
@misc{karasahin2025maintaining,
  author = {Ali Tahir Karasahin and Ziniu Wu and Basaran Bahadir Kocer},
  title = {{Maintaining Plasticity in Reinforcement Learning: A Cost-Aware Framework for Aerial Robot Control in Non-stationary Environments}},
  year = {2025},
  eprint = {2503.00282},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.00282},
}
Figure