Introduction
Reinforcement Learning (RL) has demonstrated remarkable success in training agents for complex tasks in static environments. However, a significant challenge remains: maintaining "plasticity"—the ongoing ability to learn and adapt—in non-stationary environments where external conditions shift over time. In aerial robotics, such as quadrotor control, these shifts often manifest as unpredictable wind gusts. Standard algorithms like Proximal Policy Optimization (PPO) frequently suffer from "policy collapse" during long-term training in these conditions, where the neural network loses its capacity to adapt, leading to catastrophic failure.
The Challenge of Plasticity Loss
The authors identify a critical phenomenon known as the loss of plasticity. During extended training, neural networks can develop "dormant units"—neurons that cease to contribute to the output. As the percentage of these inactive units grows, the network becomes increasingly brittle. In the context of a drone hovering task, a policy that was previously stable may suddenly fail as environmental parameters change, because the underlying network can no longer update its weights effectively to compensate for new disturbances.
The RECOM Framework
To address this, the researchers introduced the Retrospective Cost Mechanism (RECOM), a bio-inspired framework designed to regulate the learning process dynamically. RECOM is inspired by the human brain's ability to balance dopaminergic reward signals (encouraging success) with error-processing systems (signaling caution).
RECOM operates by calculating a "retrospective cost" over a sliding window of time, combining the negative of the rewards with the actual training loss. Unlike standard RL setups that use fixed or decaying learning rates, RECOM utilizes the cost gradient—the change in cost between windows—to update the learning rate ($L_r$) in real-time. This creates a negative feedback control loop: if the cost rises (indicating a potential loss of performance or stability), the learning rate is adjusted to facilitate re-learning. When combined with L2 regularization, this approach prevents network weights from becoming stagnant or exploding.
Experimental Evaluation
The framework was tested using the RotorPy simulator, a high-fidelity environment for multirotor dynamics. The task required a quadrotor to maintain a stable hover while facing variable wind disturbances (1.5 m/s to 3.0 m/s) that shifted every 2 million timesteps. The total training duration spanned 20 million timesteps to observe long-term stability and plasticity.
Results and Discussion
The experiments compared standard PPO, PPO with L2 regularization, and the proposed RECOM + L2 framework. The results highlighted several key advantages of the RECOM approach:
- Prevention of Policy Collapse: As shown in the performance metrics, standard PPO experienced a dramatic crash after approximately 10 million timesteps. In contrast, RECOM maintained stable rewards throughout the entire 20 million timestep duration.
- Success Rates: In high-wind validation tests (3 m/s), RECOM achieved a 90% success rate, significantly outperforming standard PPO, which plummeted to a 30% success rate.
- Neural Health: RECOM effectively mitigated the growth of dormant units, resulting in 11.29% fewer inactive neurons compared to L2 regularization alone. This confirms that the mechanism successfully preserves the network's capacity for ongoing adaptation.
Conclusion
RECOM provides a robust solution for maintaining plasticity in RL agents operating in unpredictable environments. By explicitly linking reward maximization and loss minimization through a dynamic learning rate, the framework prevents the neural network from becoming brittle. While currently validated in high-fidelity simulations for hovering tasks, future work will focus on applying RECOM to more dynamic maneuvers and transitioning the technology to physical hardware for real-world outdoor flight.
Summary updated