TL;DR
DRAP (Dynamics-Risk Adaptive Plasticity) is a new replay-free method for continual learning in robotics that prevents "catastrophic forgetting" by identifying and protecting the neural network components responsible for physically sensitive moments in a task. By weighting parameter importance with a Lyapunov-inspired dynamics-risk score, it successfully retains old skills while learning new ones without needing to store any previous training data.
What Problem Does This Paper Address?
In the world of "Physical AI"—think robot arms or autonomous drones—learning is a never-ending process. However, when you train a model on a new task (like "stacking blocks") after it has already learned an old one (like "opening a drawer"), it often suffers from catastrophic forgetting, completely overwriting the old skill.
Standard continual learning (CL) methods often treat all data points as equally important. But in robotics, some moments are more critical than others. A tiny error while a robot is just moving through empty space is trivial, but a tiny error during a delicate "insertion" phase can lead to a total task failure. These errors compound over time, moving the robot into states it has never seen before. The authors argue that CL for physical agents must be "physics-aware," protecting the specific "muscle memory" required for these high-risk trajectory regions.
Methods
The authors introduce Dynamics-Risk Adaptive Plasticity (DRAP). It operates in three main stages:
- Dynamics-Risk Estimation: For every step in an expert demonstration, the system fits a local linear model to the physics: $s_{t+1} \approx A_t s_t + B_t a_t + c_t$. It then calculates a "risk score" based on the largest singular value of the state-transition matrix $A_t$. If this value is high, it means a small perturbation in the robot's state will be amplified in the next step—marking a "dynamically sensitive" region.
- Risk-Weighted Importance: As the model learns a task, it tracks which "tensors" (groups of weights in the neural network) are most active during these high-risk moments. It calculates a cumulative importance score for each tensor, weighted by the dynamics-risk.
- Adaptive Plasticity: When moving to a new task, DRAP doesn't use a standard penalty. Instead, it directly modifies the AdamW optimizer. Tensors that were highly important for previous critical moments have their learning rates slashed. They become "less plastic," effectively locking in the previous knowledge while allowing less critical parts of the network to adapt to the new task.
To make this efficient for large models like the 7-billion parameter OpenVLA, they use KappaTune to select a specific subnetwork for training based on the "condition number" of the weight matrices, focusing adaptation on the most "plastic" parts of the architecture.
Datasets
The method was evaluated using the LIBERO benchmark, a standard for lifelong robot learning. The researchers tested a sequential pipeline across three suites:
- LIBERO-Spatial: Tasks involving spatial relationships.
- LIBERO-Object: Tasks involving specific object manipulations.
- LIBERO-Goal: Tasks defined by semantic goals.
The evaluation protocol is strictly "replay-free," meaning once the model moves from the Spatial suite to the Object suite, it never sees the Spatial data again.
Key Insights and Ideas
- Physics as a Regularizer: The most striking idea is that we can use classical control theory (Lyapunov stability) to guide modern deep learning. By identifying where physics is "unstable," we can tell the neural network which weights are too important to change.
- Identifying Critical Phases: The "Risk Trace" (see below) shows that the algorithm naturally assigns higher risk to phases like "contact/manipulation" and "insertion," where precision is paramount.
- Optimizer-Level Protection: Unlike methods like EWC (Elastic Weight Consolidation) that add a penalty term to the loss function, DRAP works at the optimizer level. This "immediate protection" prevents even the very first gradient step of a new task from damaging old knowledge.
Key Results
DRAP significantly outperformed standard fine-tuning and the popular EWC baseline in maintaining a balance between learning new tasks and remembering old ones.
- Retention: After training on all three suites, standard fine-tuning essentially "forgot" the first suite (Spatial) entirely, with a success rate dropping to near 0%. DRAP maintained a 34% success rate on that same suite without ever seeing the data again.
- Average Forgetting: Across the entire sequence, DRAP reduced the average forgetting rate to 31%, compared to 77% for standard fine-tuning and 63% for EWC.
- Success-Plasticity Balance: DRAP achieved the highest "Average Success" (55%) across all observed tasks at the end of the training sequence, proving it doesn't just "lock" the model but allows for meaningful new learning.
Limitations and Future Work
The authors acknowledge several areas for growth:
- Simplified Risk Model: The current risk estimator only looks at the state-transition matrix ($A_t$). Future versions should include "control-path sensitivity" ($B_t$) to see how much an action error specifically contributes to the risk.
- Offline Requirement: Currently, the dynamics-risk is precomputed from expert trajectories. An "online" version that estimates risk during live training would be more flexible.
- Longer Sequences: While three suites is a start, real-world robots might need to learn dozens or hundreds of tasks. Testing DRAP's limits over much longer sequences is a necessary next step.
Summary updated