Abstract
The promise of learning to learn for robotics rests on the hope that by extracting some information about the learning process itself we can speed up subsequent similar learning tasks. Here, we introduce a computationally efficient online meta-learning algorithm that builds and optimizes a memory model of the optimal learning rate landscape from previously observed gradient behaviors. While performing task specific optimization, this memory of learning rates predicts how to scale currently observed gradients. After applying the gradient scaling our meta-learner updates its internal memory based on the observed effect its prediction had. Our meta-learner can be combined with any gradient-based optimizer, learns on the fly and can be transferred to new optimization tasks. In our evaluations we show that our meta-learning algorithm speeds up learning of MNIST classification and a variety of learning control tasks, either in batch or online learning settings.
Introduction
In the fields of deep learning and robotics, selecting an appropriate learning rate is often a manual, trial-and-error process. While adaptive optimizers like Adam or RMSprop adjust rates based on local gradient statistics, they lack the ability to "learn" the broader structure of the optimization landscape or transfer knowledge between tasks. Most existing meta-learning approaches require a computationally expensive pre-training phase on a distribution of tasks before deployment. This paper introduces an online meta-learner that builds a "memory" of optimal learning rates during the optimization process itself, enabling immediate performance gains and transferability to new, similar tasks.
Methods: Building a Memory for Learning Rates
The proposed framework constructs a memory model, denoted as $h$, which maps gradient behaviors to optimal step sizes.
The Training Signal
The algorithm derives a meta-training signal by analyzing the relationship between consecutive gradients. If a gradient flips sign after a step, the learning rate was likely too high; if the sign remains the same, the step was likely too small. This intuitive feedback allows the meta-learner to adjust its internal model without external supervision.
Locally Weighted Regression (LWR)
To store this knowledge, the authors utilize Locally Weighted Regression (LWR). Unlike complex neural networks, LWR decomposes the learning rate landscape into $M$ local models. This approach allows the system to increase model complexity only where needed and effectively prevents "catastrophic forgetting"—a common issue where new learning erases previously acquired knowledge.
Scalability and Integration
To maintain computational efficiency, parameters are grouped (e.g., by network layer), with each group sharing a learning rate memory. This keeps the overhead to approximately $O(DM)$, where $D$ is the number of parameters. The meta-learner is designed to be modular, functioning as a standalone optimizer (MetaGD) or as an enhancement to existing methods like Adam (MetaGDMemAdam).
Experimental Evaluation
The approach was tested across three distinct domains:
- Rosenbrock Function: A standard non-convex optimization benchmark used to visualize the meta-learner's ability to navigate complex "valleys."
- MNIST Classification: Sequential tasks (e.g., learning to distinguish 1s from 2s, then 1s from 3s) were used to evaluate the transferability of the learned memory.
- Robot Inverse Dynamics: The algorithm learned the mapping from joint states to torques for a 7-DOF Kuka LWR IV robotic arm, including scenarios where the robot had to adapt to varying payloads in real-time.
Key Results
The meta-learning algorithm demonstrated significant improvements in convergence speed and data efficiency. In MNIST transfer tasks, reloading the memory from a previous task allowed the error to drop much more sharply in initial iterations compared to standard gradient descent.
In robotic control, the meta-learner (MetaGD) achieved a lower Mean Squared Error (MSE) within the first 500ms of execution. For instance, with a base rate of 0.0001, MetaGD reduced the average MSE from approximately 256 to 124. Crucially, the meta-update computation took less than 3ms, making it suitable for high-frequency (1kHz) robotic control loops.
Conclusion and Future Work
This work presents a practical, online method for learning learning rates. By treating the optimization process as a learning task itself, the algorithm achieves faster convergence and robust transfer across tasks. Future research will focus on expanding the state index to include context like momentum, improving memory management to prevent indefinite growth, and applying these techniques to the high-stakes domain of Reinforcement Learning.
Summary updated