Abstract
Meta-Reinforcement Learning (MRL) enables agents to adapt rapidly to new tasks by leveraging prior experience. However, standard second-order methods like Model-Agnostic Meta-Learning (MAML) suffer from high computational complexity ($O(d^2)$) and memory overhead, limiting their scalability. This paper introduces MEMRL, a first-order MRL algorithm that utilizes the Moreau Envelope as a smoothed surrogate objective. By transforming the bilevel optimization problem into a first-order framework, MEMRL achieves $O(d)$ complexity while maintaining provable sublinear convergence for non-convex policy gradient objectives.
Introduction
The primary challenge in MRL is finding a policy initialization that can be fine-tuned with minimal data. While MAML is the benchmark for this "learning to learn" paradigm, its reliance on Hessian-vector products makes it computationally expensive and often unstable for large-scale neural networks. Researchers have sought first-order alternatives that preserve the theoretical guarantees of second-order methods without the quadratic cost. MEMRL addresses this gap by employing the Moreau Envelope, a classical optimization tool, to create a differentiable surrogate for the meta-objective.
Methodology: The Moreau Envelope Approach
MEMRL reformulates the MRL problem as a bilevel optimization task. Instead of directly optimizing the standard MAML objective, the authors define a task-specific surrogate $V_i(w)$ using the Moreau Envelope:
$$V_i(w) = \max_{\theta_i} \left[ J_i(\theta_i) - \frac{\lambda}{2} \|\theta_i - w\|^2 \right]$$
In this formulation, $w$ represents the meta-parameters and $\theta_i$ the task-specific parameters. The parameter $\lambda$ acts as a regularizer, controlling the proximity of the adapted policy to the meta-policy.
First-Order Efficiency
The critical advantage of the Moreau Envelope is that its gradient with respect to $w$ is simply proportional to $(\theta_i^* - w)$, where $\theta_i^*$ is the optimal task-specific solution. This eliminates the need for second-order derivatives. The MEMRL algorithm proceeds in two steps:
- Inner Loop: Compute an approximate task-specific solution $\tilde{\theta}_i$ using first-order policy gradients.
- Outer Loop: Update the meta-policy $w$ by moving it toward the average of the task-specific solutions.
Experimental Results
The authors evaluated MEMRL on a multi-task 2D-navigation environment consisting of an $11 \times 11$ grid. The agent's goal was to reach three distinct target destinations (stars) from random starting points.
Performance and Convergence
Theoretical analysis confirms that MEMRL achieves a sublinear convergence rate of $O(1/\epsilon^2)$ to a first-order stationary point, even with inexact inner solvers. Empirically, the algorithm demonstrated high stability. As shown in the reward curves, the agent's performance across all three tasks improved steadily and synchronized, avoiding the volatility often associated with second-order MRL methods.
By iteration $t=120$, the adapted meta-policies were highly efficient, consistently finding the shortest paths to their respective targets. This contrast with the initial random-walk behavior at $t=0$ validates the algorithm's ability to extract and apply meta-knowledge effectively.
Conclusion
MEMRL provides a mathematically rigorous and computationally efficient framework for scaling Meta-Reinforcement Learning. By replacing Hessians with Moreau Envelopes, the algorithm reduces the computational burden to linear complexity relative to the number of parameters. While currently demonstrated on discrete grid worlds, the theoretical foundations of MEMRL suggest it is well-suited for high-dimensional continuous control and multi-agent systems, which remain promising avenues for future research.
Summary updated