Abstract
Meta-learning methods like Model-Agnostic Meta-Learning (MAML) have revolutionized few-shot learning by training models to rapidly adapt to new tasks. However, these methods typically optimize for expected performance across a specific task distribution, leaving them vulnerable to rare, difficult tasks or shifts in the task distribution between training and deployment. We introduce Task-Robust MAML (TR-MAML), a framework that minimizes the maximum loss over observed tasks. By prioritizing the worst-case scenario, TR-MAML produces an initialization that is robust to distribution shifts and performs uniformly well across diverse task sets. We provide theoretical convergence rates and demonstrate empirical gains in sinusoid regression and image classification.
Introduction
The goal of meta-learning is to acquire a "prior" or initialization that allows a model to learn a new task with minimal data. MAML achieves this by finding a parameter initialization from which a few steps of gradient descent lead to low loss on a variety of tasks. Despite its success, MAML's objective is fundamentally tied to the average performance across a training distribution.
This reliance on the mean is problematic in two primary scenarios:
- Task Heterogeneity: If some tasks are significantly harder or rarer than others, an average-optimizing model may ignore them to achieve better performance on the majority.
- Distribution Shift: If the distribution of tasks at test time differs from the training distribution, the model's performance can degrade significantly.
In safety-critical or fairness-sensitive applications—such as medical diagnosis or facial recognition across different demographics—"average" performance is insufficient. We need models that are robust to the worst-case task. TR-MAML addresses this by asking: How can we train a meta-learner that performs uniformly well across all tasks, even the hardest ones?
The TR-MAML Framework
To address these limitations, we reformulate the meta-learning objective as a distribution-agnostic problem. Instead of minimizing the expectation of the loss, we minimize the maximum loss across the set of training tasks.
Min-Max Formulation
TR-MAML is framed as a min-max game. We introduce a probability distribution (weights) over the $n$ training tasks. The objective is to find model parameters $\theta$ that minimize the weighted loss, while an "adversary" simultaneously updates the task weights to maximize this loss. This forces the model to focus its learning capacity on the tasks where it currently performs most poorly.
Optimization Algorithm
We solve this min-max problem using a robust stochastic mirror-prox algorithm. In practice, the training process alternates between:
- Model Update: A gradient descent step to update the model parameters based on the current task weights.
- Weight Update: A projected gradient ascent step to update the task weights, ensuring they remain on the probability simplex.
Despite the added complexity of the min-max formulation, TR-MAML maintains effectively the same per-iteration computational cost as standard MAML.
Theoretical Foundations
A major contribution of this work is its rigorous theoretical backing. We establish convergence rates for the proposed algorithm in both convex and non-convex settings:
- Convex Setting: The algorithm converges to an $\epsilon$-accurate point at the optimal rate of $\mathcal{O}(1/\epsilon^2)$.
- Non-convex Setting: For deep neural networks, it reaches an $(\epsilon, \delta)$-stationary point at a rate of $\mathcal{O}(\max\{1/\epsilon^5, 1/\delta^5\})$.
Furthermore, we provide a generalization bound based on Rademacher complexity. This bound suggests that TR-MAML will generalize effectively to any new task that lies within the convex hull of the training tasks, providing a formal guarantee of its distribution-agnostic nature.
Experimental Evaluation
We evaluated TR-MAML against standard MAML on benchmarks designed to highlight distribution sensitivity and worst-case performance.
Sinusoid Regression
In this synthetic experiment, the model must predict sine wave values from a few points. We introduced a distribution shift: during training, the model saw mostly "easy" tasks (low amplitude) and few "hard" tasks (high amplitude). At test time, tasks were sampled uniformly.
- MAML struggled with the hard tasks, resulting in a worst-case Mean Squared Error (MSE) of 3.89.
- TR-MAML significantly improved performance on the tails, reducing the worst-case MSE to 2.82 and cutting the standard deviation of errors across tasks from 0.88 to 0.43, proving much more uniform performance.
Fine-Grained Omniglot Classification
We tested image classification on the Omniglot dataset using a "fine-grained" setup where tasks required distinguishing between characters within the same alphabet. In a challenging 20-way, 1-shot setting:
- Worst-Case Accuracy: TR-MAML boosted the accuracy on the most difficult alphabets from 49.7% to 58.4%.
- Mean Accuracy: Because it did not overfit to the most heavily represented alphabets in the training set, TR-MAML actually improved the unweighted mean accuracy across all test alphabets from 67.6% to 74.3%.
Conclusion
TR-MAML shifts the meta-learning paradigm from average-case utility to worst-case robustness. By minimizing the maximum task loss, it creates an initialization that is inherently more stable and better prepared for real-world distribution shifts. While maintaining a computational cost similar to standard MAML, TR-MAML provides the theoretical and empirical reliability necessary for deploying meta-learning in sensitive domains. Future work will explore scaling this approach to massive task sets by grouping similar task instances to keep the weight projection step manageable.
Summary updated