Online Meta-Learning on Non-convex Setting

Zhenxun Zhuang, Yunlong Wang, Kezi Yu, Songtao Lu

Abstract

The online meta-learning framework is designed for the continual lifelong learning setting. It bridges two fields: meta-learning which tries to extract prior knowledge from past tasks for fast learning of future tasks, and online-learning which deals with the sequential setting where problems are revealed one by one. In this paper, we generalize the original framework from convex to non-convex setting, and introduce the local regret as the alternative performance measure. We then apply this framework to stochastic settings, and show theoretically that it enjoys a logarithmic local regret, and is robust to any hyperparameter initialization. The empirical test on a real-world task demonstrates its superiority compared with traditional methods.

Introduction

Standard machine learning models are typically data-hungry, whereas humans can learn new concepts from just a few examples by leveraging prior knowledge. Meta-learning aims to mimic this by extracting "prior knowledge" from past tasks to speed up learning on new ones. However, most meta-learning algorithms (like MAML) assume all training tasks are available simultaneously in a batch.

In the real world, tasks often arrive sequentially—this is the domain of online learning. While "Online Meta-Learning" (OML) exists, previous frameworks relied heavily on the assumption that the loss functions are convex. Since modern deep learning is inherently non-convex, there was a significant theoretical and practical gap: how do we guarantee that an agent is actually getting better over time when the optimization landscape is a complex mix of local minima and plateaus?

The Generalized OML Framework

The authors propose a generalized Online Meta-Learning framework that works for non-convex functions. The core process involves an agent encountering tasks sequentially, adapting its meta-parameters locally, and then updating those parameters globally based on the observed loss.

Algorithm 1: The general Online Meta-Learning framework for sequential tasks.
  1. The Framework: At each time step $t$, the agent encounters a new task $\mathcal{T}_t$, receives training data $\mathcal{D}_t^{tr}$, adapts its current meta-parameters $\mathbf{w}_t$ to $\hat{\mathbf{w}}_t$ using a local adapter $U$, and then suffers a loss $\ell_t$ on test data $\mathcal{D}_t^{ts}$. Finally, it updates its meta-parameters $\mathbf{w}_{t+1}$ using an online learning algorithm $\mathcal{A}$.
  2. AdaGrad-Norm as the Meta-Learner: They employ the AdaGrad-Norm algorithm to update the meta-parameters. This choice is crucial because it is robust to hyperparameter initialization and automatically adjusts the step size based on the geometry of the non-convex landscape.

Theoretical Contributions: Local Regret

Because finding a global optimum in non-convex settings is NP-hard, the authors move away from standard "regret" (comparing performance to the best fixed global predictor). Instead, they use local regret, which measures the sum of the squared norms of the gradients. Essentially, it tracks how quickly the algorithm moves toward local stationary points across a sliding window of tasks.

The most significant theoretical contribution is proving that their algorithm achieves a regret bound of $O(\ln T)$. This logarithmic regret implies that as the agent encounters more tasks, the "error" or "regret" grows very slowly, meaning the agent is effectively accumulating and utilizing knowledge over time.

Empirical Evaluation

The researchers evaluated their approach using the Omniglot dataset, which contains 1,623 different handwritten characters from 50 different alphabets. They used an N-way K-shot protocol (5-way, 5-shot), where the model must learn to distinguish between 5 previously unseen characters using only 5 examples of each.

The authors compared their algorithm against two benchmarks: Train from Scratch (TFS) and Train on Everything (TOE).

Performance comparison of the proposed algorithm against TFS and TOE benchmarks on the Omniglot dataset.

Key Results

  • Superior Accuracy: The proposed algorithm significantly outperformed both benchmarks on the Omniglot task.
  • Fast Adaptation: The OML approach showed a clear trend of "learning to learn"—as the number of task rounds increased, the initial loss on new tasks dropped faster than the baselines.
  • Avoiding "Negative Transfer": A common pitfall in lifelong learning is "Train on Everything" (TOE), where a model tries to find one set of parameters for all tasks. The results show that if tasks are different enough, TOE actually performs worse than starting from scratch because it gets confused by conflicting data. The OML approach avoids this by learning a starting point for adaptation rather than a single static solution.

Conclusion

This work bridges the gap between meta-learning and online learning for non-convex landscapes. While successful on Omniglot, future work includes testing the framework on much larger, more complex datasets like ImageNet or real-world robotics data. Additionally, addressing the computational overhead of maintaining task windows and the sensitivity of the "inner loop" adaptation step remains an area for further research.

Summary updated

Abstract

The online meta-learning framework is designed for the continual lifelong learning setting. It bridges two fields: meta-learning which tries to extract prior knowledge from past tasks for fast learning of future tasks, and online-learning which deals with the sequential setting where problems are revealed one by one. In this paper, we generalize the original framework from convex to non-convex setting, and introduce the local regret as the alternative performance measure. We then apply this framework to stochastic settings, and show theoretically that it enjoys a logarithmic local regret, and is robust to any hyperparameter initialization. The empirical test on a real-world task demonstrates its superiority compared with traditional methods.
BibTeX
@inproceedings{zhuang2020online,
  author = {Zhenxun Zhuang and Yunlong Wang and Kezi Yu and Songtao Lu},
  title = {{Online Meta-Learning on Non-convex Setting}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=ylDb9Oe-qO-},
}
Figure