Abstract
This paper introduces CLIF (Continual LearnIng of Few-shot Learners), a framework designed to enable AI models to accumulate knowledge over a stream of diverse tasks, thereby enhancing their ability to generalize to new tasks rapidly. We propose BiHNet+Reg, a bi-level hypernetwork approach that generates task-specific adapters while employing regularization to mitigate catastrophic forgetting. Experimental results on two large-scale benchmarks, CLIF-26 and CLIF-55, demonstrate that our approach significantly outperforms standard continual learning and few-shot baselines.
Introduction: Bridging Continual and Few-Shot Learning
In the pursuit of "General Linguistic Intelligence," two research fields have largely remained siloed. Few-Shot Learning (FSL) focuses on generalizing to new tasks from minimal examples using static, pre-trained models. Conversely, Continual Learning (CL) aims to learn a sequence of tasks without forgetting previous ones.
However, human learning is synergistic: we leverage past experiences to become faster at acquiring new skills. This paper bridges the gap by proposing the CLIF framework, which asks: Can a model learn continually to generalize more rapidly?
The CLIF Framework and Methodology
The CLIF setup involves a model encountering a sequence of "upstream" tasks (e.g., GLUE tasks) one by one. The model must learn these without revisiting previous data. Evaluation is then conducted on "Few-Shot Performance"—the speed and accuracy of adaptation to entirely unseen "downstream" tasks.
BiHNet+Reg: Bi-level Hypernetworks with Regularization
To address the challenges of CLIF, we propose BiHNet+Reg. This architecture moves beyond standard fine-tuning by utilizing:
- Adapter-Based Architecture: Small MLP layers (adapters) are inserted into a transformer model (e.g., BART). Only these adapters are updated, preserving the core pre-trained knowledge.
- Hypernetwork Generation: A hypernetwork ($g$) is trained to generate the adapter weights based on task representations, providing a layer of abstraction.
- Bi-level Task Representations: The model generates "High-resource" embeddings (averaging all task examples) for stability during training and "Few-shot" embeddings (averaging $K$ examples) to simulate the evaluation environment.
- Regularization: An $L_2$ penalty is applied to the hypernetwork to ensure that learning new tasks does not drastically alter the weight generation logic for previous tasks.
Experimental Setup and Benchmarks
We developed two benchmarks to evaluate the framework:
- CLIF-26: Comprising 9 GLUE tasks for upstream training and 17 DivFSL tasks for downstream evaluation.
- CLIF-55: A larger scale benchmark with 45 upstream and 10 downstream tasks from the HuggingFace library.
All tasks are unified into a sequence-to-sequence question-answering format, enabling a single model to handle diverse objectives like classification and NLI without architectural changes.
Key Insights and Results
- Generalization vs. Forgetting: Interestingly, while catastrophic forgetting severely impacts performance on old tasks, its effect on generalization is less pronounced. In some instances, minor forgetting prevents overfitting to specific upstream tasks, aiding flexibility.
- Superior Performance: BiHNet+Reg achieved a few-shot accuracy of 60.09% on CLIF-26, significantly higher than the 52.66% of a zero-knowledge baseline.
- Retention of Knowledge: While vanilla BART models saw performance on early tasks drop to ~19% after the full sequence, BiHNet+Reg maintained 77.22% accuracy on seen tasks.
- Efficiency: By optimizing the hypernetwork's hidden dimensions, we reduced trainable parameters from 139M to 40M without sacrificing performance.
Limitations and Future Directions
Despite its success, BiHNet+Reg still faces a "capacity gap" compared to models fine-tuned on the entire transformer (BART-Single). Future work will explore expanding adapter capacity, handling blurry task boundaries in real-world lifelong learning, and incorporating unlabeled data streams to further refine model performance.
Summary updated