Abstract
Continual learning in deep neural networks often relies on a "task oracle" to provide task identities at inference time, a requirement that limits real-world applicability. This paper proposes the use of lightweight task-assigning mappers to infer task identities autonomously. By utilizing fixed feature embeddings and efficient mapping strategies, the proposed system achieves near-oracle performance with minimal memory overhead, bridging the gap between task-dependent models and autonomous lifelong learning.
Introduction
A primary challenge in deep learning is "catastrophic forgetting," where a model loses previously acquired knowledge upon learning new tasks. While "task-dependent" architectures mitigate this by partitioning network weights or using task-specific parameters, they typically require a Task Oracle to specify the current task during testing. In practical scenarios, such as autonomous robotics, such an oracle is unavailable. This research addresses this limitation by introducing Task Mappers—modules designed to identify task IDs on the fly, enabling models to function independently of external task labels.
Proposed Methodology
The authors implement a two-stage pipeline consisting of a Task Mapper and a Fine-Grained Classifier. The Task Mapper analyzes input features to predict a task ID, which then directs the Fine-Grained Classifier (utilizing Parameter Superposition or PSP) to activate the appropriate sub-network.
Task Mapper Variants
Several mapping strategies were evaluated to determine the most effective balance of accuracy and efficiency:
- Incremental Unsupervised Mappers: Techniques such as Nearest Means (NMC) and Gaussian Mixture Models (GMMC) store task prototypes and assign inputs based on proximity in feature space.
- Supervised Prototype Mapping (ARTMAP): An incremental neural network architecture designed for category learning without forgetting.
- Perceptron with Coreset Replay (PCR): A shallow network trained for task identification, utilizing a small "coreset" buffer to maintain performance over time.
A key finding was the importance of using fixed feature embeddings from a pre-trained backbone (e.g., ImageNet-pretrained ResNet). Stable representations proved superior to evolving, task-specific features for the purpose of task identification.
Experimental Evaluation
The system was tested across three distinct lifelong learning paradigms:
- Inter-Dataset Learning (8-Datasets): Sequential learning of diverse datasets (e.g., Flowers, Birds, Cars), testing cross-domain discrimination.
- Permuted MNIST: 25 tasks involving randomly shuffled pixels of MNIST digits.
- Intra-Dataset Learning (CIFAR-100 Super-Classes): 10 tasks where each task represents a visually similar super-class, representing the most challenging scenario for task mapping.
Key Results and Insights
- High Accuracy: The GMMC mapper achieved 93.9% task identification accuracy on the 8-dataset benchmark, nearly matching oracle performance.
- Efficiency: The mappers are highly lightweight, adding only approximately 1.7% to the total parameter count. For instance, the GMMC mapper required only 82 KB of memory for prototypes.
- Performance Gains: The Mapper + PSP-BD pipeline significantly outperformed task-independent baselines like EWC or GEM, which do not utilize task IDs.
- Domain Difficulty: Task mapping is significantly more difficult for intra-dataset tasks (e.g., distinguishing between different insect types) compared to inter-dataset tasks (e.g., distinguishing a bird from a car).
Conclusion and Future Work
This work demonstrates that autonomous task identification is a viable path toward oracle-free lifelong learning. While the mappers excel in inter-dataset scenarios, a performance gap remains in fine-grained intra-dataset tasks. Future research will focus on developing more generalizable visual embeddings that can evolve during the lifelong learning process without sacrificing the stability required for accurate task mapping.
Summary updated