Abstract
We introduce a method for flexible and efficient continual learning in open-vocabulary image classification, drawing inspiration from the complementary learning systems observed in human cognition. Specifically, we propose to combine predictions from a CLIP zero-shot model and the exemplar-based model, using the zero-shot estimated probability that a sample's class is within the exemplar classes. We also propose a “tree probe” method, an adaption of lazy learning principles, which enables fast learning from new examples with competitive accuracy to batch-trained linear models. We test in data incremental, class incremental, and task incremental settings, as well as ability to perform flexible inference on varying subsets of zero-shot and learned categories. Our proposed method achieves a good balance of learning speed, target task effectiveness, and zero-shot effectiveness.
Introduction: The Challenge of Open-Vocabulary Learning
Large-scale foundation models like CLIP have revolutionized open-vocabulary classification, allowing for the identification of novel objects through natural language descriptions. However, these models face significant hurdles in incremental learning. Standard fine-tuning is computationally prohibitive for real-time updates and often results in "catastrophic forgetting," where the model's general zero-shot performance degrades as it learns specific new data.
To address this, we define three primary goals for a robust continual learning system:
- Flexible Inference: Maintaining open-vocabulary capabilities across arbitrary label sets.
- Continual Improvement: Enhancing accuracy with new data without degrading previous performance.
- Efficient Incremental Learning: Enabling near-instantaneous updates as new examples are received.
Methodology: Complementary Learning Systems
Our approach is inspired by the Complementary Learning Systems (CLS) theory of human cognition, which posits a dual-system architecture for memory: a fast-learning hippocampus for episodic experiences and a slow-learning neocortex for consolidated knowledge.
The Consolidated System
We utilize a frozen CLIP model as the "neocortex." By keeping the foundation model weights static, we ensure that the system's core zero-shot capabilities are preserved and never subject to forgetting.
The Exemplar-based System: TreeProbe
As the "hippocampus," we introduce TreeProbe, a novel hierarchical system designed for rapid learning. Unlike traditional linear probes that require $O(n)$ retraining, TreeProbe constructs a hierarchical clustering tree of training exemplars. Each leaf node maintains a local linear classifier. When new data is introduced, only the relevant leaf nodes are updated, resulting in a training complexity of $O(\log n)$. This allows for real-time model updates that scale efficiently with the number of examples.
Adaptive Instance Marginalization (AIM)
To integrate these systems, we developed Adaptive Instance Marginalization (AIM). At inference time, the system uses the base CLIP model to estimate the probability that a sample belongs to a class already present in the exemplar memory. This probability acts as a gate: samples resembling learned data rely more on TreeProbe, while novel samples rely on the zero-shot CLIP model.
Experimental Evaluation
We evaluated the system using the Multi-Task Incremental Learning (MTIL) benchmark across diverse datasets, including CIFAR100, SUN397, and specialized domains like FGVCAircraft and EuroSAT. Generalization was tested on zero-shot tasks such as ImageNet and UCF101.
Results and Key Insights
The proposed method demonstrates a superior balance of speed and accuracy:
- Superior Accuracy: TreeProbe achieved an average accuracy of 85.5% on the MTIL benchmark, outperforming the previous state-of-the-art (ZSCL at 83.6%) and the base CLIP model (65.3%).
- Zero Forgetting: The system maintained a zero-shot transfer score of 69.3%, showing virtually no degradation from the original CLIP's 69.4%, whereas other methods saw drops of up to 19%.
- Massive Speedup: TreeProbe is approximately 10x faster than fine-tuning and 22x faster than ZSCL, processing new examples in milliseconds.
Conclusion
This research highlights that foundation models do not necessarily require weight updates to improve domain-specific performance; rather, they require efficient mechanisms to map existing features to new labels. While the current system requires storing raw embeddings (approximately 4GB for 1 million exemplars), it provides a scalable path toward real-time, open-vocabulary learners. Future work will address memory pruning strategies and the potential for background consolidation of exemplar knowledge into the consolidated system.