Introduction
Deep learning models excel at static tasks but struggle with sequential learning. When a model trained on Task A is subsequently trained on Task B, it often suffers from catastrophic forgetting, where new information overwrites previously acquired knowledge. While Class Incremental Learning (CIL) methods attempt to mitigate this using exemplars or Knowledge Distillation (KD), they frequently fall victim to latest-task bias. This bias causes the model to favor recent data, leading to a performance collapse as the number of tasks increases.
Methodology: The TKIL Framework
The Tangent Kernel for Incremental Learning (TKIL) framework addresses these challenges through a multi-stage training process designed to maintain class balance and preserve historical knowledge.
1. Gradients Tangent Kernel (GTK) Loss
A core innovation of TKIL is the Gradients Tangent Kernel (GTK) Loss. Drawing from Neural Tangent Kernel (NTK) theory, the authors propose that the gradients of feature layers serve as a robust proxy for representation distance. Instead of simply aligning outputs or feature maps, GTK loss minimizes the cosine distance between the gradients of the current model and those of the previous "expert" model.
By anchoring the learning direction to previous gradients, the model ensures that feature extraction remains consistent across tasks, preventing the representation space from drifting too far toward the newest data manifold.
2. Average Weights Training
To further combat bias, TKIL employs an Average Weights Training approach. During each mini-batch, the system updates separate task-specific model parameters and then averages these weights. This structural balancing ensures that the base model remains "neutral" rather than becoming overly specialized for the most recent task.
3. Inference and Task Prediction
During inference, the model first predicts which task the input belongs to using a self-supervised task-prediction head. Once the task is identified, the system applies task-specific fine-tuning to produce the final class prediction. This two-step process leverages the model's high task-prediction accuracy (often near 100%) to ensure precise classification.
Experimental Results
TKIL was evaluated on standard benchmarks including CIFAR-100, ImageNet-100, and ImageNet-1k, consistently outperforming state-of-the-art methods like iCaRL, iTAML, and PODNet.
- Superior Accuracy: On CIFAR-100 (10 tasks), TKIL achieved 82.5% accuracy, significantly higher than iTAML (76.6%) and iCaRL (52.6%).
- Long-Term Stability: In a challenging 25-stage incremental scenario on CIFAR-100, TKIL maintained a high accuracy of 78.97%, while competing methods like AFC trailed by over 14%.
- Scalability: The method demonstrated strong performance on large-scale datasets, leading ImageNet-1k (10 tasks) with 64.8% accuracy, proving its effectiveness on high-resolution imagery.
Conclusion and Future Work
TKIL provides a theoretically grounded approach to incremental learning by focusing on gradient alignment and structural weight averaging. While the method introduces some computational overhead due to task-specific gradient calculations, its ability to maintain high accuracy over long sequences of tasks represents a significant advancement. Future research will focus on optimizing these computations to improve training efficiency in large-scale production environments.
Summary updated