TKIL: Tangent Kernel Approach for Class Balanced Incremental Learning

Jinlin Xiang, Eli ShlizermanarXiv 2022 · DOI · arXiv

Introduction

Deep learning models excel at static tasks but struggle with sequential learning. When a model trained on Task A is subsequently trained on Task B, it often suffers from catastrophic forgetting, where new information overwrites previously acquired knowledge. While Class Incremental Learning (CIL) methods attempt to mitigate this using exemplars or Knowledge Distillation (KD), they frequently fall victim to latest-task bias. This bias causes the model to favor recent data, leading to a performance collapse as the number of tasks increases.

Methodology: The TKIL Framework

The Tangent Kernel for Incremental Learning (TKIL) framework addresses these challenges through a multi-stage training process designed to maintain class balance and preserve historical knowledge.

Overview of the TKIL framework showing sequential training stages and the inference process.

1. Gradients Tangent Kernel (GTK) Loss

A core innovation of TKIL is the Gradients Tangent Kernel (GTK) Loss. Drawing from Neural Tangent Kernel (NTK) theory, the authors propose that the gradients of feature layers serve as a robust proxy for representation distance. Instead of simply aligning outputs or feature maps, GTK loss minimizes the cosine distance between the gradients of the current model and those of the previous "expert" model.

The GTK loss mechanism aligns current gradients ($G_t$) with previous gradients ($G_{t-1}$) to minimize the angular divergence $\theta$.

By anchoring the learning direction to previous gradients, the model ensures that feature extraction remains consistent across tasks, preventing the representation space from drifting too far toward the newest data manifold.

2. Average Weights Training

To further combat bias, TKIL employs an Average Weights Training approach. During each mini-batch, the system updates separate task-specific model parameters and then averages these weights. This structural balancing ensures that the base model remains "neutral" rather than becoming overly specialized for the most recent task.

3. Inference and Task Prediction

During inference, the model first predicts which task the input belongs to using a self-supervised task-prediction head. Once the task is identified, the system applies task-specific fine-tuning to produce the final class prediction. This two-step process leverages the model's high task-prediction accuracy (often near 100%) to ensure precise classification.

Experimental Results

TKIL was evaluated on standard benchmarks including CIFAR-100, ImageNet-100, and ImageNet-1k, consistently outperforming state-of-the-art methods like iCaRL, iTAML, and PODNet.

  • Superior Accuracy: On CIFAR-100 (10 tasks), TKIL achieved 82.5% accuracy, significantly higher than iTAML (76.6%) and iCaRL (52.6%).
  • Long-Term Stability: In a challenging 25-stage incremental scenario on CIFAR-100, TKIL maintained a high accuracy of 78.97%, while competing methods like AFC trailed by over 14%.
  • Scalability: The method demonstrated strong performance on large-scale datasets, leading ImageNet-1k (10 tasks) with 64.8% accuracy, proving its effectiveness on high-resolution imagery.

Conclusion and Future Work

TKIL provides a theoretically grounded approach to incremental learning by focusing on gradient alignment and structural weight averaging. While the method introduces some computational overhead due to task-specific gradient calculations, its ability to maintain high accuracy over long sequences of tasks represents a significant advancement. Future research will focus on optimizing these computations to improve training efficiency in large-scale production environments.

Summary updated

Abstract

When learning new tasks in a sequential manner, deep neural networks tend to forget tasks that they previously learned, a phenomenon called catastrophic forgetting. Class incremental learning methods aim to address this problem by keeping a memory of a few exemplars from previously learned tasks, and distilling knowledge from them. However, existing methods struggle to balance the performance across classes since they typically overfit the model to the latest task. In our work, we propose to address these challenges with the introduction of a novel methodology of Tangent Kernel for Incremental Learning (TKIL) that achieves class-balanced performance. The approach preserves the representations across classes and balances the accuracy for each class, and as such achieves better overall accuracy and variance. TKIL approach is based on Neural Tangent Kernel (NTK), which describes the convergence behavior of neural networks as a kernel function in the limit of infinite width. In TKIL, the gradients between feature layers are treated as the distance between the representations of these layers and can be defined as Gradients Tangent Kernel loss (GTK loss) such that it is minimized along with averaging weights. This allows TKIL to automatically identify the task and to quickly adapt to it during inference. Experiments on CIFAR-100 and ImageNet datasets with various incremental learning settings show that these strategies allow TKIL to outperform existing state-of-the-art methods.
BibTeX
@misc{xiang2022tkil,
  author = {Jinlin Xiang and Eli Shlizerman},
  title = {{TKIL: Tangent Kernel Approach for Class Balanced Incremental Learning}},
  year = {2022},
  eprint = {2206.08492},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.08492},
}
Figure