TAP: Tapping the Discriminative Potential of Frozen Representations for Class-Incremental Learning

Zihao Wang, Yu-Wei Zhan, Xiao-Qian Liu, Zhen-Duo Chen, Weitong Liu, Xin Luo, Xin-Shun XuarXiv 2026 · arXiv

TL;DR

TAP (Tapping the Discriminative Potential) is a lightweight framework for class-incremental learning that achieves state-of-the-art performance by keeping pre-trained CLIP encoders completely frozen. Instead of risky fine-tuning, it combines robust visual distribution modeling with class-adaptive text template weighting to achieve superior accuracy with a fraction of the training time and memory of existing methods.

What Problem Does This Paper Address?

Class-Incremental Learning (CIL) is the challenge of teaching a model new classes over time without forgetting the old ones. Most modern approaches try to adapt large pre-trained models to new tasks, but this often leads to "catastrophic forgetting"—where updates for new classes disrupt the feature structures of old ones. Furthermore, fine-tuning these massive models is computationally expensive.

While some researchers have tried using "frozen" features to avoid these issues, they often rely on simple class-mean prototypes, which fail to capture the complex variations within a class. Others use text-based classification (like CLIP), but find that generic text templates (e.g., "a photo of a [CLASS]") don't work equally well for every category. TAP addresses these gaps by more fully exploiting the visual and semantic information already present in frozen representations.

Methods

TAP introduces three interconnected modules that work together without ever updating the underlying image or text encoders:

  1. Class-Conditional Visual Distribution Modeling: Instead of just storing a single mean vector per class, TAP estimates Gaussian distributions. It uses a "shrinkage" technique to blend class-specific covariances with a shared global covariance, which stabilizes the estimation when training data is limited.
  2. Class-Adaptive Semantic Calibration: Rather than using a single text prompt, TAP maintains a pool of templates. It learns class-specific weights to combine these templates, effectively "calibrating" the text representation to better match the visual features of a specific downstream class.
  3. Visual-Semantic Posterior Fusion: This is the "Product of Experts" (PoE) step. It takes the probability distributions from the visual expert (the Gaussian models) and the semantic expert (the calibrated CLIP matching) and fuses them. This allows the model to make a final prediction that benefits from both statistical visual patterns and high-level linguistic concepts.
Figure 1. Overview of TAP: (a) visual distribution modeling estimates class means and regularized covariances; (b) semantic calibration learns class-specific template weights; and (c) posterior fusion combines visual and semantic predictions. Both CLIP encoders remain frozen. Surfaces illustrate class densities.

Datasets

The authors evaluated TAP across a diverse set of nine benchmarks to ensure its robustness:

  • General/Natural Images: ImageNet-R, CIFAR-100, ObjectNet.
  • Fine-grained Classification: CUB-200 (birds), Aircraft, Cars, Food-101.
  • Scene Recognition: SUN-397.
  • Action Recognition: UCF-101.

These were tested under two protocols: B0/Inc (starting from zero and adding classes) and B50/B100/B150 (starting with a large base of classes and then adding more incrementally).

Key Insights and Ideas

  • Frozen is Better: The paper demonstrates that you don't need to update the weights of a strong pre-trained model like CLIP to get SOTA results in CIL. In fact, keeping it frozen prevents the "drift" that causes forgetting.
  • Covariance Matters: Moving beyond simple "mean" prototypes to full Gaussian distributions significantly improves the model's ability to handle within-class variation.
  • Experts are Complementary: One of the most interesting findings is that the visual and semantic experts often fail in different ways. By fusing them, TAP can correct errors that neither expert could solve individually. As shown in the figure below, the "net gain" from fusion increases as the model encounters more classes.
Figure 3. Accuracy gains and losses from posterior fusion on ImageNet-R B0. Bars show corrected and introduced errors; the curve shows the net gain.

Key Results

TAP set a new high-water mark for exemplar-free CIL:

  • State-of-the-Art Accuracy: TAP achieved the highest average incremental and final-stage accuracies across all 18 evaluated dataset-protocol settings. For example, on ImageNet-R (B0/Inc20), it reached 88.06% average accuracy, outperforming the next best method (HASTEN) by over 2 percentage points.
  • Extreme Efficiency: Because it only optimizes a small vector of template weights, TAP is incredibly fast. It completed training on ImageNet-R in just 161.3 seconds—roughly 15x faster than the ENGINE method and 10x faster than BOFA.
  • Low Resource Footprint: TAP used significantly less GPU memory (2181.5 MiB) compared to competitors like ENGINE (5312.9 MiB), making it much more accessible for researchers with limited hardware.

Limitations and Future Work

While TAP is highly effective, the authors and the methodology suggest a few areas for further exploration:

  • Reliance on CLIP: The performance is inherently tied to the quality of the pre-trained model. While CLIP is excellent, TAP's effectiveness might vary if used with less robust foundation models.
  • Gaussian Assumption: The visual modeling assumes class features follow a Gaussian distribution. While this is a common and often effective simplification, it may not perfectly capture highly non-linear or multi-modal class structures in the feature space.
  • Template Pool: The semantic calibration relies on a predefined pool of text templates. Future work could explore automatically generating or discovering more diverse templates to further improve the semantic expert.

Summary updated

Abstract

Pre-trained models provide transferable representations for class-incremental learning (CIL). However, sequentially adapting these representations introduces optimization costs and may disrupt old-class feature structure. This motivates us to explore whether the discriminative potential of these representations can be further exploited by continually accumulating class information while keeping the encoders frozen. To this end, we propose TAP, a new and novel framework for tapping the discriminative potential of frozen representations in class-incremental learning. It accumulates visual and semantic class information in the frozen CLIP embedding space with only lightweight optimization. TAP comprises three modules: class-conditional visual distribution modeling, class-adaptive semantic calibration, and visual-semantic posterior fusion. The first module retains class means and within-class scatters, combining class-specific and shared covariances to stabilize distribution estimation. The second module learns class-specific weights over text templates from current-task images, adapting text representations to downstream classes. The third module combines the visual and semantic posteriors through a weighted product of experts for prediction over all seen classes without retaining images from previous tasks. Across nine datasets under two incremental protocols, TAP achieves state-of-the-art performance, attaining the highest average incremental and final-stage accuracies among the compared methods. The code is provided as an ancillary file.
BibTeX
@misc{wang2026tap,
  author = {Zihao Wang and Yu-Wei Zhan and Xiao-Qian Liu and Zhen-Duo Chen and Weitong Liu and Xin Luo and Xin-Shun Xu},
  title = {{TAP: Tapping the Discriminative Potential of Frozen Representations for Class-Incremental Learning}},
  year = {2026},
  eprint = {2611.25478},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25478},
}
Figure