[CLS] and [EOS] Tokens are not Enough for CLIP-based Class-Incremental Learning

Hao Sun, Tao Hu, Shixiong Kai, Siyuan Xu, Wenqian Zhao, Mingxuan Yuan, Da-Wei ZhouarXiv 2026 · arXiv

TL;DR

PACE (PAtch-token Cross-modal Enhancement) improves CLIP-based Class-Incremental Learning by moving beyond global [CLS] and [EOS] tokens to leverage fine-grained local features. By selectively fusing discriminative visual patches with textual tokens and employing a subspace projection strategy to protect learned weights, PACE achieves state-of-the-art performance across nine major benchmarks while effectively mitigating catastrophic forgetting.

What Problem Does This Paper Address?

Class-Incremental Learning (CIL) is the challenge of teaching a model new classes over time without it "forgetting" what it previously learned—a phenomenon known as catastrophic forgetting. Recently, researchers have turned to Vision-Language Models (VLMs) like CLIP because their pre-trained, cross-modal alignment provides a strong foundation for generalization.

However, most existing CLIP-based CIL methods rely solely on global representations: the visual [CLS] token and the textual [EOS] token. The authors argue that these global summaries are "not enough." While a global token might identify a "bird," it often misses the fine-grained local details—like the specific shape of a beak or the color of a wing patch—necessary to distinguish a sparrow from a robin. Furthermore, simply fine-tuning these models on new tasks often destroys the delicate cross-modal alignment learned previously.

Methods

The authors propose PACE, which consists of three main components illustrated in the framework overview:

Fig. 1: Illustration of PACE. Left: Patch-token information selection, where top-k discriminative patch-level visual features and token-level textual semantics are selected. Top Right: Cross-modal fusion, which utilizes a cross-attention fusion mechanism. Here, one modality’s global representation serves as the query, while the other’s local feature sequence serves as the key and value, thereby enhancing the cross-modal semantic representation of the global features. Bottom Right: Local knowledge preservation, which utilizes subspace projection to constrain shared weight updates, thus mitigating catastrophic forgetting.
  1. Patch-Token Information Selection: To avoid processing noisy background patches or "stop words" in text, PACE uses the global tokens as a guide. It computes the affinity between the global [CLS] token and all visual patches, selecting only the top-$k$ most discriminative ones. It does the same for textual tokens relative to the [EOS] token.
  2. Cross-Modal Fusion Module: Instead of keeping modalities separate until the final matching step, PACE introduces a cross-attention mechanism. The global visual feature acts as a query to retrieve relevant local textual semantics, and vice versa. This "injects" local knowledge into the global representations, making them more robust.
  3. Local Knowledge Preservation: To prevent forgetting, PACE uses a subspace projection-based update. After each task, it performs Singular Value Decomposition (SVD) on the shared fusion weights to identify the "critical directions" that encode previous knowledge. When training on new tasks, weight updates are constrained to be orthogonal to these preserved directions, ensuring new learning doesn't overwrite the old.

The authors also introduce PACE++, which replaces simple class prompts (e.g., "a photo of a [CLASS]") with detailed, LLM-generated visual descriptions to provide even richer local textual semantics for the fusion module.

Datasets

The model was evaluated on a comprehensive suite of nine benchmark datasets:

  • General/Large-scale: CIFAR100, ImageNet-R, ImageNet-100, and ImageNet-1K.
  • Fine-grained: CUB200 (birds), FGVCAircraft, and Stanford Cars.
  • Specialized: Food101, SUN397 (scenes), UCF101 (actions), and ObjectNet.

Key Insights and Ideas

  • Local Details are Discriminative: The "aha moment" is that while global tokens capture the concept, local patches capture the evidence. Fusing these explicitly allows the model to handle fine-grained increments much better than global-only methods.
  • Global-Local Synergy: Using global features as queries for local features is an efficient way to filter noise. It ensures the model attends to the "beak" of the bird rather than a random patch of "sky."
  • Subspace Stability: The use of SVD for weight protection highlights that not all parameter updates are equal. By identifying the specific subspace responsible for past performance, the model can remain "plastic" in other directions to learn new classes.

Key Results

PACE consistently outperforms state-of-the-art CIL methods like PROOF, RAPF, and SimpleCIL. As shown in the incremental accuracy plots, the performance gap between PACE and other methods actually widens as more tasks are added, demonstrating superior resistance to forgetting.

Incremental performance on Aircraft, CIFAR100, and Cars datasets.
  • Accuracy Gains: On the Aircraft dataset, PACE outperformed the runner-up by 2.65% in final accuracy. On the Cars dataset, the lead was 2.5%.
  • Efficiency: Despite using local features, PACE maintains a comparable number of trainable parameters to other PEFT (Parameter-Efficient Fine-Tuning) methods and offers a favorable trade-off between accuracy and computational overhead (FLOPs).

Limitations and Future Work

The authors note that their current framework still relies on a fixed prompt template for the textual branch (even if enhanced by an LLM). This might limit the model's ability to capture the full semantic diversity of certain complex CIL scenarios.

Future directions include exploring more dynamic prompt engineering strategies, such as learnable prompts, to further enhance the textual representation and its alignment with visual patches. Additionally, the authors suggest that investigating how to automatically determine the optimal number of local features ($k$) per sample, rather than using a fixed value, could yield further improvements.

Summary updated

Abstract

Class-Incremental Learning (CIL) aims to empower models to continuously acquire new knowledge while mitigating catastrophic forgetting. With the advent of vision-language models like CLIP, leveraging their cross-modal alignment capabilities to address CIL challenges has emerged as a dominant paradigm. Existing methods leverage CLIP's global representations (i.e., the visual [CLS] token and the textual [EOS] token) for cross-modal alignment. While achieving effective classification, they overlook the rich patch-level visual features and token-level textual semantics inherent in CLIP. For example, the [CLS] token aggregates local information to capture the general concept of "cat", while the patch-level details retain specific localized features essential for recognition, such as the tail, fur, and face. We propose PACE (PAtch-token Cross-modal Enhancement) to uncover the long-neglected local representations within CLIP. First, based on the global representations like [CLS] token and [EOS] token, we identify the top-k most crucial visual patches and textual tokens. Subsequently, to better utilize the extracted key local features, we design a cross-modal fusion module. This module leverages local textual semantics to refine visual representations, while augmenting textual semantics with local visual features. As the fusion module is shared across all tasks, directly fine-tuning it on new tasks would interfere with the knowledge learned from previous tasks, leading to catastrophic forgetting. To alleviate this, we employ a projection-based update strategy to effectively preserve the critical parameter space of previous tasks during the adaptation to new tasks. Extensive experiments demonstrate that PACE significantly outperforms state-of-the-art methods across multiple CIL benchmarks.
BibTeX
@misc{sun2026cls,
  author = {Hao Sun and Tao Hu and Shixiong Kai and Siyuan Xu and Wenqian Zhao and Mingxuan Yuan and Da-Wei Zhou},
  title = {{[CLS] and [EOS] Tokens are not Enough for CLIP-based Class-Incremental Learning}},
  year = {2026},
  eprint = {2611.21440},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21440},
}
Figure