Core Tokensets for Data-efficient Sequential Training of Transformers

Subarnaduti Paul, Manuel Brack, Patrick Schramowski, Kristian Kersting, Martin Mundt

Introduction

Sequential training, or continual learning, is a critical paradigm for deep learning models that must adapt to new information over time without losing previously acquired knowledge—a phenomenon known as "catastrophic forgetting." A common mitigation strategy involves maintaining a memory buffer of past data, known as a coreset, to be replayed during new task training. However, traditional coresets store entire data instances (e.g., full images), which often contain significant amounts of redundant information. In this work, we introduce Core Tokensets, a hierarchical data summarization approach that identifies and stores only the most informative tokens within the most representative samples, drastically reducing memory overhead while maintaining high performance.

Methodology: From Coresets to Core Tokensets

The proposed Core Tokenset approach moves beyond sample-level selection to feature-level selection within Transformer architectures. The process consists of three main stages:

  1. Sample Selection: Using established algorithms like GradMatch or CRAIG, we first identify a subset of the most informative data instances from the current task.
  2. Core Token Selection: Within these selected samples, we apply feature attribution techniques (such as Atman or GradLRP) to rank the importance of individual input tokens. By retaining only the top-ranked tokens, we create a sparse representation of the original data.
  3. Robustness via Random Token Drop: To ensure the Transformer can effectively process these sparse "partial" inputs during replay, we introduce a Random Token Drop strategy during the initial training phase. This regularizes the model to remain accurate even when significant portions of the input sequence are missing.
Overview of the Core Tokenset selection and training pipeline.

Experimental Evaluation

We evaluated Core Tokensets across three domains: Image Classification (ImageNet100), Visual Question Answering (VQA 2.0, CLEVR, Visual Genome), and Image Captioning (MS COCO).

Key Results

Our experiments reveal that Core Tokensets provide a superior trade-off between memory usage and model accuracy. A primary finding is the "1% is the new 10%" rule: a Core Tokenset with only 1% data retention achieves performance comparable to a traditional coreset that is ten times larger. On ImageNet100, our method achieved 73.63% accuracy with 1% retention, significantly outperforming random coresets (67.21%) at the same retention level.

Performance comparison across tasks and storage budgets. The right plot demonstrates that Core Tokensets consistently outperform traditional coresets and token-only selection across various retention rates.

Discussion and Insights

The success of Core Tokensets suggests that current memory buffers are highly redundant. By prioritizing "breadth over depth"—storing a few critical tokens from many diverse images rather than all tokens from a few images—we provide the model with a more comprehensive memory of past tasks. Furthermore, the necessity of the Random Token Drop strategy highlights that standard Transformers are inherently brittle to input sparsity; explicit training for robustness is required to leverage token-level pruning effectively.

Conclusion and Future Work

Core Tokensets represent a significant step toward memory-efficient continual learning, enabling the deployment of large-scale models on resource-constrained edge devices. Future research will focus on dynamic retention rates tailored to individual image complexity and the development of unified, single-stage selection strategies to reduce the computational overhead of the current two-step selection process.

Summary updated

Abstract

Deep networks are frequently tuned to novel tasks and continue learning from ongoing data streams. Such sequential training requires consolidation of new and past information, a challenge predominantly addressed by retaining the most important data points - formally known as coresets. Traditionally, these coresets consist of entire samples, such as images or sentences. However, recent transformer architectures operate on tokens, leading to the famous assertion that an image is worth 16x16 words. Intuitively, not all of these tokens are equally informative or memorable. Going beyond coresets, we thus propose to construct a deeper-level data summary on the level of tokens. Ours, respectively named core tokensets, both select the most informative data points and leverage feature attribution to store only their most relevant features. We demonstrate that core tokensets yield significant performance retention in incremental image classification, open-ended visual question answering, and continual image captioning with significantly reduced memory. In fact, we empirically find that a core tokenset of 1\% of the data performs comparably to at least a twice as large and up to 10 times larger coreset.
BibTeX
@misc{paul2024core,
  author = {Subarnaduti Paul and Manuel Brack and Patrick Schramowski and Kristian Kersting and Martin Mundt},
  title = {{Core Tokensets for Data-efficient Sequential Training of Transformers}},
  year = {2024},
  eprint = {2410.05800},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.05800},
}
Figure