Core Tokensets for Data-efficient Sequential Training of Transformers

Subarnaduti Paul, Manuel Brack, Patrick Schramowski, Kristian Kersting, Martin MundtSubmission to ICLR 2025 · Rejected

TL;DR

When fine-tuning Vision Transformers on continuous data streams, storing full images in memory buffers wastes significant capacity on background pixels. Core Tokensets compress replay memory at the sub-image level by selecting the most informative samples and saving only their most salient tokens via feature attribution. This simple shift allows a memory buffer containing just 1% of the data to match or exceed the performance of traditional replay coresets that are 2× to 10× larger.

What Problem Does This Paper Address?

Sequential or continual training often causes deep models to suffer from catastrophic forgetting—erasing previously acquired knowledge as new tasks are learned. The go-to defense is experience replay using coresets, which are small, carefully chosen subsets of past training instances.

However, standard coresets operate instance-by-instance: an entire image is either stored or discarded. In vision, the critical subject frequently occupies only a fraction of an image, leaving much of the memory budget wasted on uninformative background patches. Because Vision Transformers (ViTs) divide images into discrete sequences of patch tokens, the authors ask: Why store whole images when we can store only the most informative tokens?

Methods

The authors propose a two-stage data summarization pipeline called Core Tokensets (CTS):

  1. Instance-Level Selection (Coresets): The framework first identifies representative training samples using gradient-matching algorithms (e.g., GradMatch or CRAIG).
  2. Token-Level Attribution & Filtering: For every selected instance, feature attribution methods—such as Layer-wise Relevance Propagation (GradLRP) or attention perturbation (AtMan)—rank the relevance of each token. Only the top-scoring tokens are retained, alongside their original spatial positions.
  3. Adapting Transformers to Partial Inputs: Directly feeding sparse tokens into standard ViTs degrades accuracy because the architecture expects a complete grid of patch embeddings. To resolve this:
  4. Missing tokens in the replay buffer are zero-padded to retain positional integrity.
  5. During training on incoming tasks, random patch dropout (Bernoulli masking) is applied to full images, training the network to remain robust when processing incomplete inputs.
Incoming Stream ──► Coreset Selection ──► Attribution / Token Pruning ──► Zero-Padded Token Replay Buffer
                          │                                                       │
                          ▼                                                       ▼
                Full Images + Input Dropout ─────────────────────────────► Joint Training Step

Datasets

The authors evaluate Core Tokensets across three sequential domains:

  • Sequential Image Classification: ImageNet-100 (split into 5 tasks of 20 classes each), ImageNet-Rendition (10- and 20-task splits), and SKILL-102 (a 9-task benchmark including domains like OfficeHome Clipart and Facial Expressions).
  • Continual Visual Question Answering (VQA): Sequentially fine-tuning a multimodal BLIP model across VQA-v2 $\to$ CLEVR $\to$ Visual Genome (VG), evaluated via open-ended generation.
  • Sequential Image Captioning: MS COCO split into 5 category-based tasks, evaluated using BLEU-4 and ROUGE metrics.

Key Insights and Ideas

  • Tokens beat full instances under tight budgets: Discarding 90% of the tokens across high-value samples preserves far more predictive signal than discarding 90% of entire images.
  • The "Cascaded" Advantage: With a 1% total data budget, standard coresets retain only 1 out of 100 images. Core tokensets can instead retain the top 10% most relevant images and the top 10% most critical tokens within them, capturing richer cross-sample diversity.
  • Transformers require structural conditioning: Dropping patches without adaptation causes an immediate ~15% performance drop. Simple zero-padding combined with input dropout during new-task training fully recovers performance without altering the backbone architecture.
  • Attribution choice matters: Multi-layer, gradient-aware (GradLRP) and perturbation-based (AtMan) attribution strategies substantially outperform simple Grad-CAM and random token masking.

Key Results

  • Image Classification (ImageNet-100): At an aggressive 1% retention rate, GradMatch + AtMan Core Tokensets achieve 73.63% accuracy, outperforming full-image GradMatch at 1% (71.67%) and matching full-image GradMatch at 10% retention (74.21%)—effectively delivering a ~10× memory reduction.
  • Visual Question Answering: On sequential VQA with BLIP, CTS at 1% retention reaches 71.98%, outperforming a 10× larger GradMatch coreset buffer (71.77% at 10% data) and approaching the 20% coreset mark.
  • Benchmark Contextualization: On ImageNet-R (10 tasks), Core Tokensets achieve 81.32% average accuracy, proving competitive with parameter-efficient continual learning baselines such as InfLoRA (80.82%). On SKILL-102, CTS achieves 90.71%, outperforming CLR (89.92%).

Limitations and Future Work

  • Predetermined Budgets: The token retention rate must be set manually beforehand rather than dynamically adapted per image complexity.
  • Two-Stage Latency Overhead: Decoupling sample selection from token attribution requires additional computation (forward/backward passes for attribution). Developing a unified, single-stage gradient selection objective that simultaneously chooses instances and tokens is an open problem.
  • Synergy with Token Pruning: Connecting core tokensets with dynamic token pruning and merging techniques used for inference acceleration represents a promising research avenue.

Summary updated

Abstract

Deep networks are frequently tuned to novel tasks and continue learning from ongoing data streams. Such sequential training requires consolidation of new and past information, a challenge predominantly addressed by retaining the most important data points - formally known as coresets. Traditionally, these coresets consist of entire samples, such as images or sentences. However, recent transformer architectures operate on tokens, leading to the famous assertion that an image is worth 16x16 words. Intuitively, not all of these tokens are equally informative or memorable. Going beyond coresets, we thus propose to construct a deeper-level data summary on the level of tokens. Ours, respectively named core tokensets, both select the most informative data points and leverage feature attribution to store only their most relevant features. We demonstrate that core tokensets yield significant performance retention in incremental image classification, open-ended visual question answering, and continual image captioning with significantly reduced memory. In fact, we empirically find that a core tokenset of 1\% of the data performs comparably to at least a twice as large and up to 10 times larger coreset.
BibTeX
@misc{paul2024core,
  author = {Subarnaduti Paul and Manuel Brack and Patrick Schramowski and Kristian Kersting and Martin Mundt},
  title = {{Core Tokensets for Data-efficient Sequential Training of Transformers}},
  year = {2024},
  eprint = {2410.05800},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.05800},
}
Figure