TL;DR
When fine-tuning Vision Transformers on continuous data streams, storing full images in memory buffers wastes significant capacity on background pixels. Core Tokensets compress replay memory at the sub-image level by selecting the most informative samples and saving only their most salient tokens via feature attribution. This simple shift allows a memory buffer containing just 1% of the data to match or exceed the performance of traditional replay coresets that are 2× to 10× larger.
What Problem Does This Paper Address?
Sequential or continual training often causes deep models to suffer from catastrophic forgetting—erasing previously acquired knowledge as new tasks are learned. The go-to defense is experience replay using coresets, which are small, carefully chosen subsets of past training instances.
However, standard coresets operate instance-by-instance: an entire image is either stored or discarded. In vision, the critical subject frequently occupies only a fraction of an image, leaving much of the memory budget wasted on uninformative background patches. Because Vision Transformers (ViTs) divide images into discrete sequences of patch tokens, the authors ask: Why store whole images when we can store only the most informative tokens?
Methods
The authors propose a two-stage data summarization pipeline called Core Tokensets (CTS):
- Instance-Level Selection (Coresets): The framework first identifies representative training samples using gradient-matching algorithms (e.g., GradMatch or CRAIG).
- Token-Level Attribution & Filtering: For every selected instance, feature attribution methods—such as Layer-wise Relevance Propagation (GradLRP) or attention perturbation (AtMan)—rank the relevance of each token. Only the top-scoring tokens are retained, alongside their original spatial positions.
- Adapting Transformers to Partial Inputs: Directly feeding sparse tokens into standard ViTs degrades accuracy because the architecture expects a complete grid of patch embeddings. To resolve this:
- Missing tokens in the replay buffer are zero-padded to retain positional integrity.
- During training on incoming tasks, random patch dropout (Bernoulli masking) is applied to full images, training the network to remain robust when processing incomplete inputs.
Incoming Stream ──► Coreset Selection ──► Attribution / Token Pruning ──► Zero-Padded Token Replay Buffer
│ │
▼ ▼
Full Images + Input Dropout ─────────────────────────────► Joint Training Step
Datasets
The authors evaluate Core Tokensets across three sequential domains:
- Sequential Image Classification: ImageNet-100 (split into 5 tasks of 20 classes each), ImageNet-Rendition (10- and 20-task splits), and SKILL-102 (a 9-task benchmark including domains like OfficeHome Clipart and Facial Expressions).
- Continual Visual Question Answering (VQA): Sequentially fine-tuning a multimodal BLIP model across VQA-v2 $\to$ CLEVR $\to$ Visual Genome (VG), evaluated via open-ended generation.
- Sequential Image Captioning: MS COCO split into 5 category-based tasks, evaluated using BLEU-4 and ROUGE metrics.
Key Insights and Ideas
- Tokens beat full instances under tight budgets: Discarding 90% of the tokens across high-value samples preserves far more predictive signal than discarding 90% of entire images.
- The "Cascaded" Advantage: With a 1% total data budget, standard coresets retain only 1 out of 100 images. Core tokensets can instead retain the top 10% most relevant images and the top 10% most critical tokens within them, capturing richer cross-sample diversity.
- Transformers require structural conditioning: Dropping patches without adaptation causes an immediate ~15% performance drop. Simple zero-padding combined with input dropout during new-task training fully recovers performance without altering the backbone architecture.
- Attribution choice matters: Multi-layer, gradient-aware (GradLRP) and perturbation-based (AtMan) attribution strategies substantially outperform simple Grad-CAM and random token masking.
Key Results
- Image Classification (ImageNet-100): At an aggressive 1% retention rate, GradMatch + AtMan Core Tokensets achieve 73.63% accuracy, outperforming full-image GradMatch at 1% (71.67%) and matching full-image GradMatch at 10% retention (74.21%)—effectively delivering a ~10× memory reduction.
- Visual Question Answering: On sequential VQA with BLIP, CTS at 1% retention reaches 71.98%, outperforming a 10× larger GradMatch coreset buffer (71.77% at 10% data) and approaching the 20% coreset mark.
- Benchmark Contextualization: On ImageNet-R (10 tasks), Core Tokensets achieve 81.32% average accuracy, proving competitive with parameter-efficient continual learning baselines such as InfLoRA (80.82%). On SKILL-102, CTS achieves 90.71%, outperforming CLR (89.92%).
Limitations and Future Work
- Predetermined Budgets: The token retention rate must be set manually beforehand rather than dynamically adapted per image complexity.
- Two-Stage Latency Overhead: Decoupling sample selection from token attribution requires additional computation (forward/backward passes for attribution). Developing a unified, single-stage gradient selection objective that simultaneously chooses instances and tokens is an open problem.
- Synergy with Token Pruning: Connecting core tokensets with dynamic token pruning and merging techniques used for inference acceleration represents a promising research avenue.
Summary updated