TRACER: Persistent Regularization for Robust Multimodal Finetuning

Hesam Asadollahzadeh, Feng Liu, Christopher Leckie, Sarah Monazam Erfani

Finetuning a massive pretrained model like CLIP is akin to sending a polymath to a specialized trade school. While the model will certainly master the new domain (in-distribution accuracy), it often begins to forget the broad, general-purpose knowledge that made it so versatile in the first place. This phenomenon, known as catastrophic forgetting, results in a severe degradation of out-of-distribution (OOD) robustness.

To address this, researchers have introduced TRACER (Trajectory-Robust Anchoring for Contrastive Encoder Regularization), a novel framework that fundamentally rethinks how models are regularized during finetuning to prevent over-specialization.

The Problem: The "Robustness Trade-off" and EMA Collapse

When multimodal models are finetuned on a downstream task like ImageNet, they face a robustness trade-off. They become highly accurate on the specific task but fragile when encountering data that deviates slightly from the training set—such as sketches, adversarial examples, or unusual lighting conditions.

Existing solutions often employ a "teacher-student" architecture to guide the model. A standard approach uses an Exponential Moving Average (EMA) teacher, which maintains a running average of the student model's weights. However, the authors identify a critical and largely overlooked flaw in this approach: EMA collapse. As the student model converges on the target task, the EMA teacher eventually becomes identical to the student. Consequently, the regularizing "teaching signal" vanishes exactly when the model is most vulnerable to overfitting, leading to a loss of OOD robustness.

The TRACER Methodology

To overcome EMA collapse, the authors developed a theoretical framework that reformulates contrastive finetuning as a matrix least-squares problem. This geometric decomposition splits the finetuning process into two distinct parts:

  1. Task Subspace Mixing: Where the model adapts to learn the new task.
  2. Orthogonal Preservation: Where the model retains its original, general-purpose knowledge.

The Weighted Moving Average (WMA) Teacher

The core innovation of TRACER is replacing the EMA teacher with a Weighted Moving Average (WMA) teacher. Unlike EMA, which heavily prioritizes the most recent training steps, WMA utilizes a Beta kernel (an arcsine-like distribution) to weight the entire training trajectory.

This WMA teacher acts as a persistent "restoring force" or tether. Because it remembers the early, robust states of the model, it exerts a continuous pull that prevents the student from drifting too far into task-specific overfitting.

Multi-Perspective Distillation

TRACER combines this WMA teacher with multi-perspective distillation to ensure comprehensive knowledge retention:

  • Feature Distillation (FD): Aligns raw embeddings between teacher and student.
  • Contrastive Relational Distillation (CRD): Preserves the structural relationships between different samples in a batch.
  • Interactive Contrastive Learning (ICL): Mixes student and teacher features to enrich the learning signal.
  • Cross Knowledge Distillation (Cross-KD): Aligns cross-modal logits.

Experimental Setup

The researchers evaluated TRACER using ImageNet-1K as the in-distribution (ID) task. To rigorously test OOD robustness, they utilized a suite of five challenging benchmarks:

  • ImageNet-V2: New test sets for ImageNet.
  • ImageNet-R (Rendition): Art, cartoons, and sketches.
  • ImageNet-A (Adversarial): Naturally occurring "hard" examples.
  • ImageNet-S (Sketch): Black and white sketches.
  • ObjectNet: Real-world objects in unusual poses and backgrounds.

Additionally, a synthetic ColoredMNIST dataset was used to validate the theoretical framework in a controlled environment with spurious correlations.

Key Findings

TRACER demonstrates that self-distillation can be "surgical"—allowing the model to adapt within the necessary task subspace while leaving orthogonal pretrained knowledge untouched. Key empirical results include:

  • Superior Robustness: On a CLIP ViT-B/16 backbone, TRACER achieved an average OOD accuracy of 64.07%. This significantly outperforms both the zero-shot baseline (58.39%) and the previous state-of-the-art method, CaRot (62.54%).
  • Better Calibration: TRACER consistently achieved the lowest Expected Calibration Error (ECE). This means the model's confidence scores are more reliable, making it less likely to be confidently incorrect.
  • Simplicity and Stability: Unlike previous methods that require complex, sparse update schedules to prevent collapse, TRACER is highly stable across various hyperparameter choices. It performs well whether the teacher is updated every step or once every 2,500 steps.
  • Computational Efficiency: TRACER adds only a ~38% computational overhead compared to direct finetuning. In contrast, competing robust methods (like spectral-based regularization) can nearly double training time.

Limitations and Future Directions

The authors note that their theoretical derivations rely on a linearized version of the encoders. While empirical results on non-linear Transformers confirm the theory holds in practice, future work utilizing more complex analyses (such as Neural Tangent Kernels) could bridge this theoretical gap.

Currently, TRACER is optimized for CLIP-style vision-language models. Extending this framework to larger Multimodal Large Language Models (MLLMs) and exploring its compatibility with Parameter-Efficient Finetuning (PEFT) techniques, such as LoRA, represent promising avenues for future research.

Summary updated

Abstract

Mainstream strategies for finetuning pretrained multimodal models often degrade out-of-distribution (OOD) robustness, a phenomenon known as catastrophic forgetting. In this paper, we develop a theoretical framework for multimodal contrastive finetuning, yielding closed-form solutions and a geometric decomposition for these strategies. This framework shows that self-distillation is more effective than other regularization approaches to retain the knowledge of the pretrained model. Our analysis reveals a largely overlooked limitation: standard Exponential Moving Average (EMA) teachers, widely used in robust finetuning, suffer from collapse. To solve this, we prove that a Weighted Moving Average (WMA) teacher maintains a persistent regularizing force over finite horizons and yields bias-free convergence in the task subspace while preserving orthogonal knowledge. These insights motivate **TRACER** (**T**rajectory-**R**obust **A**nchoring for **C**ontrastive **E**ncoder **R**egularization), which combines contrastive learning with WMA-guided multi-perspective distillation. Extensive experiments on CLIP finetuning demonstrate consistent OOD accuracy and calibration gains across three backbone architectures, and comprehensive ablations confirm that TRACER is both principled and robust to hyperparameter choices. Code is available at [https://github.com/HesamAsad/TRACER](https://github.com/HesamAsad/TRACER).
BibTeX
@inproceedings{asadollahzadeh2026tracer,
  author = {Hesam Asadollahzadeh and Feng Liu and Christopher Leckie and Sarah Monazam Erfani},
  title = {{TRACER: Persistent Regularization for Robust Multimodal Finetuning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=XOYXLQRlj8},
}
Figure