TRACER: Persistent Regularization for Robust Multimodal Finetuning

Hesam Asadollahzadeh, Feng Liu, Christopher Leckie, Sarah M. Erfani

TL;DR

Finetuning pretrained vision-language models like CLIP often destroys their out-of-distribution (OOD) robustness—a dilemma known as catastrophic forgetting. This paper introduces TRACER, a finetuning framework that replaces standard Exponential Moving Average (EMA) teachers with a Weighted Moving Average (WMA) teacher. By anchoring the teacher to the optimization trajectory, TRACER maintains a persistent regularizing force that prevents model collapse and preserves robust, pretrained knowledge without sacrificing downstream performance.

What Problem Does This Paper Address?

When we finetune a foundation model on a specific downstream dataset, its in-distribution (ID) accuracy shoots up, but its ability to generalize to out-of-distribution (OOD) data typically plummets. The model essentially overwrites its broadly applicable pretrained representations to overfit to task-specific patterns and spurious correlations.

To combat this, researchers often use self-distillation with a dynamic "mean teacher" (typically updated via EMA) to keep the student model from straying too far from its roots. However, the authors identify a critical, overlooked flaw in EMA teachers: as training progresses and the student converges, the teacher-student gap rapidly collapses. This causes the regularizing force to vanish exactly when the model's OOD robustness is most vulnerable.

Methods

The authors first build a rigorous theoretical framework by reformulating the linearized multimodal contrastive learning loss into a matrix least-squares problem using what they call a Contrastive Target Matrix ($Y_{FT}$). This allows them to derive closed-form solutions for common finetuning strategies and geometrically map exactly where forgetting occurs.

To fix the vanishing regularization of EMA, they introduce TRACER (Trajectory-Robust Anchoring for Contrastive Encoder Regularization). TRACER's core innovation is the Weighted Moving Average (WMA) Teacher. Instead of exponentially decaying past states, the WMA teacher weights the student's entire optimization trajectory using a position-aware, U-shaped kernel (specifically, a $\text{Beta}(0.5, 0.5)$ arcsine distribution). This ensures that the robust, early pretrained states retain a non-trivial weight, providing a continuous "restoring force" throughout finite training horizons.

Overview of the TRACER framework combining base contrastive learning with WMA-guided self-distillation.

TRACER operationalizes this via a multi-perspective distillation loss that aligns the student and teacher across four dimensions:

  1. Feature Distillation (FD): Directly aligns embedding spaces.
  2. Contrastive Relational Distillation (CRD): Matches batch-wise similarity distributions.
  3. Interactive Contrastive Learning (ICL): Cross-modal alignment between student and teacher.
  4. Cross Knowledge Distillation (Cross-KD): Aligns cross-modal logits to transfer relational structure.

Datasets

The authors validated their approach using both synthetic and large-scale real-world benchmarks:

  • Synthetic: A multimodal variant of MNIST paired with a "Colored MNIST" finetuning task designed to introduce severe spurious color correlations.
  • Real-World Evaluation: ImageNet-1K served as the in-distribution (ID) downstream task.
  • OOD Robustness Suite: Five distinct distribution shifts were evaluated: ImageNet-V2, ImageNet-Rendition (IN-R), ImageNet-Adversarial (IN-A), ImageNet-Sketch (IN-S), and ObjectNet.

Key Insights and Ideas

  • The Geometry of Self-Distillation: The paper proves that self-distillation operates with surgical precision—it acts as an identity function outside the task subspace (preserving orthogonal pretrained knowledge) while forming a convex combination of old and new knowledge inside the task subspace.
  • The Failure of EMA: Standard EMA teachers suffer from an inherent learning signal collapse. Because the teacher tracks the student too closely, the KL divergence between them decays steadily to zero, stripping away OOD protection late in training.
  • The Power of U-Shaped Weighting: By using an arcsine-like Beta kernel, TRACER heavily weights both the initialization and the latest iterates. This eliminates the static anchor bias of traditional self-distillation while ensuring the teacher remains safely separated from the student's immediate state.
Geometric interpretation of how different finetuning strategies modify pretrained weights.

Key Results

  • Spurious Correlation Mitigation: In the Colored MNIST toy task, standard direct finetuning suffered a massive forgetting rate of 37.9% on the original task. TRACER virtually eliminated this, dropping the forgetting rate to a mere 0.1% while still mastering the new task.
  • State-of-the-Art OOD Accuracy: On CLIP ViT-B/16, TRACER achieved an average OOD accuracy of 64.07%, outperforming direct finetuning (57.08%) and strong recent baselines like CaRot (62.54%). The gains were most pronounced on severe adversarial shifts like ImageNet-A, where TRACER beat static self-distillation by +12.4%.
  • Representational Fidelity: Layer-wise similarity analysis (using CKA and SVCCA) showed that while standard finetuning causes deeper layers to drift drastically away from pretrained geometries, TRACER maintains a near-perfect similarity score ($>0.97$) across all layers.
  • Computational Efficiency: Unlike methods that rely on expensive spectral regularization scaling at $\mathcal{O}(d^3)$, TRACER operates on batch similarity matrices ($\mathcal{O}(B^2)$). This reduced per-epoch training time on an NVIDIA H100 from 29 minutes (CaRot) to 22 minutes.

Limitations and Future Work

While highly effective, the authors note a few boundaries to their current work:

  • Scope of Models: The empirical evaluation focuses strictly on CLIP-style vision-language backbones. TRACER has not yet been tested on broader modalities or massive autoregressive multimodal LLMs.
  • Linearized Theory: The mathematical proofs for bias-free convergence are grounded in a linearized encoder regime.

Future Directions: The authors propose lifting their theoretical framework into the Neural Tangent Kernel (NTK) regime and random-feature models to better capture non-linear dynamics. They also highlight exploring how TRACER can be natively combined with parameter-efficient finetuning (PEFT) methods like LoRA.

Summary updated

Abstract

Mainstream strategies for finetuning pretrained multimodal models often degrade out-of-distribution (OOD) robustness, a phenomenon known as catastrophic forgetting. In this paper, we develop a theoretical framework for multimodal contrastive finetuning, yielding closed-form solutions and a geometric decomposition for each strategy. This framework shows that self-distillation is more effective than other regularization approaches to retain the knowledge of the pretrained model. Our analysis reveals a largely overlooked limitation: standard Exponential Moving Average (EMA) teachers, widely used in robust finetuning, suffer from collapse. To solve this, we prove that a Weighted Moving Average (WMA) teacher maintains a persistent regularizing force over finite horizons and yields bias-free convergence in the task subspace while preserving orthogonal knowledge. These insights motivate **TRACER** (**T**rajectory-**R**obust **A**nchoring for **C**ontrastive **E**ncoder **R**egularization), which combines contrastive learning with WMA-guided multi-perspective distillation. Extensive experiments on CLIP finetuning demonstrate consistent OOD accuracy and calibration gains across three backbone architectures, and comprehensive ablations confirm that TRACER is both principled and robust to hyperparameter choices. Code is available at [https://github.com/HesamAsad/TRACER](https://github.com/HesamAsad/TRACER).
BibTeX
@inproceedings{asadollahzadeh2026tracer,
  author = {Hesam Asadollahzadeh and Feng Liu and Christopher Leckie and Sarah M. Erfani},
  title = {{TRACER: Persistent Regularization for Robust Multimodal Finetuning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=XOYXLQRlj8},
}
Figure