TL;DR
Finetuning pretrained vision-language models like CLIP often destroys their out-of-distribution (OOD) robustness—a dilemma known as catastrophic forgetting. This paper introduces TRACER, a finetuning framework that replaces standard Exponential Moving Average (EMA) teachers with a Weighted Moving Average (WMA) teacher. By anchoring the teacher to the optimization trajectory, TRACER maintains a persistent regularizing force that prevents model collapse and preserves robust, pretrained knowledge without sacrificing downstream performance.
What Problem Does This Paper Address?
When we finetune a foundation model on a specific downstream dataset, its in-distribution (ID) accuracy shoots up, but its ability to generalize to out-of-distribution (OOD) data typically plummets. The model essentially overwrites its broadly applicable pretrained representations to overfit to task-specific patterns and spurious correlations.
To combat this, researchers often use self-distillation with a dynamic "mean teacher" (typically updated via EMA) to keep the student model from straying too far from its roots. However, the authors identify a critical, overlooked flaw in EMA teachers: as training progresses and the student converges, the teacher-student gap rapidly collapses. This causes the regularizing force to vanish exactly when the model's OOD robustness is most vulnerable.
Methods
The authors first build a rigorous theoretical framework by reformulating the linearized multimodal contrastive learning loss into a matrix least-squares problem using what they call a Contrastive Target Matrix ($Y_{FT}$). This allows them to derive closed-form solutions for common finetuning strategies and geometrically map exactly where forgetting occurs.
To fix the vanishing regularization of EMA, they introduce TRACER (Trajectory-Robust Anchoring for Contrastive Encoder Regularization). TRACER's core innovation is the Weighted Moving Average (WMA) Teacher. Instead of exponentially decaying past states, the WMA teacher weights the student's entire optimization trajectory using a position-aware, U-shaped kernel (specifically, a $\text{Beta}(0.5, 0.5)$ arcsine distribution). This ensures that the robust, early pretrained states retain a non-trivial weight, providing a continuous "restoring force" throughout finite training horizons.
TRACER operationalizes this via a multi-perspective distillation loss that aligns the student and teacher across four dimensions:
- Feature Distillation (FD): Directly aligns embedding spaces.
- Contrastive Relational Distillation (CRD): Matches batch-wise similarity distributions.
- Interactive Contrastive Learning (ICL): Cross-modal alignment between student and teacher.
- Cross Knowledge Distillation (Cross-KD): Aligns cross-modal logits to transfer relational structure.
Datasets
The authors validated their approach using both synthetic and large-scale real-world benchmarks:
- Synthetic: A multimodal variant of MNIST paired with a "Colored MNIST" finetuning task designed to introduce severe spurious color correlations.
- Real-World Evaluation: ImageNet-1K served as the in-distribution (ID) downstream task.
- OOD Robustness Suite: Five distinct distribution shifts were evaluated: ImageNet-V2, ImageNet-Rendition (IN-R), ImageNet-Adversarial (IN-A), ImageNet-Sketch (IN-S), and ObjectNet.
Key Insights and Ideas
- The Geometry of Self-Distillation: The paper proves that self-distillation operates with surgical precision—it acts as an identity function outside the task subspace (preserving orthogonal pretrained knowledge) while forming a convex combination of old and new knowledge inside the task subspace.
- The Failure of EMA: Standard EMA teachers suffer from an inherent learning signal collapse. Because the teacher tracks the student too closely, the KL divergence between them decays steadily to zero, stripping away OOD protection late in training.
- The Power of U-Shaped Weighting: By using an arcsine-like Beta kernel, TRACER heavily weights both the initialization and the latest iterates. This eliminates the static anchor bias of traditional self-distillation while ensuring the teacher remains safely separated from the student's immediate state.
Key Results
- Spurious Correlation Mitigation: In the Colored MNIST toy task, standard direct finetuning suffered a massive forgetting rate of 37.9% on the original task. TRACER virtually eliminated this, dropping the forgetting rate to a mere 0.1% while still mastering the new task.
- State-of-the-Art OOD Accuracy: On CLIP ViT-B/16, TRACER achieved an average OOD accuracy of 64.07%, outperforming direct finetuning (57.08%) and strong recent baselines like CaRot (62.54%). The gains were most pronounced on severe adversarial shifts like ImageNet-A, where TRACER beat static self-distillation by +12.4%.
- Representational Fidelity: Layer-wise similarity analysis (using CKA and SVCCA) showed that while standard finetuning causes deeper layers to drift drastically away from pretrained geometries, TRACER maintains a near-perfect similarity score ($>0.97$) across all layers.
- Computational Efficiency: Unlike methods that rely on expensive spectral regularization scaling at $\mathcal{O}(d^3)$, TRACER operates on batch similarity matrices ($\mathcal{O}(B^2)$). This reduced per-epoch training time on an NVIDIA H100 from 29 minutes (CaRot) to 22 minutes.
Limitations and Future Work
While highly effective, the authors note a few boundaries to their current work:
- Scope of Models: The empirical evaluation focuses strictly on CLIP-style vision-language backbones. TRACER has not yet been tested on broader modalities or massive autoregressive multimodal LLMs.
- Linearized Theory: The mathematical proofs for bias-free convergence are grounded in a linearized encoder regime.
Future Directions: The authors propose lifting their theoretical framework into the Neural Tangent Kernel (NTK) regime and random-feature models to better capture non-linear dynamics. They also highlight exploring how TRACER can be natively combined with parameter-efficient finetuning (PEFT) methods like LoRA.
Summary updated