Introduction
Fine-tuning massive pre-trained models on downstream tasks is a cornerstone of modern machine learning. However, this process often suffers from "catastrophic forgetting," where the model discards general, robust features learned during pre-training to over-fit the specific nuances of a smaller dataset. To address this, researchers have traditionally turned to two distinct strategies: Sharpness-Aware Minimization (SAM), which seeks flat minima in the parameter space, and Trust Region (TR) methods, which regularize the model's functional representations.
The TRAM Framework
Trust Region Aware Minimization (TRAM) bridges these two approaches. While standard SAM uses a fixed-radius adversarial neighborhood ($\rho$) to find flat regions in the loss landscape, it does not explicitly account for the preservation of pre-trained representation structures. TRAM replaces this fixed neighborhood with a dynamic trust region bound ($d$) that is sensitive to changes in the model's internal representations.
By tying the SAM perturbation to a trust region metric—such as KL divergence from the pre-trained model ($\theta_0$) or the previous training step ($\theta_{t-1}$)—TRAM ensures that the "sharpness-seeking" optimization stays within a region where the model's functional behavior remains stable. This allows the optimizer to find solutions that are both parameter-wise flat and representation-wise smooth.
Experimental Evaluation
The authors evaluated TRAM across vision, cross-domain language modeling, and cross-lingual transfer tasks. In vision tasks using ViT-base models, TRAM consistently outperformed baselines like ASAM and FSAM. For instance, on the Oxford Flowers dataset, TRAM reached 97.07% accuracy, significantly higher than FSAM's 95.26%.
In cross-domain language modeling (M2D2 corpus), TRAM demonstrated exceptional robustness. As shown in the figure above, TRAM variants (stars) consistently achieve lower perplexity on out-of-distribution (OOD) tasks compared to standard optimizers like Adam or SAM. Notably, in "hard" transfer scenarios where training and test performance are anticorrelated (e.g., Art and Philosophy domains), TRAM effectively "bucks the trend," maintaining pre-trained generality while learning new domain-specific information.
Key Insights and Conclusion
TRAM's success stems from its ability to maintain high representation similarity to the pre-trained model, as verified by Centered Kernel Alignment (CKA) analysis. By making the optimizer "aware" of representation curvature, TRAM achieves lower out-of-domain sharpness than SAM itself.
While TRAM introduces some computational overhead due to additional forward passes, variants like TRAM-Fisher offer a more efficient alternative. Future work may explore different distance metrics beyond KL divergence to further refine the trust region boundaries. Ultimately, TRAM provides a powerful new tool for robust fine-tuning, ensuring that the wealth of knowledge in pre-trained models is preserved and effectively adapted.
Summary updated