Test-Time Adaptation (TTA) allows a pre-trained model to adjust to new, unseen environments—like a self-driving car encountering sudden fog—without needing the original training data. For years, the "secret sauce" of TTA was Batch Normalization (BN). By simply updating the running statistics of BN layers, researchers like Muhammad Jehanzeb Mirza and Horst Possegger showed that models could adapt to domain shifts on the fly with minimal computation.
However, the field is currently undergoing a major shift. Modern "foundation" models, such as Vision Transformers (ViTs) and Large Language Models (LLMs), have largely abandoned Batch Normalization in favor of Layer Normalization (LN). This architectural change has broken many classic TTA methods, forcing researchers to find new "knobs" to turn during inference.
The Normalization Dependency
The success of early TTA methods like DUA (The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalization) and AugBN (SITA: Single Image Test-time Adaptation) relied on a specific property of Batch Normalization: it captures the "style" or distribution of a dataset within its mean and variance parameters. When a model encounters a new domain, these statistics shift. By recalibrating them, the model effectively "re-centers" its features to match the new environment.
As noted by Yadan Luo and Liang Zheng, this dependency creates a compatibility gap. Layer Normalization, used in Transformers, operates on individual samples rather than batches. It does not inherently store the cross-sample statistical links that methods like BATS (Boosting Out-of-distribution Detection with Typical Features) use to flag out-of-distribution data. Consequently, a method designed for a ResNet often fails when applied to a ViT.
Moving Toward Architecture Agnostic Adaptation
To bridge this gap, researchers are moving away from layer-specific hacks toward more universal mechanisms.
1. Feature Alignment and Memory Banks
Instead of just updating normalization statistics, newer frameworks like UniDG (Towards Unified and Effective Domain Generalization) by Yiyuan Zhang and Kaixiong Gong use differentiable memory banks. These banks store "source" features, allowing the model to align its current "target" features with known good representations, regardless of whether the model uses BN or LN. Similarly, CFA (Robustifying Vision Transformer without Retraining from Scratch by Test-Time Class-Conditional Feature Alignment) by Takeshi Kojima aligns internal feature distributions directly, making it one of the first methods to effectively robustify Transformers without retraining.
2. Visual Prompting
Perhaps the most radical departure from normalization-based TTA is OT-VP (OT-VP: Optimal Transport-guided Visual Prompting for Test-Time Adaptation). Instead of modifying the model's weights or statistics, this approach optimizes a few "input tokens" (prompts) to minimize the distance between the source and target distributions. This makes the adaptation entirely architecture-agnostic; as long as the model accepts input, it can be adapted.
3. Beyond Normalization Layers
Researchers are also exploring other parts of the network as adaptation targets. Jingjie Guo and Matthew Sinclair demonstrated in AdaAtlas (Pay Attention to the Atlas) that adapting attention mechanisms provides more "degrees of freedom" than normalization layers alone, which is crucial for complex tasks like 3D medical image segmentation.
Why This Works: The Intuition of Flexibility
The core intuition behind this shift is expressivity. Batch Normalization was a convenient, low-dimensional proxy for a domain. However, as Robert A. Marsden and Mario Döbler argue in their work on ROID (Universal Test-time Adaptation through Weight Ensembling), relying solely on normalization can lead to "model collapse" when the data stream is non-random or temporally correlated.
By moving toward weight ensembling, visual prompting, and attention-based updates, the research community is developing TTA methods that are not just "patches" for specific architectures, but robust frameworks capable of maintaining model stability across the diverse landscape of modern neural networks. As Jiachen Zhu and Kaiming He explore in Transformers without Normalization (Transformers without Normalization), the very necessity of normalization is being questioned, further accelerating the need for TTA methods that do not depend on it.