Abstract
Test Time Adaptation (TTA) has garnered significant attention in recent years, with the research focus on addressing distribution shifts during test time. As one fundamental component of many TTA methods, the Batch Normalization (BN) layer plays a crucial role in enabling the model adaptability. However, existing BN strategies can prove detrimental when the batch size is (extremely) small. In numerous real-world scenarios, limited hardware resources or just-in-time demand often necessitates adjusting models with very small batch sizes, making existing methods less practical. In this paper, we first showcase and thoroughly analyze the negative transfer phenomenon in previous TTA methods encountering extremely small batch sizes. Subsequently, we propose a novel batch size-agnostic method called NanoAdapt to effectively mitigate the negative transfer even with batch size 1. NanoAdapt is composed of three key components: a dynamic BN calibration strategy that leverages historical information and the Taylor series to refine the statistics estimations, an entropy-weighted gradient accumulation strategy that uses the entropy of each sample's label prediction to weigh and accumulate the loss for backpropagation, and a novel proxy computation graph to capture the sample interactions. Extensive experiments are conducted to validate the superiority of NanoAdapt, showing its consistent efficacy in improving existing TTA methods.
The Challenge of Small-Batch Adaptation
Deep learning models often struggle when deployment data differs from training data—a phenomenon known as distribution shift (e.g., a model trained on clear photos encountering blurry or snowy images). Test-Time Adaptation (TTA) aims to fix this by updating the model on-the-fly using unlabeled test data.
Most TTA methods rely heavily on Batch Normalization (BN) layers, which require a sufficiently large batch of images to calculate accurate statistics (mean and variance). However, in many real-world applications—like mobile devices or real-time robotics—data arrives one sample at a time, or hardware limits batch sizes to $N=1$. In these "extremely small batch" scenarios, BN statistics become wildly inaccurate, leading to negative transfer. This phenomenon causes the adaptation process to destroy the model's performance, often resulting in error rates near 100%.
The NanoAdapt Framework
To stabilize adaptation in resource-constrained environments, we propose NanoAdapt, a batch-size-agnostic method consisting of three main components:
- Dynamic BN Calibration (DBC): Instead of relying on a single noisy sample, DBC maintains a running estimate of statistics using an Exponential Moving Average (EMA). Because model weights change during adaptation, NanoAdapt uses a Taylor series expansion to "correct" historical statistics, ensuring they remain accurate relative to the most recent weights.
- Entropy-Weighted Gradient Accumulation (EWGA): Not all test samples are equally reliable. EWGA uses the entropy of a model’s prediction as a proxy for reliability, assigning higher weights to low-entropy (more certain) samples to reduce noise during backpropagation.
- Proxy Computation Graph (PCG): Mathematical analysis shows that at $N=1$, the "interaction terms" between samples in the gradient calculation disappear. PCG addresses this by creating multiple augmented views of a single sample to simulate a larger batch, restoring the necessary sample interactions for stable optimization.
Experimental Evaluation
NanoAdapt was evaluated on standard distribution shift benchmarks, including CIFAR-10-C, CIFAR-100-C, and ImageNet-C, focusing on severe corruption levels.
The results demonstrate that NanoAdapt effectively rescues TTA from total failure at small batch sizes. On CIFAR-100-C with $N=1$, the popular TTA method "Tent" suffers a 99% error rate (complete collapse). Adding NanoAdapt brings that error down to 40.05%, which is significantly better than the unadapted "Source" model (46.76%).
Conclusion
NanoAdapt provides a robust solution for making Test-Time Adaptation practical for real-world, edge-computing scenarios where batch sizes are extremely limited. By addressing the mathematical roots of model collapse at $N=1$, it enables reliable adaptation without requiring source training data or model retraining. Future work will explore extending these principles to other normalization layers beyond BN.
Summary updated