Abstract
Test Time Adaptation (TTA) is a powerful paradigm for adjusting pre-trained models to new data distributions on the fly. However, traditional TTA methods often suffer from "negative transfer" when batch sizes are extremely small (e.g., $N=1$), leading to catastrophic model collapse. We introduce NanoAdapt, a framework designed to stabilize TTA in resource-constrained or real-time scenarios. NanoAdapt combines Taylor-series-based Batch Normalization (BN) calibration, entropy-weighted gradients, and a "proxy" computation graph that simulates sample interactions. Our evaluations on CIFAR and ImageNet benchmarks demonstrate that NanoAdapt maintains robust performance even at $N=1$, where baseline methods like TENT fail.
Introduction: The Small Batch Challenge
Test Time Adaptation (TTA) allows models to adapt to distribution shifts—such as noise, blur, or weather changes—without requiring original training labels. Most TTA methods rely on Batch Normalization (BN) layers to estimate the statistics of the new data.
However, BN requires a sufficiently large batch to calculate accurate mean and variance. In edge computing scenarios, data often arrives one sample at a time ($N=1$). In these cases, BN statistics become wildly inaccurate, and the resulting gradients become noisy, leading to a performance collapse where error rates can approach 100%. NanoAdapt is designed specifically to bridge this gap and enable reliable adaptation at the edge.
Methods: Stabilizing Adaptation at the Edge
NanoAdapt addresses the instability of small-batch TTA through three core synergistic components:
- Dynamic BN Calibration (DBC): Standard moving averages fail when model weights are updated at every step. DBC uses a Taylor series expansion to adjust historical statistics, aligning them with the current model state.
- Entropy-Weighted Gradient Accumulation (EWGA): To prevent noisy updates from uncertain predictions, NanoAdapt down-weights gradients from samples with high prediction entropy.
- Proxy Computation Graph (PCG): BN gradients naturally include interaction terms between samples. At $N=1$, these terms disappear. PCG re-introduces them by creating multiple augmented views of the single input sample, forcing the model to capture sample interactions even when only one real image is present.
Key Results
NanoAdapt demonstrates significant robustness across various batch sizes. On the CIFAR-100-C benchmark at $N=1$, standard TENT collapses to a 99.00% error rate. In contrast, NanoAdapt achieves an error rate of 40.05%, outperforming even the unadapted source model (46.76%). Similar gains are observed on CIFAR-10-C, where NanoAdapt reaches a 13.99% error rate compared to 90% for baselines. Unlike previous attempts, NanoAdapt remains effective across different batch sizes ($N=1, 10, 200$), making it truly batch-size agnostic.
Conclusion and Future Work
While NanoAdapt significantly improves performance on complex datasets like ImageNet-C at $N=1$, adaptation remains challenging, with final error rates occasionally exceeding the source model. Additionally, the use of multiple augmentations in the PCG increases computational overhead, which is a consideration for extremely low-power devices. Future work will focus on optimizing these computational requirements and refining hyperparameter tuning for diverse architectures.
Summary updated