MECTA: Memory-Economic Continual Test-Time Model Adaptation

Junyuan Hong, Lingjuan Lyu, Jiayu Zhou, Michael SprangerICLR 2023 · OpenReview

TL;DR

MECTA (Memory-Economic Continual Test-time Adaptation) is a new framework that slashes the memory overhead of adapting AI models to new environments by up to 70%. By intelligently pruning intermediate data caches and adapting only when necessary, it allows high-performance models to stay accurate on shifting data even when deployed on resource-constrained hardware like sensors and smartphones.

The "Memory Tax" of Continual Adaptation

Deep learning models often struggle when they encounter "out-of-distribution" (OOD) data—think of a self-driving car trained in sunny California suddenly navigating a Canadian blizzard. Continual Test-time Adaptation (CTA) addresses this by allowing a model to update itself on the fly using the unlabeled data it encounters during deployment.

While CTA methods like Tent and EATA significantly improve accuracy, they come with a massive "memory tax." Even if only a small fraction of parameters are updated, the standard backpropagation process requires caching intermediate activations for every layer. For a standard ResNet50 model, the memory required for these updates can be five times higher than simple inference. This makes advanced CTA nearly impossible to run on "edge" devices with limited RAM.

MECTA Framework Comparison

How MECTA Works

To solve this bottleneck, the authors introduce MECTA, which optimizes memory across three dimensions: batch, channel, and layer.

  1. MECTA Norm (The "B" Component): Standard adaptation requires large batches of data to calculate stable statistics. MECTA uses an adaptive normalization layer with a shift-aware forget gate. This allows the model to maintain stable predictions even with tiny batch sizes by intelligently balancing long-term and short-term memory of data distributions.
  2. Stochastic Cache Pruning (The "C" Component): To compute gradients, systems usually store a "cache" of every channel in every layer. MECTA randomly drops a percentage of these channels from the cache. Crucially, this only happens in the backward pass; the forward pass (inference) remains full-strength, ensuring high-quality features while drastically reducing the memory needed for updates.
  3. Adaptive Layer Training (The "L" Component): Not every layer needs constant updating. MECTA monitors the "distribution gap" (how much the new data differs from the model's training). If the gap is small, it stops caching and training for that specific layer, saving both memory and computation.

Key Insights

  • The Cache is the Bottleneck: While most research focuses on reducing the number of parameters to update, MECTA proves that the real memory hog is the activation cache required for backpropagation.
  • Pruning as Regularization: Surprisingly, randomly dropping channels from the gradient cache doesn't just save memory—it acts as a form of regularization. It prevents the model from "over-adapting" to specific noise in the test data, mitigating the common problem of "catastrophic forgetting."
  • On-Demand Adaptation: By using normalization statistics as a proxy for distribution shift, the model essentially "decides" when an update is worth the energy and memory, rather than blindly training on every single frame.

Key Results

The researchers evaluated MECTA on standard robustness benchmarks, including CIFAR-10-C, CIFAR-100-C, and ImageNet-C (which features high-resolution images and modern architectures like ResNet50).

Accuracy vs. Memory Trade-off
  • Massive Memory Savings: On ImageNet-C, MECTA reduced the peak memory cost of ResNet50 by at least 70% compared to standard CTA methods.
  • Superior Accuracy under Constraints: When memory was strictly limited (forcing baseline methods to use tiny batch sizes), MECTA improved accuracy by 8.5% to 73% across various datasets.
  • High Efficiency: Unlike other memory-saving techniques like Gradient Checkpointing (GC), MECTA adds negligible computational overhead (less than a 0.2% increase in GFLOPs), whereas GC can increase computation by up to 20%.

Limitations and Future Work

While MECTA significantly reduces the cache required for backpropagation, it does not address the memory required for the initial forward pass, which remains a fixed cost. Additionally, the current implementation is evaluated primarily on convolutional neural networks (CNNs). While the core principles should apply to Transformers (like ViTs), specific architectural tweaks might be necessary.

Future work could involve a holistic solution that jointly optimizes both inference time and memory in a hardware-aware manner, potentially tailoring pruning rates to specific mobile GPU or NPU architectures.

Summary updated

Abstract

Continual Test-time Adaptation (CTA) is a promising art to secure accuracy gains in continually-changing environments. The state-of-the-art adaptations improve out-of-distribution model accuracy via computation-efficient online test-time gradient descents but meanwhile cost about times of memory versus the inference, even if only a small portion of parameters are updated. Such high memory consumption of CTA substantially impedes wide applications of advanced CTA on memory-constrained devices. In this paper, we provide a novel solution, dubbed MECTA, to drastically improve the memory efficiency of gradient-based CTA. Our profiling shows that the major memory overhead comes from the intermediate cache for back-propagation, which scales by the batch size, channel, and layer number. Therefore, we propose to reduce batch sizes, adopt an adaptive normalization layer to maintain stable and accurate predictions, and stop the back-propagation caching heuristically. On the other hand, we prune the networks to reduce the computation and memory overheads in optimization and recover the parameters afterward to avoid forgetting. The proposed MECTA is efficient and can be seamlessly plugged into state-of-the-art CTA algorithms at negligible overhead on computation and memory. On three datasets, CIFAR10, CIFAR100, and ImageNet, MECTA improves the accuracy by at least 6% with constrained memory and significantly reduces the memory costs of ResNet50 on ImageNet by at least 70% with comparable accuracy. Our codes can be accessed at https://github.com/SonyAI/MECTA.
BibTeX
@inproceedings{hong2023mecta,
  author = {Junyuan Hong and Lingjuan Lyu and Jiayu Zhou and Michael Spranger},
  title = {{MECTA: Memory-Economic Continual Test-Time Model Adaptation}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=N92hjSf5NNh},
}
Figure