Dynamic Test-Time Augmentation via Differentiable Functions

Shohei Enomoto, Monikka Roslianna Busto, Takeharu Eda

TL;DR

Real-world distribution shifts (like blur, noise, and weather) severely degrade deep learning model accuracy, but fixing this usually requires expensive retraining or sacrificing performance on clean images. This paper introduces DynTTA (Dynamic Test-Time Augmentation), a novel image enhancement method that uses differentiable functions and image blending to dynamically optimize corrupted images for frozen classification models, boosting robustness without losing clean accuracy.

What Problem Does This Paper Address?

Deep neural networks are notoriously brittle when deployed in the real world. A model trained on pristine studio images will often fail when faced with "distribution shifts" caused by rain, sensor noise, motion blur, or compression artifacts.

To solve this without the high cost of retraining the core classifier, researchers use image enhancement models at test time to "clean up" images before feeding them to the network. However, existing approaches suffer from a frustrating trade-off:

  1. DNN-based transformers (like URIE) are powerful but tend to overfit to specific corruption patterns, leading to excessive modifications that actually reduce accuracy on clean images.
  2. Test-time augmentation (TTA) methods select the best classic data augmentation for an image, which preserves clean accuracy but operates in a highly restricted search space, yielding only minor robustness gains.

Methods

The authors propose DynTTA, an end-to-end differentiable framework that treats image enhancement as an optimization problem over an infinite, continuous augmentation space.

Overview diagram of the DynTTA pipeline

As shown in the pipeline above, DynTTA works through a three-step process:

  • Parameter Prediction: A lightweight backbone network takes the corrupted input image and predicts continuous magnitude parameters ($m_k$) and blend weights ($w_k$) for a predefined set of candidate augmentations.
  • Differentiable Blending: Instead of choosing just one augmentation, DynTTA applies all candidate transformations according to their predicted magnitudes and linearly blends the resulting images together based on the weights $w_k$.
  • End-to-End Optimization: Because every operation in the pipeline—including the augmentations (leveraging the Kornia library) and the blending—is fully differentiable, DynTTA can be trained directly using the classification loss of the downstream recognition model, which remains completely frozen.

The candidate augmentation space includes standard geometric and color adjustments, discretized low-pass and high-pass filters, and even a deep neural network-based image transformer (URIE) treated as a candidate augmentation.

Additionally, the authors propose an Inverse Augmentation Retraining strategy. By analyzing which test-time augmentations DynTTA weights heavily on a shifted dataset, they can deduce the nature of the shift and apply the inverse operations during training to make the core classifier natively robust.

Datasets

The method was evaluated across several prominent computer vision benchmarks:

  • CUB (Caltech-UCSD Birds 200): Used for fine-grained image classification.
  • ImageNet: The standard benchmark for large-scale visual recognition.
  • PACS: A domain generalization dataset containing four distinct styles (Art, Cartoons, Photos, Sketches).
  • Algorithmically Corrupted Variants (CUB-C, ImageNet-C): Datasets featuring 19 different corruption types across 5 severity levels, evaluated in both "blind" (completely unknown test corruptions) and "non-blind" settings.
  • Complex Shifts: Stylized-ImageNet, ImageNet-A (natural adversarial examples), and ImageNet-R (renditions).

Key Insights and Ideas

  • Blending Beats Selecting: Traditional TTA tries to pick the single best discrete augmentation for an image. DynTTA proves that an affine combination (blending) of many augmented images creates a much smoother, more expressive space that provides superior robustness.
  • Taming DNN Transformers: Pure DNN-based image restorers overfit because they always transform the image, even when it's clean. By embedding a DNN restorer inside DynTTA's blending framework alongside standard augmentations, the model naturally learns to dial down the DNN's weight to zero on clean images, preserving baseline performance.
  • Gradient Descent Hyperparameter Tuning: By utilizing differentiable augmentations, the optimal transformation magnitudes are found automatically via backpropagation during training, eliminating the need for tedious manual hyperparameter tuning at test time.

Key Results

DynTTA successfully breaks the trade-off between baseline clarity and environmental robustness.

Pros and cons of test-time augmentation, image transformation, and DynTTA
  • Robustness Gains: On the massive ImageNet dataset, DynTTA improved the robustness of a pretrained ResNet50 by 5.30 percentage points under distribution shifts, outperforming existing methods.
  • Architecture Agnostic: In "blind" settings on the CUB dataset, DynTTA consistently improved performance across diverse architectures, including ResNet50, MLP-Mixer, and DeiT (Vision Transformers).
  • Retraining Breakthrough: Using DynTTA's inverse augmentation estimation to retrain a ResNet50 on the PACS dataset yielded a substantial 6.43 percentage point improvement in accuracy.

Limitations and Future Work

The primary limitation of DynTTA is its computational overhead. Executing dozens of data augmentations and blending them during inference increases latency. To mitigate this, the authors introduced a thresholding technique that skips augmentations with near-zero predicted weights, which successfully cut down execution runs by 72% while maintaining accuracy. However, real-time deployment on edge devices remains a challenge.

For future work, the authors plan to explore optimal backbone architectures for DynTTA and expand its application beyond image classification to more complex tasks like object detection and semantic segmentation.

Summary updated

Abstract

Distribution shifts, which often occur in the real world, degrade the accuracy of deep learning systems, and thus improving robustness to distribution shifts is essential for practical applications. To improve robustness, we study an image enhancement method that generates recognition-friendly images without retraining the recognition model. We propose a novel image enhancement method, DynTTA, which is based on differentiable data augmentation techniques and generates a blended image from many augmented images to improve the recognition accuracy under distribution shifts. In addition to standard data augmentations, DynTTA also incorporates deep neural network-based image transformation, further improving the robustness. Because DynTTA is composed of differentiable functions, it can be directly trained with the classification loss of the recognition model. In experiments with widely used image recognition datasets using various classification models, DynTTA improves the robustness with almost no reduction in classification accuracy for clean images, thus outperforming the existing methods. Furthermore, the results show that robustness is significantly improved by estimating the training-time augmentations for distribution-shifted datasets using DynTTA and retraining the recognition model with the estimated augmentations. DynTTA is a promising approach for applications that require both clean accuracy and robustness. Our code is available at \url{https://github.com/s-enmt/DynTTA}.
BibTeX
@inproceedings{enomoto2024dynamic,
  author = {Shohei Enomoto and Monikka Roslianna Busto and Takeharu Eda},
  title = {{Dynamic Test-Time Augmentation via Differentiable Functions}},
  booktitle = {IEEE Access},
  year = {2024},
  url = {https://openreview.net/forum?id=8ZUqde9No6},
}

Loading related content...

Figure