TL;DR
This paper introduces DynTTA, a novel test-time augmentation (TTA) method that dynamically generates recognition-friendly images by blending multiple differentiable augmentations. DynTTA significantly improves model robustness to distribution shifts without sacrificing clean accuracy and can even be used to estimate effective training-time augmentations for retraining models.
What Problem Does This Paper Address?
Deep learning models excel in controlled environments but struggle when faced with "distribution shifts" – real-world variations like changes in lighting, weather, or image compression. These shifts can drastically reduce model accuracy, making them unreliable for critical applications like autonomous driving. Existing solutions either retrain models (costly and prone to forgetting) or use simpler TTA methods that have limited augmentation spaces and thus limited robustness gains. This paper aims to bridge this gap by proposing a more powerful and flexible TTA approach.
Methods
DynTTA tackles distribution shifts by enhancing input images during inference without retraining the original recognition model. Its core innovation lies in two key ideas:
- Differentiable Augmentations and Blending: DynTTA leverages a vast space of differentiable data augmentations (including standard ones and even DNN-based transformations) and combines them using learned "blend weights." This allows it to create a more sophisticated, blended image that is tailored to be recognized by the classification model.
- Optimization of Augmentation Parameters: DynTTA uses gradient descent to jointly optimize both the magnitude parameters of individual augmentations and the blend weights. This allows it to find the optimal combination of transformations for a given input image, going beyond simply selecting the "best" single augmentation.
A novel aspect is DynTTA's ability to estimate effective training-time augmentations for distribution-shifted datasets. By analyzing the learned blend weights, the method can infer which augmentations are most beneficial for a particular shift and then use these to retrain the classification model for enhanced robustness.
Datasets
The authors evaluated DynTTA on several standard image recognition datasets:
- CUB (Caltech-UCSD Birds 200): Used for experiments on robustness and generalization.
- ImageNet: A large-scale dataset used for evaluating performance on a broad range of images.
- PACS: A domain generalization dataset comprising four domains (art, cartoons, photos, sketches).
- Stylized-ImageNet, ImageNet-A, ImageNet-R: Datasets used to test robustness against more complex, out-of-distribution shifts.
- Corruption Datasets: Specifically, the 19 algorithmically generated corruption datasets from Hendrycks and Dietterich [12] were used, with a distinction made between "Seen" (used for training) and "Unseen" corruptions.
The paper also introduces a "blind setting" for evaluation, where the test-time distribution shifts are unknown, making it more realistic than traditional "non-blind" settings.
Key Insights and Ideas
- Blending is Better than Selecting: DynTTA's approach of blending multiple augmented images significantly outperforms methods that simply select the single "best" augmentation. This suggests that a combination of transformations is more effective than any individual one.
- Differentiable Augmentations Unlock Optimization: The differentiability of the augmentations is crucial, allowing for end-to-end training and the optimization of both augmentation parameters and blending weights.
- Estimating Training Augmentations is Powerful: DynTTA's ability to infer effective training augmentations from test-time behavior and then use them for retraining offers a powerful way to boost robustness beyond what TTA alone can achieve.
- Robustness Without Sacrificing Clean Accuracy: A key achievement is DynTTA's ability to improve robustness to distribution shifts with minimal degradation in accuracy on clean (un-shifted) images, a common pitfall for other methods.
- Generalizability of Enhancement Models: DynTTA-trained enhancement models show good generalizability to different classification backbones, even those not used during training.
Key Results
- DynTTA consistently improved robustness across various datasets and classifiers, often by significant margins. For example, on ImageNet, it improved robustness by 5.30 percentage points for ResNet50 while maintaining clean accuracy.
- Retraining with DynTTA-estimated augmentations on the PACS dataset improved accuracy by 6.43 percentage points.
- In the "blind setting" (unknown distribution shifts), DynTTA demonstrated superior performance compared to existing methods like URIE and Loss Predictor, especially in challenging scenarios.
- DynTTA achieved these improvements with computational overhead comparable to other TTA methods, and significantly less than larger classification models.
Limitations and Future Work
- Computational Overhead: While DynTTA is more efficient than some alternatives, it still incurs computational cost during inference due to performing multiple augmentations. The authors propose a technique to reduce the number of executed augmentations by skipping those with very small blend weights, achieving a 72% reduction in executions with minimal accuracy loss.
- Generalizability to Other Tasks: The current study focuses on image classification. Future work includes applying and evaluating DynTTA for tasks like object detection and segmentation.
- Backbone Architecture: Further exploration into optimal backbone architectures for DynTTA is suggested.
- Differentiable Augmentations for Other Modalities: The authors believe DynTTA can be extended to other data types (e.g., language) if differentiable augmentations can be defined.
Summary updated