Abstract
Distribution shifts, which often occur in the real world, degrade the accuracy of deep learning systems, and thus improving robustness is essential for practical applications. To improve robustness, we study an image enhancement method that generates recognition-friendly images without retraining the recognition model. We propose a novel image enhancement method, DynTTA, which is based on differentiable data augmentation techniques and generates a blended image from many augmented images to improve the recognition accuracy under distribution shifts. In addition to standard data augmentations, DynTTA also incorporates deep neural network-based image transformation, which further improves the robustness. Because DynTTA is composed of differentiable functions, it is directly trained with the classification loss of the recognition model. We experiment with widely used image recognition datasets using various classification models, including Vision Transformer and MLP-Mixer. DynTTA improves the robustness with almost no reduction in classification accuracy for clean images, which is a better result than the existing methods. Furthermore, we show that estimating the training time augmentation for distribution-shifted datasets using DynTTA and retraining the recognition model with the estimated augmentation significantly improves robustness.
The Challenge of Distribution Shifts
Deep learning models are notoriously fragile when deployed in real-world environments. A model trained on high-quality, curated datasets often fails when it encounters "distribution shifts"—phenomena such as rain, sensor noise, or motion blur. These shifts create a gap between the training data and the test-time reality, leading to significant drops in accuracy.
Current solutions to this problem have significant drawbacks. Retraining a model is often impossible if the original training data is unavailable or if the model is accessed via a third-party API. DNN-based enhancement methods, such as Universal Image Enhancement (URIE), frequently "over-enhance" images, which can actually reduce accuracy on clean, undistorted data. Meanwhile, standard Test-Time Augmentation (TTA) techniques like simple flipping or cropping are too limited to handle complex distortions like haze or compression artifacts.
DynTTA: A Differentiable Approach to Image Enhancement
To address these limitations, we propose DynTTA (Dynamic Test-Time Augmentation). Unlike traditional TTA, which selects from a discrete set of fixed transformations, DynTTA treats the entire augmentation process as a differentiable optimization problem. This allows the system to find the optimal "recognition-friendly" version of an image for a specific, frozen classifier.
The Architecture
The DynTTA framework consists of several key components:
- Backbone Network: A small neural network (e.g., ResNet18) processes the input image to predict magnitude parameters (e.g., rotation angle, contrast level) and blend weights for various augmentations.
- Differentiable Augmentations: The system utilizes 14 types of differentiable transformations. Because these functions are differentiable, the system can calculate exactly how adjusting a parameter will minimize the final classification loss.
- Image Blending: Rather than selecting a single "best" transformation, DynTTA creates a weighted linear combination of multiple augmented images. This blending allows the model to neutralize complex, multi-factor distortions effectively.
- End-to-End Training: The enhancement module is trained by passing the blended image through a frozen recognition model. The classification error is backpropagated through the frozen model into the DynTTA backbone, teaching it to produce images that the classifier can accurately recognize.
Experimental Evaluation
We evaluated DynTTA across several challenging benchmarks, including CUB (fine-grained bird classification), ImageNet, and PACS (domain generalization across Art, Cartoon, Photo, and Sketch). We also tested against 19 types of algorithmically generated corruptions at various severity levels.
Key Results and Insights
- Robustness without Sacrifice: On ImageNet, DynTTA improved the robustness of a pre-trained ResNet50 by 5.30 percentage points with virtually no reduction in accuracy on clean images. This distinguishes it from DNN-based enhancers that often degrade clean-image performance.
- Superiority of Blending: Our findings indicate that blending multiple augmentations is superior to selecting just one, as it provides the flexibility needed to find a middle ground for complex distortions.
- The "Inverse Operation" Hypothesis: DynTTA can be used to "diagnose" model weaknesses. By observing which augmentations DynTTA applies to fix test images, we can infer what was missing in the training data. Retraining the primary model with the inverse of these preferred augmentations led to massive gains in base robustness (e.g., a 6.43 percentage point improvement on PACS).
- Efficiency: By implementing a threshold for blend weights and ignoring augmentations with negligible contributions, we reduced the number of required computations by 72% with minimal impact on accuracy.
Conclusion and Future Work
DynTTA provides a robust, "safe," and flexible framework for improving model performance under distribution shifts without requiring access to training data or model weights. While the current implementation adds some computational latency at test time, its ability to preserve clean-image accuracy while boosting robustness makes it a powerful tool for real-world deployment. Future research will explore extending these "recognition-friendly" enhancements to other computer vision tasks, such as object detection and semantic segmentation.
Summary updated