Abstract
Deep neural networks (DNNs) often make overconfident yet incorrect predictions on out-of-distribution samples, posing a challenge for real-world deployments. When models are deployed as black boxes, retraining is often impossible. We propose the REinforcement Spatial Transform learner (REST), a module that transforms distorted input data into samples regarded as in-distribution by black-box models. By training only the REST module using reinforcement learning and a confidence score reward, we improve robustness to geometric transformations without retraining the original model.
Introduction
Deep learning models are notoriously brittle when faced with out-of-distribution (OOD) data. For instance, a classifier trained on centered images may fail if the input is tilted or scaled. In many practical scenarios, these models are "black boxes"—proprietary or legacy systems where internal weights and gradients are inaccessible. Standard solutions like Spatial Transformer Networks (STNs) typically require end-to-end retraining, which is not feasible for black boxes. REST addresses this by acting as a pre-processing "un-distorter" that learns to map real-world distorted data back into the training distribution the black box expects.
Methodology: The REST Framework
The REST module is placed in front of a black-box image classifier and operates through an iterative reinforcement learning loop.
Architecture
REST consists of two primary components:
- RL Module: An Actor-Critic architecture trained via Proximal Policy Optimization (PPO). It observes the current image state and predicts transformation parameters (rotation, scaling, shearing, and translation).
- Warp Module: A differentiable module that applies the predicted transformation to the image.
The Iterative Loop
Unlike one-shot transformers, REST works iteratively. It transforms the image, passes it to the black box, and receives a confidence score. This score serves as the reward signal for the RL agent. The goal is to maximize this confidence, effectively "moving" the image back into the distribution the black box was originally optimized for.
Experimental Evaluation
The researchers evaluated REST across several benchmarks, including MNIST, SVHN, CIFAR10, and STL10, modified with geometric distortions of increasing complexity (Rotation, Scaling, Shearing, and Translation).
Key Results
- Robustness to Complexity: On the most complex MNIST distortions (RSST), the baseline black-box model's accuracy was only 17.99%. Adding REST boosted this to 85.05%. Similarly, SVHN accuracy improved from 26.09% to 83.82%.
- Generalization: REST demonstrated superior generalization compared to standard CNNs and STNs. In tests where training and test transformations were disjoint (e.g., training on "rotate right" and testing on "rotate left"), REST maintained high performance by learning a general policy of exploration rather than memorizing specific distortions.
- Sample Efficiency: REST can learn effective strategies with as few as 1,000 distorted samples, making it significantly more efficient than standard data augmentation techniques.
Discussion and Conclusion
REST provides a "zero-touch" improvement for black-box models, requiring no access to internal parameters. Its primary strength lies in its ability to learn a transformation policy that generalizes to unseen geometric variations.
However, challenges remain. The iterative nature of REST introduces inference latency, as it may take multiple steps to "fix" an image. Additionally, if a black-box model is overconfident on incorrect OOD predictions, the RL agent may receive misleading rewards. Future work could extend this approach to non-spatial distortions like denoising or deblurring, further enhancing the robustness of black-box deployments.
Summary updated