Deep learning models are notoriously overconfident. When presented with an image from a category they have never seen—known as out-of-distribution (OOD) data—they often assign it a high-confidence label from their known, in-distribution (ID) classes. This overconfidence poses a major safety concern for real-world deployments in critical fields like autonomous driving or medical diagnosis.
Existing solutions to this problem typically involve retraining the model or utilizing complex scoring functions based on the model's internal activations. However, retraining is computationally expensive, and standard scoring functions often fail because the "score gap" between ID and OOD data remains too narrow.
Instead of retraining complex deep learning models to better identify "unknown" data, what if we just changed the data itself? This paper introduces watermarking, a method that adds a learned, static pattern to input images to "reprogram" a fixed classifier, significantly boosting its ability to distinguish between ID and OOD samples.
Reprogramming via Data-Level Watermarking
The authors identify a missed opportunity in current OOD detection strategies: the reprogramming property of deep models. They demonstrate that a model can be made better at OOD detection by simply modifying the input data at test time, treating the model itself as a fixed black box.
The proposed technique, Watermarking, works through the following steps:
- The Static Watermark: A unified pattern ($w$) is learned with the same dimensions as the input image. At test time, this static pattern is simply added to every input ($x + w$).
- Zero-Cost Inference: Unlike previous adversarial or perturbation-based methods (like ODIN) which require a unique perturbation for every single image via a slow backward pass at test time, the watermark is static. Once learned, it is added to the input with virtually no computational overhead.
- The Learning Objective: The watermark is optimized to maximize the detection "score" (using metrics like Softmax or Free Energy) for ID data while minimizing it for OOD data.
- Robust Optimization: To ensure the watermark works reliably across different perturbations, the authors employ Sharpness-Aware Minimization (SAM). This helps find a "flat" minimum in the loss landscape, making the watermark less sensitive to small variations.
The "Edge" Strategy and Noise Proxies
Visually, the learned watermarks tend to preserve the center of the image—where the semantic object usually resides—while adding complex, colorful patterns to the edges. This suggests the model uses these modified edges as a "signature" to confirm the data is in-distribution.
Furthermore, since we cannot know what OOD data will look like in advance, the authors use Gaussian noise as a proxy during training. One of the most surprising findings of the paper is that training the watermark against simple Gaussian noise is sufficient to make the model significantly better at detecting real-world OOD data, such as textures or street signs.
Key Results: Widening the Score Gap
The researchers evaluated their method across standard benchmarks, using CIFAR-10, CIFAR-100, and ImageNet as ID datasets, and a variety of "far" (SVHN, Textures, Places365) and "near" OOD sets.
Because watermarking happens entirely at the data level, it is orthogonal to—and can be combined with—almost any existing classification-based OOD scoring function. The improvements are substantial across the board:
- CIFAR-10: Using Softmax scoring, watermarking reduced the average FPR95 (False Positive Rate at 95% True Positive Rate) from 55.70% to 42.86%.
- ImageNet: On this massive dataset, watermarking reduced the average FPR95 from 54.93% to 40.50% for Softmax scoring, and from 52.73% to 43.23% for Free Energy scoring.
- Near OOD: In the difficult task of distinguishing CIFAR-10 from CIFAR-100, watermarking improved the AUROC (Area Under the ROC Curve) from 55.47% to 65.69% when using specialized "shifting augmentations" during training.
Limitations and Future Work
While powerful, the watermarking method does have a few caveats. Performance is sensitive to hyperparameters, specifically the standard deviation of the noise proxy and the SAM constraint; finding the "sweet spot" is crucial so the noise doesn't overwhelm the watermark pattern. Additionally, a watermark is tied to the specific ID distribution the model was trained on—a watermark learned for CIFAR-10 will not necessarily transfer to a CIFAR-100 model.
Finally, data-level optimization can sometimes get stuck in suboptimal points, suggesting that more advanced optimization strategies could yield even better results in the future.
Ultimately, watermarking offers a "low-cost, high-reward" path for OOD detection. It proves that sometimes the best way to fix a model's behavior isn't to change its parameters, but to change what it sees.
Summary updated