TL;DR
InterNeg is a new framework that significantly improves out-of-distribution (OOD) detection for Vision-Language Models (VLMs) like CLIP. By ensuring that negative samples are selected based on inter-modal distances (image-to-text) rather than just text-to-text distances, it achieves state-of-the-art performance, including a 3.47% reduction in false positives on ImageNet and a 5.50% boost in accuracy on challenging "Near-OOD" tasks.
What Problem Does This Paper Address?
In the "open-world" deployment of AI, models often encounter objects they’ve never seen before. Ideally, a model should flag these as out-of-distribution (OOD) rather than confidently misclassifying them into a known category.
Recent research has turned to Vision-Language Models (VLMs) like CLIP for this task. Because CLIP is pre-trained on massive amounts of internet data, it already "knows" about millions of concepts. Current methods (like NegLabel) try to detect OOD samples by comparing a test image against a set of "negative" text labels (e.g., if an image looks more like "ash" or "dill" than the known class "cat," it’s probably OOD).
However, the authors identify a critical flaw: Distance Inconsistency. Most existing methods select these negative texts based on intra-modal distance—how far the text "ash" is from the text "cat" in the language space. But CLIP is optimized for inter-modal alignment—how well an image matches a text. Because the mapping between image and text spaces isn't perfectly identical, a text that seems "far" from a label might actually be "close" to an image of that label, leading the model to mistakenly flag known objects as OOD.
Methods
The authors propose InterNeg, which enforces consistency by using inter-modal distances throughout the detection process. The framework consists of two main strategies:
- Inter-modal Guided Negative Text Selection: Instead of picking negative texts that are semantically distant from ID (In-Distribution) labels, InterNeg picks texts that are distant from ID image proxies (the average visual features of a few example images). This ensures the "negatives" are truly visually distinct from the known classes.
- Inter-modal Guided Extra Negative Texts: During inference, if the model encounters an image it is highly confident is OOD, it performs modality inversion. It optimizes a set of "pseudo-tokens" to create a new text embedding that describes that specific OOD image. This dynamically expands the model's "negative vocabulary" on the fly.
- Dynamic Filtering: To prevent noisy or incorrect OOD images from ruining the results, the model uses a filtering mechanism. It only keeps the "inverted" text embeddings if they maintain a clear inter-modal distance from the known ID image proxies.
Datasets
The researchers evaluated InterNeg against a wide array of benchmarks:
- In-Distribution (ID): ImageNet-1K, CIFAR-10, and CIFAR-100.
- OOD Benchmarks: The "Traditional Four" (iNaturalist, SUN, Places, and Textures).
- OpenOOD Benchmark: A comprehensive suite that distinguishes between "Near-OOD" (objects very similar to ID classes) and "Far-OOD" (highly distinct objects).
Key Insights and Ideas
- The "Modality Gap" Matters: The most important insight is that the text-to-text distance is a poor proxy for image-to-text distance in CLIP. Aligning the detection strategy with CLIP’s training objective (inter-modal alignment) is the key to unlocking better performance.
- Two Types of Errors: The authors categorize ID misclassification into "Max-OOD dominant" (one very bad negative text causes a mistake) and "Sum-OOD dominant" (many mediocre negative texts aggregate to cause a mistake). InterNeg's selection criteria effectively mitigate both.
- Data Efficiency: Remarkably, the "image proxies" used for selection don't require a full training set. The authors found that using as few as 4 to 16 images per class is enough to create effective proxies, making this highly practical for low-data scenarios.
Key Results
InterNeg sets a new standard for zero-shot OOD detection (methods that don't require re-training the model):
- ImageNet-1K: Achieved a 3.47% reduction in FPR95 (the rate of false positives when 95% of true positives are detected) compared to previous leaders.
- Near-OOD: On the challenging Near-OOD benchmark, it showed a 5.50% improvement in AUROC, proving it can distinguish between very similar categories (like different breeds of dogs) better than previous methods.
- Competitive with Supervised Methods: Perhaps most impressively, InterNeg often outperforms methods that do require training on ID data, despite being essentially a zero-shot approach.
Limitations and Future Work
- Inference Latency: The "modality inversion" step (turning an OOD image into a text embedding) requires an optimization loop during inference. While the authors describe the overhead as "minor" (roughly 0.0067 seconds per image on an RTX 3090), it is still slower than a simple forward pass.
- Proxy Dependency: While it only needs a few images, it still requires some visual examples of the ID classes to build the proxies. A purely text-based version that maintains this inter-modal consistency remains an open challenge.
- Future Directions: The authors suggest exploring more efficient ways to perform modality inversion and investigating if this "inter-modal consistency" principle can improve other VLM tasks like zero-shot retrieval or grounding.
Summary updated