In the "closed-world" of traditional machine learning, we assume our models will only encounter data similar to what they saw during training. In the real world, this assumption is dangerous. Whether it's an autonomous car encountering a bizarrely shaped vehicle or a quality control AI seeing a new type of product defect, systems must be able to say, "I don't know what this is."
A new survey paper, "Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era," provides a much-needed map for this territory. As Vision-Language Models (VLMs) like CLIP and Large VLMs (LVLMs) like GPT-4V redefine computer vision, the boundaries between safety-critical tasks like Out-of-Distribution (OOD) detection and Anomaly Detection (AD) are blurring.
TL;DR
This paper introduces Generalized OOD Detection v2, a unified framework that tracks how Vision-Language Models have transformed the landscape of detecting "unknown" data. It reveals that while some sub-fields are merging or becoming inactive, OOD detection and Anomaly Detection remain the primary frontiers, shifting from traditional training-heavy approaches to zero-shot and prompt-based reasoning.
What Problem Does This Paper Address?
For years, researchers have worked on five closely related but distinct problems:
- OOD Detection: Finding samples from classes not seen during training.
- Anomaly Detection (AD): Identifying defects or "abnormal" versions of known objects.
- Novelty Detection (ND): Detecting new categories in a single-class setting.
- Open Set Recognition (OSR): Classifying knowns while simultaneously rejecting unknowns.
- Outlier Detection (OD): Finding "weird" samples in an unlabeled dataset.
Previously, these were separated by technicalities (e.g., whether you had one class or many, or if you were training or testing). However, the rise of VLMs—which are pre-trained on massive amounts of internet data—has disrupted these definitions. If a model has already "seen" almost everything on the internet, what does "out-of-distribution" even mean? The authors tackle this confusion by proposing a new taxonomy to unify these fields for the VLM era.
Methods
The authors categorize VLM-based detection methods based on three main axes:
- Data Availability: Zero-shot (no images of the target classes used) vs. Few-shot (a handful of examples used).
- Training Type: Training-free (using the model as-is), ID Training (fine-tuning on known data), or Auxiliary Training (using external datasets to define "bad" samples).
- Prompting Strategy: Whether the method uses "OOD Prompts" (e.g., specifically telling the model what a "non-dog" looks like) or relies solely on known class descriptions.
A key shift highlighted is the move toward post-hoc methods like MCM (Maximum Concept Matching), which simply scales the model's output scores to better separate knowns from unknowns without requiring any new training.
Datasets
The survey tracks the evolution of benchmarks from small-scale (MNIST, CIFAR) to large-scale:
- Standard OOD: ImageNet-1K is used as the "known" set, with iNaturalist, SUN, and Places acting as the "unknowns."
- Hard OOD: Benchmarks like ImageNet-10/20 and the new ImageNet-X force models to distinguish between very similar categories (e.g., different breeds of dogs), where one is "known" and the other is "OOD."
- Anomaly Detection: MVTec-AD and VisA remain the gold standards for industrial inspection.
- The New Frontier: The paper introduces MM-UPD Bench, designed for "Unsolvable Problem Detection"—testing if an LVLM can recognize when a question about an image is nonsensical or unanswerable.
Key Insights and Ideas
- Field Consolidation: The authors argue that OSR has effectively been integrated into "Hard OOD Detection," and fields like Novelty Detection are becoming inactive because VLMs' broad knowledge makes the "single-class" assumption less relevant.
- The "Prompting" Paradigm: In Anomaly Detection, the "aha moment" was realizing that language is a powerful prior. Simply asking a model to compare "a photo of a [object]" vs. "a photo of a damaged [object]" can often outperform complex specialized algorithms.
- From Detection to Reasoning: With LVLMs (like GPT-4V), OOD detection is evolving into Unsolvable Problem Detection (UPD). This isn't just about pixels; it's about whether the model can reason that "There are no fish in this picture, so I cannot answer how many fish there are."
Key Results
- Saturation: On "easy" benchmarks like ImageNet-20, current VLM methods are hitting >97% AUROC, suggesting these benchmarks are nearly solved.
- The Near-OOD Gap: On harder, semantically similar benchmarks (ImageNet-X), performance drops significantly to around 80%, indicating a major area for improvement.
- Zero-shot Prowess: Methods like CLIPN and NegLabel show that we can now detect OOD samples with high accuracy without ever showing the model a single image of the "known" classes during a specialized training phase.
Limitations and Future Work
- Object-Level Detection: Most current work looks at the whole image. Detecting a single OOD object within a complex, busy scene (e.g., using Segment Anything Models) is an open challenge.
- Medical Constraints: While VLMs are great for industrial parts, medical anomalies are harder to describe in natural language, making VLM-based medical AD a difficult frontier.
- LVLM Hallucinations: Large models still "hardly hesitate to answer," often making up answers for unsolvable questions rather than admitting they don't know. Improving this "abstention" capability is critical for safety.
For researchers looking to dive deeper, the authors maintain a curated list of resources at https://github.com/AtsuMiyai/Awesome-OOD-VLM.
Summary updated