What Remains OOD? OODConceptBench for Open-World Concept Recognition in Vision-Language Models

Zhiran Luo, Zhiyong Yang, Zitai Wang, Sicong Li, Qianqian Xu, Qingming HuangarXiv 2026 · arXiv

TL;DR

Researchers have introduced OODConceptBench, a new hierarchical benchmark designed to test how well Vision-Language Models (VLMs) recognize "open-world" concepts that fall outside traditional training taxonomies. By evaluating 37 different models, they discovered that newer, larger models aren't necessarily better at recognizing these concepts, and that improvements in overall accuracy often mask significant performance drops in specific categories like "Cultural blindspots."

What Problem Does This Paper Address?

In the "closed-world" era of computer vision, out-of-distribution (OOD) detection was simple: if a model was trained on 1,000 ImageNet classes, anything else was OOD. However, modern VLMs like CLIP are pretrained on billions of web-scale image-text pairs, giving them impressive zero-shot generalization. This blurs the boundary of what is "known."

The authors argue that a predefined label set no longer describes what a VLM can recognize. A model might "know" what a Moai statue is even if it was never a formal training label. The challenge is that we lack a systematic way to measure which open-world concepts these models actually understand and where they fail. Existing benchmarks often focus on specific failure modes (like counting or spatial reasoning) rather than the breadth of concepts encountered in the wild.

Figure 1: From predefined-class OOD to model-specific concept recognition. OODConceptBench fixes image–concept pairs and compares recognition across models using a shared calibration rule with model-specific recognition thresholds.

Methods

The authors constructed OODConceptBench using a hierarchical approach:

  1. Vocabulary Design: They defined 560 concept phrases across seven diagnostic categories: Time-lagged (new concepts like the "Xiaomi SU7"), Synthetic/fictional, Cultural blindspots, Fine-grained/texture, Environmental shift, Viewpoint/scale, and Typographic conflict.
  2. Image Collection: They gathered nearly 48,000 images via web search and AI generation.
  3. Filtering & Layering: They applied three filtering rules to ensure the concepts were truly "open-world" (distinct from common ImageNet categories). This resulted in two layers:
  4. L1 (Full Set): 30,752 images.
  5. L2 (Hard Subset): 14,285 images selected for having lower image-text similarity, making them more challenging for models to align.
  6. Model-Specific Calibration: Because different VLMs have different similarity scales, the authors used a shared calibration pool (2,000 images from MNIST, CIFAR, etc.) to set a unique "recognition threshold" for each model. A concept is only "recognized" if its similarity score exceeds this calibrated baseline.

Datasets

  • OODConceptBench: The primary contribution, featuring 560 concepts and two difficulty layers (L1 and L2).
  • Reference Banks: ImageNet-1K and a 200-vector subset of ImageNet-21K were used during the filtering process to ensure benchmark images were sufficiently distant from "common" categories.
  • Calibration Pool: Images from MNIST, CIFAR-100, STL-10, and Oxford-IIIT Pet were used to establish model-specific thresholds.
Image-source composition and target-concept coverage across layers.

Key Insights and Ideas

  • The L2 Persistence: Across all 37 models, performance was consistently lower on the L2 subset than L1. This suggests that the "hardness" defined by the construction model (OpenAI CLIP) generalizes across the entire VLM family.
  • Non-Monotonic Progress: Surprisingly, model performance does not improve steadily by release year. Some models released in 2023 performed worse than those from 2022. Simply scaling up or using newer data doesn't automatically solve OOD recognition.
  • The "Aggregate Score" Trap: An increase in a model's overall recognition score often hides declines in specific areas. For instance, while overall performance improved slightly from 2024 to 2025, five out of seven categories actually saw a decline in recognition.
  • Cultural Blindspots: This category proved particularly volatile. While newer models improved on "Time-lagged" concepts (likely due to more recent training data), they often regressed on "Cultural blindspots," highlighting a potential bias or forgetting in newer training pipelines.

Key Results

  • Widespread Difficulty: Recognition rates varied wildly, from a dismal 1.3% to 79.8% on L1 and 0.5% to 59.6% on L2. No model reached 80% recognition on the full set.
  • Universal L2 Drop: In 258 out of 259 model-category comparisons, recognition was lower on L2 than L1.
  • Release Year Trends: The average L2 recognition for 2021 models was 22.8%, jumping to 48.4% in 2022, but then dipping to 41.1% in 2023 before recovering to ~52% in 2024-2025.
Figure 2: L2 concept-recognition landscape for the 2025 release-year group. The 91 concepts are positioned by group-mean recognition, with lower recognition toward the outer regions and higher recognition toward the center.

Limitations and Future Work

The authors acknowledge that their benchmark, while broad, only covers 560 concepts—a tiny fraction of the "open world." The conclusions are also limited to the 37 models evaluated.

Future work could involve:

  • Expanding the vocabulary to include even more diverse and niche concepts.
  • Investigating why certain categories (like Cultural blindspots) decline in newer models.
  • Developing training algorithms specifically designed to improve OOD recognition rather than just relying on general scale.
  • Exploring how different prompt templates or calibration methods might shift these recognition boundaries.

Summary updated

Abstract

In the era of large-scale pretrained models, zero-shot generalization has increasingly blurred the boundary between known and unknown concepts. However, current evaluations do not directly reveal which open-world concepts different vision-language models can recognize. To study this problem, we introduce OODConceptBench, a hierarchical benchmark for open-world concept recognition on image-concept pairs. The full benchmark set (L1) contains 30,752 images, and the harder subset (L2) contains 14,285 images. The benchmark covers 560 concept phrases across seven open-world categories. We evaluate 37 vision-language models across release year, model family, and variant. We obtain three main findings. First, target-concept recognition is lower on L2 than on L1 across all 37 evaluated models. Second, later-released models do not necessarily achieve stronger open-world concept recognition. Third, an increase in overall recognition does not imply consistent improvement across all benchmark categories. Together, these results show that OOD remains a challenge even in the era of large-scale vision-language pretraining.
BibTeX
@misc{luo2026what,
  author = {Zhiran Luo and Zhiyong Yang and Zitai Wang and Sicong Li and Qianqian Xu and Qingming Huang},
  title = {{What Remains OOD? OODConceptBench for Open-World Concept Recognition in Vision-Language Models}},
  year = {2026},
  eprint = {2610.33564},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.33564},
}
Figure