Zero-Shot Detection of Out-of-Context Objects Using Foundation Models

Anirban Roy, Adam D. Cobb, Ramneet Kaur, Sumit Jha, Nathaniel D. Bastian, Alexander M. Berenbeim, Robert H. Thomson, Iain Cruickshank, Alvaro Velasquez, Susmit Jha

Abstract

Detecting objects that appear out of context (OOC) is a complex task requiring deep world knowledge. Traditional supervised methods are often limited by their training data and fail to generalize to novel scenarios. This paper introduces a zero-shot framework that utilizes the "common sense" encoded in foundation models—specifically BLIP-2, GPT-4, and Stable Diffusion—to detect and localize OOC objects. Our approach achieves state-of-the-art results on major benchmarks without any task-specific fine-tuning.

1. Introduction

Visual context serves as the "glue" for scene understanding. Humans immediately recognize the absurdity of a car flying through the air or an elephant sitting in a bird's nest. However, teaching AI to recognize these violations is difficult because context is nuanced and vast.

Examples of out-of-context scenarios: a couch in a pool, a patterned elephant in a room, a low-flying plane, and a jumping car.

Existing methods typically rely on rigid rules or co-occurrence statistics learned from specific datasets. These systems are brittle and struggle with the diversity of real-world contexts. We propose a flexible, zero-shot alternative that leverages the broad knowledge of large-scale models to reason about contextual consistency.

2. Methodology

Our pipeline treats OOC detection as a multimodal reasoning and verification task consisting of four primary stages:

  1. Image Captioning: We utilize BLIP-2 to generate a detailed textual description of the scene, converting visual relationships into a format suitable for linguistic analysis.
  2. Linguistic Reasoning: The generated caption is passed to GPT-4. Using specific prompts, GPT-4 acts as a "common sense" engine to evaluate the likelihood of the described scenario.
  3. Counterfactual Verification & Localization: If an anomaly is detected, the system identifies the likely OOC object. It then employs a segmentation model to mask the object and Stable Diffusion 2.0 to "in-paint" the area with contextually appropriate content.
  4. Final Validation: The modified image is re-captioned and re-evaluated. If the resulting scene is deemed "normal," the system confirms the original object was indeed out of context.
The counterfactual verification process: an out-of-context elephant is identified (left) and replaced with contextually appropriate branches (right).

3. Experimental Results

We evaluated our framework on two benchmarks: MIT-OOC (real-world images) and IJCAI22-COCO-OOC (synthetic transplants).

3.1 Detection Performance

Our method significantly outperformed previous state-of-the-art (SOTA) supervised models:

  • MIT-OOC: Achieved 90.82% accuracy (previous SOTA: 73.29%).
  • COCO-OOC: Achieved 87.26% accuracy (previous SOTA: 84.85%).

3.2 Localization and Reasoning

The model successfully localized OOC objects with 63.32% accuracy on MIT-OOC and 83.72% on COCO-OOC. Notably, directly asking Vision-Language Models (like BLIP-2) if a scene was out of context resulted in poor performance (23-26%), highlighting the critical role of the multi-step reasoning and counterfactual verification pipeline.

4. Discussion and Conclusion

The success of this framework demonstrates that GPT-4 possesses a more sophisticated understanding of context than specialized vision models. By moving beyond binary classifications to explanatory reasoning, the system can better handle the nuances of visual scenes.

However, limitations remain. The system is dependent on the initial captioning model; if BLIP-2 fails to describe the anomalous attribute, the reasoning engine cannot detect the error. Furthermore, the computational cost of running multiple large models is high. Future work will focus on improving captioning granularity and optimizing the pipeline for efficiency.

Summary updated

Abstract

We address the problem of detecting out-of-context (OOC) objects in a scene. Given an image, we aim to detect whether the image has objects that are not present in their usual context and localize such OOC objects. Existing approaches for OOC detection rely on defining the common context in terms of the manually constructed features, such as the co-occurrence of objects, spatial relations between objects, and shape and size of the objects, and then learning such context for a given dataset. But context is often nu-anced ranging from very common to very surprising. Further, learned context from specific datasets may not be generalized as datasets may not truly represent the human notion of what is in context. Motivated by the success of large language models and more generally, foundation models (FMs) in common sense reasoning, we investigate the FM's ability to capture a more generalized notion of context. We find that a pre-trained FM, such as GPT-4, provides a more nuanced notion of OOC and enables zero-shot OOC detection when coupled with other pre-trained FMs for caption generation such as BLIP-2, and image in-painting with Sta-ble Diffusion 2.0. Our approach does not need any dataset-specific training. We demonstrate the efficacy of our approach on two OOC object detection datasets, achieving 90.8% zero-shot accuracy on the MIT-OOC dataset and 87.26% on the IJCAI22-COCO-OOC dataset.
BibTeX
@inproceedings{roy2025zero,
  author = {Anirban Roy and Adam D. Cobb and Ramneet Kaur and Sumit Jha and Nathaniel D. Bastian and Alexander M. Berenbeim and Robert H. Thomson and Iain Cruickshank and Alvaro Velasquez and Susmit Jha},
  title = {{Zero-Shot Detection of Out-of-Context Objects Using Foundation Models}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00890},
}
Figure