Percept, Memory, and Imagine: World Feature Simulating for Open-Domain Unknown Object Detection

Aming Wu, Cheng DengCVPR 2025 · OpenReview

Abstract

To accelerate the safe deployment of object detectors, we focus on reducing the impact of both covariate and semantic shifts. We consider a realistic yet challenging scenario, Open-Domain Unknown Object Detection (ODU-OD), which aims to detect unknown objects in unseen target domains without accessing any auxiliary data. Inspired by human cognition, we propose World Feature Simulation (WFS), which utilizes multi-level perception, a memory recorder, and an unknown-feature generator. By synthesizing virtual features that deviate from in-distribution objects, WFS learns a robust discriminative boundary. Significant performance gains across four detection tasks demonstrate the superiority of our method. The code is available at https://github.com/AmingWu/WFS.

Introduction: The Challenge of ODU-OD

Standard object detectors are often fragile when encountering real-world shifts. These shifts typically fall into two categories: covariate shift, where the environment or style changes (e.g., sunny vs. snowy weather), and semantic shift, where entirely new object classes appear (e.g., a moose appearing on a road where only cars were expected).

While existing research often addresses these shifts in isolation, real-world applications like autonomous driving require handling both simultaneously. This paper introduces Open-Domain Unknown Object Detection (ODU-OD), a task where a model must identify novel objects in unseen environments without the benefit of auxiliary data.

Conceptual overview of the WFS framework mapping cognitive processes to the detection task.

Methodology: World Feature Simulation (WFS)

To solve ODU-OD, the authors propose the World Feature Simulation (WFS) framework. WFS mimics human cognitive processes—perception, memory, and imagination—to synthesize virtual features of unknown objects, thereby training the model to recognize what it hasn't seen.

Multi-level Perception

WFS moves beyond standard convolutional features by employing a dual-stream perception module. It uses a Mamba block to capture global context and a Graph Network to model connective relationships between image regions. This provides a holistic representation of objects, making the model more robust to local variations.

Memory Recorder and Imagination Bank

The system maintains a Memory Recorder consisting of a learnable codebook of "codewords." These codewords store fragmented characteristics of known objects. To simulate unknown objects, the model performs a "modulated operation" on these memories, creating an Imagination Bank. This bank contains features that share some characteristics with known objects but are intentionally perturbed to represent potential outliers.

Detailed architecture of the WFS framework, featuring multi-level representation and the unknown-feature generator.

Unknown-Feature Generator

A dedicated recurrent generator synthesizes virtual "unknown" features from the imagination bank. During training, the model is tasked with maximizing the divergence between these synthesized features and known in-distribution features. This process allows the model to learn a sharp decision boundary between "known" and "unknown" categories.

Experiments and Results

The WFS framework was evaluated across four benchmarks: ODU-OD, Out-of-Distribution Object Detection (OOD-OD), Open-Set Object Detection (OSOD), and Single-Domain Generalized Object Detection (Single-DGOD).

  • ODU-OD Performance: On the BDD dataset transformed into an 'Acrylic' style, WFS reduced the False Positive Rate (FPR95) from 65.71% to 56.48% when integrated with the VOS baseline.
  • Open-Set Detection: In the VOC-COCO-60 benchmark, WFS achieved significant improvements in Mean Average Precision (mAP) and reduced Absolute Open-Set Error (AOSE) compared to state-of-the-art methods like OpenDet.
  • Robustness: Qualitative results show that WFS effectively flags out-of-distribution objects (like a moose in a sketch style) as "unknown," whereas baseline models often misclassify them with high confidence.

Conclusion

The World Feature Simulation (WFS) approach provides a robust solution for detecting unknown objects in open domains. By leveraging multi-level perception and synthesized "imagination" features, WFS effectively bridges the gap between known training data and the unpredictable nature of real-world environments. Future work may involve optimizing the computational efficiency of the recurrent generator and exploring the application of WFS to complex autonomous world models.

Summary updated

Abstract

To accelerate the safe deployment of object detectors, we focus on reducing the impact of both covariate and semantic shifts. And we consider a realistic yet challenging scenario, namely Open-Domain Unknown Object Detection (ODU-OD), which aims to detect unknown objects in unseen target domains without accessing any auxiliary data. Towards ODU-OD, it is feasible to learn a robust discriminative boundary by synthesizing virtual features. Generally, perception, memory, and imagination are three essential capacities for human beings. Through multi-level perception and rich memory about known objects, the characteristics of unknown objects can be imagined sufficiently, enhancing the ability of discriminating known from unknown objects. Inspired by this idea, an approach of World Feature Simulation (WFS) is proposed, mainly consisting of a multi-level perception, memory recorder, and unknown-feature generator. Specifically, after extracting the features of the input, we separately employ a Mamba and Graph Network to obtain the global-level and connective-level representations. Next, a codebook containing multiple learnable codewords is defined to preserve fragmented memory of known objects. Meanwhile, we perform a modulated operation on the memory to form the imagination bank involving unknown characteristics. Finally, to alleviate the impact of lacking supervision data, based on the multi-level representation and imagination bank, a dedicated unknown-feature generator is designed to recurrently synthesize outlier features deviating from in-distribution (ID) objects. The significant performance gains on four different detection tasks demonstrate the superiorities of our method. The code will be released at https://github.com/AmingWu/WFS.
BibTeX
@inproceedings{wu2025percept,
  author = {Aming Wu and Cheng Deng},
  title = {{Percept, Memory, and Imagine: World Feature Simulating for Open-Domain Unknown Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=QNZfaXedRc},
}
Figure