Abstract
To accelerate the safe deployment of object detectors, we focus on reducing the impact of both covariate and semantic shifts. We consider a realistic yet challenging scenario, Open-Domain Unknown Object Detection (ODU-OD), which aims to detect unknown objects in unseen target domains without accessing any auxiliary data. Inspired by human cognition, we propose World Feature Simulation (WFS), which utilizes multi-level perception, a memory recorder, and an unknown-feature generator. By synthesizing virtual features that deviate from in-distribution objects, WFS learns a robust discriminative boundary. Significant performance gains across four detection tasks demonstrate the superiority of our method. The code is available at https://github.com/AmingWu/WFS.
Introduction: The Challenge of ODU-OD
Standard object detectors are often fragile when encountering real-world shifts. These shifts typically fall into two categories: covariate shift, where the environment or style changes (e.g., sunny vs. snowy weather), and semantic shift, where entirely new object classes appear (e.g., a moose appearing on a road where only cars were expected).
While existing research often addresses these shifts in isolation, real-world applications like autonomous driving require handling both simultaneously. This paper introduces Open-Domain Unknown Object Detection (ODU-OD), a task where a model must identify novel objects in unseen environments without the benefit of auxiliary data.
Methodology: World Feature Simulation (WFS)
To solve ODU-OD, the authors propose the World Feature Simulation (WFS) framework. WFS mimics human cognitive processes—perception, memory, and imagination—to synthesize virtual features of unknown objects, thereby training the model to recognize what it hasn't seen.
Multi-level Perception
WFS moves beyond standard convolutional features by employing a dual-stream perception module. It uses a Mamba block to capture global context and a Graph Network to model connective relationships between image regions. This provides a holistic representation of objects, making the model more robust to local variations.
Memory Recorder and Imagination Bank
The system maintains a Memory Recorder consisting of a learnable codebook of "codewords." These codewords store fragmented characteristics of known objects. To simulate unknown objects, the model performs a "modulated operation" on these memories, creating an Imagination Bank. This bank contains features that share some characteristics with known objects but are intentionally perturbed to represent potential outliers.
Unknown-Feature Generator
A dedicated recurrent generator synthesizes virtual "unknown" features from the imagination bank. During training, the model is tasked with maximizing the divergence between these synthesized features and known in-distribution features. This process allows the model to learn a sharp decision boundary between "known" and "unknown" categories.
Experiments and Results
The WFS framework was evaluated across four benchmarks: ODU-OD, Out-of-Distribution Object Detection (OOD-OD), Open-Set Object Detection (OSOD), and Single-Domain Generalized Object Detection (Single-DGOD).
- ODU-OD Performance: On the BDD dataset transformed into an 'Acrylic' style, WFS reduced the False Positive Rate (FPR95) from 65.71% to 56.48% when integrated with the VOS baseline.
- Open-Set Detection: In the VOC-COCO-60 benchmark, WFS achieved significant improvements in Mean Average Precision (mAP) and reduced Absolute Open-Set Error (AOSE) compared to state-of-the-art methods like OpenDet.
- Robustness: Qualitative results show that WFS effectively flags out-of-distribution objects (like a moose in a sketch style) as "unknown," whereas baseline models often misclassify them with high confidence.
Conclusion
The World Feature Simulation (WFS) approach provides a robust solution for detecting unknown objects in open domains. By leveraging multi-level perception and synthesized "imagination" features, WFS effectively bridges the gap between known training data and the unpredictable nature of real-world environments. Future work may involve optimizing the computational efficiency of the recurrent generator and exploring the application of WFS to complex autonomous world models.
Summary updated