Abstract
Standard object detection assumes a closed-set world with static environments. However, real-world deployment often involves both domain shifts (e.g., weather changes) and the appearance of novel, unseen object classes. This paper formalizes the task of Adaptive Open-set Object Detection (AOOD) and proposes SOMA (Structured Motif Matching). SOMA leverages high-order graph motifs to distinguish novel objects from environmental noise and known classes. By matching these structural patterns across domains, SOMA achieves state-of-the-art performance in identifying base classes while successfully discovering novel ones in challenging cross-domain scenarios.
Introduction: The Challenge of Open-Set Adaptation
In traditional computer vision, models operate under the "closed-set" assumption: all classes encountered during testing were seen during training. Domain Adaptation (DA) relaxes the environment constraint but maintains the class constraint. Real-world applications, such as autonomous driving, face a more complex reality where both the scene and the objects can be novel.
Existing methods struggle with this dual challenge because domain shifts and novel objects both push feature representations into low-confidence regions. Simple metrics like Euclidean distance or confidence scores often fail to differentiate between a degraded known object (e.g., a car in heavy fog) and a clear unknown object (e.g., a novel street sign).
The SOMA Framework
The authors introduce SOMA (Structured Motif Matching), built upon the Deformable DETR architecture. The key innovation is the shift from low-order relations (pairwise distances) to high-order relations using graph motifs—statistically significant subgraphs, specifically triangles, that capture complex feature space patterns.
SOMA operates through two primary modules:
1. Structure-aware Novel-class Learning (SNL)
SNL functions in the source domain to teach the model the characteristics of "unknown" objects. It constructs an instance-oriented graph connecting known class centers. Using a high-order metric ($\Gamma_{IG}$), it identifies motifs that capture class-agnostic features (like generic object boundaries). This allows the model to categorize novel objects as "something" rather than dismissing them as background noise.
2. Structure-aware Transfer Learning (STL)
STL handles the domain shift in the target environment. It builds a semantic-oriented graph using class "extremes"—the peripheral points of a class's distribution. By matching motifs between source and target domains, STL ensures "structural awareness" during adaptation. This prevents the model from collapsing novel class features into known class clusters during the alignment process.
Experimental Evaluation
The authors established three AOOD benchmarks using Cityscapes, Foggy Cityscapes, Pascal VOC, Clipart, and BDD100k. These benchmarks cover weather, style, and geographic shifts.
Key Results
SOMA demonstrates significant improvements over baseline DAOD and Open-set detectors:
- Base Class Performance: On the Cityscapes $\rightarrow$ Foggy Cityscapes task, SOMA improved base class mAP from 42.38% to 50.87%.
- Novel Class Discovery: SOMA achieved a novel class recall (ARn) of up to 15.67%, whereas many baselines failed to detect novel classes entirely (0% recall).
- Error Reduction: The framework significantly reduced Absolute Open-Set Error (AOSE), effectively preventing the misclassification of unknown objects (like dustbins) as known categories (like cars).
Discussion and Future Work
SOMA's success stems from its "structural signature" approach. By looking for motifs that sit between disparate class centers, it identifies candidates for novel classes with higher stability than distance-based methods.
However, challenges remain. Graph matching and motif traversal increase computational overhead, though the authors mitigate this by restricting SOMA to the final decoder layer. Additionally, the current framework treats all novel objects as a single "unknown" category. Future research could extend this to Open-World Detection, where the model incrementally learns to distinguish between different types of novel objects over time.
Summary updated