Standard object detectors typically operate under a "closed-world" assumption: they are trained to recognize a fixed list of categories (e.g., "dog," "car," "chair") and treat everything else as background. However, in the real world, autonomous systems and robots constantly encounter "novel" objects they were never explicitly trained to recognize.
To address this, researchers have introduced GOOD (Geometry-guided Open-world Object Detector), a novel approach that tackles the "appearance bias" inherent in traditional object detection. By leveraging geometric cues like depth and surface normals, GOOD significantly outperforms state-of-the-art methods in detecting unseen objects, achieving a 24% relative improvement in recall even when trained on just a single object category.
The Problem: The Appearance Trap
The primary challenge with current open-world detectors is their heavy reliance on RGB appearance. Because these models are trained on specific textures, colors, and discriminative parts, they often overfit to visual shortcuts. For example, a model trained on cars might recognize a truck due to visual similarities (like wheels or metallic textures), but it will likely ignore a completely novel object because its appearance doesn't match anything in the model's training history.
To break this dependence on appearance, the authors of GOOD propose looking at the underlying physical structure of the scene.
The Solution: Geometry-Guided Detection
While a "hot dog" and a "refrigerator" look nothing alike in RGB space, they both possess distinct geometric boundaries and surface orientations that separate them from the background. Geometry is inherently class-agnostic.
To harness this, the researchers developed a two-phase training pipeline:
- Phase I: Geometric Pseudo-Labeling: Instead of relying on RGB images, the system uses off-the-shelf monocular estimators (specifically the Omnidata model) to extract depth maps and surface normals from the images. An object proposal network is then trained specifically on these geometric maps. Because geometry (shapes, edges, relative distance) is more universal than texture, this network can find "object-like" regions even for categories it has never seen. These detections serve as "pseudo-labels" for unannotated objects in the training set.
- Phase II: Final Detector Training: The final detector is trained on standard RGB images. However, its supervision comes from a combination of the original ground-truth labels and the high-quality geometric pseudo-labels generated in Phase I.
By ensembling cues from both depth and normals, the model learns a much more robust, class-agnostic sense of "objectness."
Key Insights and Generalization
The researchers evaluated GOOD across several challenging benchmarks, including COCO, PASCAL-VOC, ADE20K, UVO, and LVIS, revealing several key insights:
- Complementary Strengths: RGB is highly effective at detecting very small objects where depth maps might be noisy. Conversely, geometry is far superior for identifying novel shapes and large objects. Combining them provides the best of both worlds.
- Robustness to Semantic Shift: Geometric cues are far less sensitive to "semantic shifts"—such as moving from training on animals to testing on kitchen appliances.
- State-of-the-Art Performance: On the extreme COCO "person-only" task (training only on the "person" class and testing on 79 other categories), GOOD achieved an Average Recall (AR@100) of 25.9% for novel objects. This surpasses the previous state-of-the-art by 5.0%, representing a 24% relative improvement.
- Zero-Shot Capabilities: The model successfully detects "wild" objects it never saw in training simply because they exhibit "object-like" geometry.
Limitations and Future Directions
While GOOD represents a significant step forward for open-world detection, it does have limitations. Geometric cues can become "blurry" or unreliable for very small or thin objects (like utensils), leading to lower recall compared to RGB-based methods for tiny items.
Furthermore, the system's performance is inherently tied to the quality of the monocular depth and normal estimators. If these estimators fail—such as in extremely low light or on highly reflective surfaces—the resulting pseudo-labels will degrade. The two-phase training also adds computational overhead, though the authors note that at inference time, the model only requires the RGB image.
Future work may focus on integrating geometric estimation directly into an end-to-end trainable framework, potentially improving both efficiency and accuracy for real-world robotic and autonomous applications.
Summary updated