Abstract
Zero-Shot Object Detection (ZSD) aims to recognize and localize objects from classes not seen during training by leveraging auxiliary semantic information. Traditional methods often struggle with the visual-semantic gap, where visual appearance deviates significantly from semantic prototypes. This paper introduces a Dual-Path Inference Fusion (DPIF) framework that mimics human cognitive processes by incorporating an "associative semantics" path alongside standard visual-semantic mapping. By reasoning through analogy with seen objects, the proposed model achieves state-of-the-art performance on MSCOCO and Pascal VOC benchmarks, outperforming even transductive methods.
Introduction
Standard object detection models are limited by their dependence on large-scale labeled datasets for every target category. Zero-Shot Object Detection (ZSD) seeks to overcome this by enabling the detection of "unseen" objects using text descriptions or attribute vectors. However, a significant challenge remains: the visual-semantic gap. Direct mapping from visual features to semantic embeddings often fails when an object's appearance is deceptive. For instance, a "leaf butterfly" may visually resemble a dry leaf more than a typical butterfly, leading to misclassification in single-path models.
To address this, we propose a system that supplements direct semantic mapping with a "reasoning by analogy" path, establishing target-centric associations between seen and unseen classes to provide semantic compensation.
Methodology: Dual-Path Inference Fusion
The proposed framework, Dual-Path Inference Fusion (DPIF), is built upon the Faster R-CNN architecture and introduces two parallel reasoning paths for classification.
1. Visual-Semantic (V-S) Mapping
This path follows the traditional ZSD approach, aligning Region of Interest (RoI) features with class embeddings (e.g., word vectors). It provides a direct prediction based on the learned relationship between visual features and semantic prototypes.
2. Multi-Associative Semantic Transfer (MAST)
The core innovation is the MAST path, which implements "reasoning by analogy." It leverages Target-Centric Multi-Association (TCMA) to establish relationships between seen and unseen classes. Instead of generic clustering, TCMA uses unseen classes as anchors to identify the most relevant seen "proxy" classes. This ensures that the model can learn to recognize unseen concepts by their proximity to known concepts in the semantic space.
Inference Fusion
During the inference stage, the model generates scores from both paths. These scores are fused using a mechanism that weighs the primary classification by the associative scores. This allows the "analogy" path to act as a regularizer, cross-checking and refining the direct mapping results.
Experimental Results
The DPIF model was evaluated on MSCOCO, Pascal VOC, and the large-scale Visual Genome dataset.
- MSCOCO: The multi-association model (DPIF-M) achieved 19.82% mAP, representing a 7.42% absolute improvement over previous inductive models and surpassing leading transductive competitors by 5.25%.
- Pascal VOC: The model demonstrated superior performance in the Generalized ZSD setting, effectively detecting both seen and unseen objects simultaneously.
- Visual Genome: On this complex dataset with hundreds of classes, the model achieved an mAP of 1.81%, a significant advancement in large-scale ZSD.
Discussion and Conclusion
The success of DPIF highlights the power of analogy as a semantic regularizer. By forcing the model to learn associations between concepts, it compensates for the visual-semantic gap that plagues direct-mapping approaches. Notably, this inductive approach outperforms several transductive models that utilize unlabeled test data during training.
Future research will focus on improving the class-agnostic nature of the Region Proposal Network (RPN). Currently, unseen objects are often treated as background during RPN training, which can prevent the model from even proposing bounding boxes for them. Enhancing the RPN's ability to generalize to novel objects is a critical next step for robust Zero-Shot Object Detection.
Summary updated