Abstract
This paper introduces the task of Sketch-Guided Object Localization, where a hand-drawn sketch serves as a query to identify all instances of a target object within a complex natural image. Traditional object localization relies on text or photographic queries, but sketches present unique challenges due to their high level of abstraction, human variability, and the significant domain gap between line drawings and RGB photos. The authors propose a novel end-to-end framework featuring a cross-modal attention mechanism that "primes" a Region Proposal Network (RPN) to generate query-specific proposals. Experimental results on MS-COCO and PASCAL-VOC benchmarks demonstrate that this approach significantly outperforms one-shot detection baselines, particularly in zero-shot scenarios for unseen object categories.
Introduction
While computer vision has made strides in localizing objects via text or photographic queries, using a sketch as a query remains a difficult frontier. Unlike photos, sketches lack texture and color, and unlike text, they provide spatial cues that are often distorted or simplified.
The primary hurdles identified in this work include:
- Abstraction: Sketches are sparse representations lacking the rich detail of natural images.
- Variability: Individual drawing styles vary greatly, leading to inconsistent query representations.
- Domain Gap: The statistical distribution of black-and-white strokes differs fundamentally from dense RGB pixel data.
Methodology
The proposed framework addresses these challenges through a two-stage process: query-guided proposal generation and proposal scoring.
Query-Guided Proposal Generation
Standard detectors like Faster R-CNN utilize a class-agnostic RPN. To make the RPN query-aware, the authors introduce a Cross-modal Attention module. Two ResNet-50 backbones extract features from the image and the sketch. A compatibility score is computed between the global sketch vector and local image patches, creating an attention map. This map is fused with the image features, ensuring the RPN prioritizes regions matching the sketch's spatial characteristics.
Proposal Scoring and Multi-Query Fusion
Generated proposals are cropped and ranked against the sketch query using a learned scoring function optimized with a margin-based ranking loss. To mitigate the noise of individual drawing styles, the authors also explore multi-query fusion. By aggregating information from multiple sketches (e.g., through Attention Fusion), the model achieves a more robust consensus of the user's intent.
Experimental Results
The model was evaluated using sketches from the 'Quick, Draw!' dataset paired with natural images from MS-COCO and PASCAL-VOC.
- Seen Categories: On MS-COCO, the method achieved 50.0% AP@50, a substantial improvement over standard one-shot baselines like Matchnet (48.5%) and modified Faster R-CNN (31.5%).
- Zero-Shot Localization: For categories not seen during training, the model reached 15.0% AP@50, outperforming baselines by approximately 3%. This highlights the effectiveness of RPN priming for generalization.
- Multi-Query Benefits: Increasing the number of sketch queries from one to five improved mAP from 0.30 to 0.32, validating the strategy of filtering out individual drawing quirks through fusion.
Conclusion
Sketch-guided localization is a promising yet challenging task. By injecting query information directly into the proposal generation phase via cross-modal attention, this work bridges the gap between abstract sketches and natural images. Future research may focus on more sophisticated sketch encoders, such as Transformers or stroke-based models, to further narrow the performance gap between seen and unseen object categories.
Summary updated