Abstract Fine-grained sketch-based image retrieval (FG-SBIR) is traditionally limited by a scarcity of paired sketch-photo data. While photo-based pre-training is standard, it is typically treated as a one-time parameter initialization. This paper proposes a paradigm shift where pre-training provides enduring supervision throughout the fine-tuning process. By enforcing Neighborhood Topology (NT) compliance—ensuring the structural relationships learned from millions of photos are preserved during sketch training—we achieve significant performance gains. Our method establishes a new state-of-the-art across five benchmarks, notably surpassing human performance on the QMUL-Shoe-V2 dataset.
Introduction
The "sketch community" faces a chronic data drought. Unlike the millions of labeled photos available in datasets like ImageNet, high-quality, fine-grained sketch-photo pairs number only in the thousands. This disparity forces researchers to rely on photo-based pre-training. Conventionally, this is a "one-and-done" process: models are initialized with weights from a photo classifier and then fine-tuned on sketches. However, this often leads to "catastrophic forgetting" of the rich visual relationships learned from photos, as the model overfits to the small sketch dataset.
Methods
We introduce the principle of Neighborhood Topology (NT) compliance to transform pre-training into a continuous source of supervision.
- Topology as Supervision ($L_{NT}$): Instead of relying solely on triplet losses to align specific pairs, we extract a "ranking matrix" from the pre-trained photo model. This matrix defines the relative distances between photos in the feature space. We then enforce a constraint: the model must maintain this relative ordering even when processing sketches during fine-tuning.
- Cross-Domain Adaptation: To prevent feature drift between domains, we substitute the "anchor" photo in the loss function with its corresponding sketch. This forces the model to respect photo-space topology while operating within the sketch domain.
- Meta-Learning Objective: We treat $L_{NT}$ as a meta-objective rather than a simple auxiliary task. By simulating a "look-ahead" step, the model evaluates if a sketch-retrieval update compromises the preserved photo topology. This "learning-to-learn" approach provides greater stability and synergy between tasks compared to standard multi-task learning.
Key Results
Our approach yields substantial improvements across five product-level FG-SBIR benchmarks: QMUL-Shoe-V1/V2, QMUL-Chair-V1/V2, and QMUL-Handbag.
As shown in the results table, our method consistently outperforms previous state-of-the-art (SoTA) techniques. On the challenging QMUL-Shoe-V2 dataset, we achieved an acc@1 of 50.75%, a significant leap from the previous best of approximately 43%. Notably, our system is the first to outperform human subjects in a trial study for fine-grained shoe retrieval.
Beyond raw accuracy, the model demonstrates increased robustness to "messy" or incomplete sketches, maintaining successful retrieval even when 30% of strokes are removed. Qualitatively, the system produces more visually coherent retrieval galleries, reflecting a deeper understanding of the underlying product topology.
Conclusion
By reimagining pre-training as an enduring resource rather than a mere starting point, we bridge the data gap in sketch-based retrieval. The Neighborhood Topology compliance framework allows models to adapt to hand-drawn styles while remaining grounded in the rich visual structures of the photo domain. Future work will explore integrating additional "foundation" tasks, such as depth estimation or segmentation, as further sources of enduring supervision.
Summary updated