Photo Pre-Training, But for Sketch

Ke Li, Kaiyue Pang, Yi-Zhe Song

Abstract Fine-grained sketch-based image retrieval (FG-SBIR) is traditionally limited by a scarcity of paired sketch-photo data. While photo-based pre-training is standard, it is typically treated as a one-time parameter initialization. This paper proposes a paradigm shift where pre-training provides enduring supervision throughout the fine-tuning process. By enforcing Neighborhood Topology (NT) compliance—ensuring the structural relationships learned from millions of photos are preserved during sketch training—we achieve significant performance gains. Our method establishes a new state-of-the-art across five benchmarks, notably surpassing human performance on the QMUL-Shoe-V2 dataset.

Introduction

The "sketch community" faces a chronic data drought. Unlike the millions of labeled photos available in datasets like ImageNet, high-quality, fine-grained sketch-photo pairs number only in the thousands. This disparity forces researchers to rely on photo-based pre-training. Conventionally, this is a "one-and-done" process: models are initialized with weights from a photo classifier and then fine-tuned on sketches. However, this often leads to "catastrophic forgetting" of the rich visual relationships learned from photos, as the model overfits to the small sketch dataset.

Methods

We introduce the principle of Neighborhood Topology (NT) compliance to transform pre-training into a continuous source of supervision.

Conceptual shift from one-time initialization to enduring supervision via Neighborhood Topology.
  1. Topology as Supervision ($L_{NT}$): Instead of relying solely on triplet losses to align specific pairs, we extract a "ranking matrix" from the pre-trained photo model. This matrix defines the relative distances between photos in the feature space. We then enforce a constraint: the model must maintain this relative ordering even when processing sketches during fine-tuning.
  2. Cross-Domain Adaptation: To prevent feature drift between domains, we substitute the "anchor" photo in the loss function with its corresponding sketch. This forces the model to respect photo-space topology while operating within the sketch domain.
  3. Meta-Learning Objective: We treat $L_{NT}$ as a meta-objective rather than a simple auxiliary task. By simulating a "look-ahead" step, the model evaluates if a sketch-retrieval update compromises the preserved photo topology. This "learning-to-learn" approach provides greater stability and synergy between tasks compared to standard multi-task learning.

Key Results

Our approach yields substantial improvements across five product-level FG-SBIR benchmarks: QMUL-Shoe-V1/V2, QMUL-Chair-V1/V2, and QMUL-Handbag.

Comparison of the proposed method against state-of-the-art FG-SBIR techniques and human performance.

As shown in the results table, our method consistently outperforms previous state-of-the-art (SoTA) techniques. On the challenging QMUL-Shoe-V2 dataset, we achieved an acc@1 of 50.75%, a significant leap from the previous best of approximately 43%. Notably, our system is the first to outperform human subjects in a trial study for fine-grained shoe retrieval.

Beyond raw accuracy, the model demonstrates increased robustness to "messy" or incomplete sketches, maintaining successful retrieval even when 30% of strokes are removed. Qualitatively, the system produces more visually coherent retrieval galleries, reflecting a deeper understanding of the underlying product topology.

Conclusion

By reimagining pre-training as an enduring resource rather than a mere starting point, we bridge the data gap in sketch-based retrieval. The Neighborhood Topology compliance framework allows models to adapt to hand-drawn styles while remaining grounded in the rich visual structures of the photo domain. Future work will explore integrating additional "foundation" tasks, such as depth estimation or segmentation, as further sources of enduring supervision.

Summary updated

Abstract

The sketch community has faced up to its unique challenges over the years, that of data scarcity however still remains the most significant to date. This lack of sketch data has imposed on the community a few “peculiar” design choices - the most representative of them all is perhaps the coerced utilisation of photo-based pre-training (i.e., no sketch), for many core tasks that otherwise dictates specific sketch understanding. In this paper, we ask just the one question - can we make such photo-based pre-training, to actually benefit sketch? Our answer lies in cultivating the topology of photo data learned at pre-training, and use that as a “free” source of supervision for downstream sketch tasks. In particular, we use fine-grained sketch-based image retrieval (FG-SBIR), one of the most studied and data-hungry sketch tasks, to showcase our new perspective on pre-training. In this context, the topology-informed supervision learned from photos act as a constraint that take effect at every fine-tuning step - neighbouring photos in the pre-trained model remain neighbours under each FG-SBIR updates. We further portray this neighbourhood consistency constraint as a photo ranking problem and formulate it into a neat cross-modal triplet loss. We also show how this target is better leveraged as a meta objective rather than optimised in parallel with the main FG-SBIR objective. With just this change on pre-training, we beat all previously published results on all five product-level FG-SBIR benchmarks with significant margins (sometimes >10%). And the most beautiful thing, as we note, is such gigantic leap is made possible within just a few extra lines of code! Our implementation is available at https://github.com/KeLi-SketchX/Photo-Pre-Training-But-for-Sketch
BibTeX
@inproceedings{li2023photo,
  author = {Ke Li and Kaiyue Pang and Yi-Zhe Song},
  title = {{Photo Pre-Training, But for Sketch}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00270},
}
Figure