ARNet: Self-Supervised FG-SBIR with Unified Sample Feature Alignment and Multi-Scale Token Recycling

Jianan Jiang, Hao Tang, Zhilin Jiang, Weiren Yu, Di WuAAAI 2025 · DOI

Introduction

Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) is the challenging task of matching a specific hand-drawn sketch to its exact photographic counterpart. Unlike general retrieval, FG-SBIR requires capturing minute details to distinguish between highly similar objects. Current research in this field faces three primary hurdles: early training saturation in standard models, the sensitivity and instability of traditional triplet loss functions, and the inefficient use of spatial information in Vision Transformers (ViTs), where localized "patch tokens" are typically discarded in favor of a single global [CLS] token.

Methodology

ARNet addresses these issues through a novel architecture featuring dual weight-sharing networks and a specialized token recycling mechanism designed to bridge the gap between abstract sketches and detailed photographs.

Dual Weight-Sharing and Contrastive Alignment

To overcome model saturation, ARNet employs two identical networks that share weights. This structure facilitates a unified contrastive alignment strategy. Unlike traditional triplet loss, which often fails to provide strong enough gradient signals for optimal alignment, ARNet's approach simultaneously optimizes intra-sample (sketch vs. augmented sketch) and inter-sample (sketch vs. image) distributions. This provides more stable gradients and prevents the performance plateaus common in single-branch encoders.

ARNet Architecture and MSTR Module

Multi-Scale Token Recycling (MSTR)

A core innovation of ARNet is the Multi-Scale Token Recycling (MSTR) module. Standard ViT-based retrieval methods rely solely on the [CLS] token for final representation, discarding patch tokens that contain high-resolution local details. MSTR salvages these discarded tokens, projecting them across three scales to create a Multi-Scale Contrast Map (MAP). These recycled features are then fused back into the final embedding, ensuring the representation captures the fine-grained "fingerprint" of the object.

Experimental Results

ARNet was evaluated on standard benchmarks (QMUL-Chair-V2, QMUL-Shoe-V2) and a newly introduced professional fashion dataset, Clothes-V1, which contains 1,200 professional sketches and 500 images.

  • QMUL-Chair-V2: Achieved a state-of-the-art 75.45% Acc@1, a significant jump over the previous best of 71.22%.
  • Clothes-V1: Demonstrated dominant performance with 95.27% Acc@1, proving the method's effectiveness on professional-grade fashion data.
  • Ablation Studies: The MSTR module alone consistently provided a ~2% boost in top-1 accuracy, confirming that recycling discarded data is a viable strategy for performance gains.

Key Insights and Analysis

Statistical analysis of the feature maps reveals that the MSTR module significantly enhances the diversity of the learned representations. By recycling discarded tokens, ARNet increases feature variance by 36.87%. This increased diversity allows the model to distinguish between highly similar objects more effectively by utilizing unique semantic information that was previously ignored.

Comparison of Feature Diversity and Variance

Furthermore, the researchers discovered that training saturation was a structural issue rather than a data limitation. Moving to the dual-network setup allowed the model to continue learning effectively up to 500 epochs, whereas previous methods often plateaued at the 150-epoch mark.

Conclusion

ARNet introduces a robust framework for FG-SBIR by rethinking how information is processed in Vision Transformers and how models are aligned during training. By recycling "useless" tokens and employing a unified contrastive loss, it sets new performance standards for fine-grained retrieval. Future work may involve applying the MSTR module to other computer vision tasks, such as image generation or fine-grained classification, where local patch details are equally critical.

Summary updated

BibTeX
@inproceedings{jiang2025arnet,
  author = {Jianan Jiang and Hao Tang and Zhilin Jiang and Weiren Yu and Di Wu},
  title = {{ARNet: Self-Supervised FG-SBIR with Unified Sample Feature Alignment and Multi-Scale Token Recycling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i4.32417},
}
Figure