Revisiting Self-Similarity: Structural Embedding for Image Retrieval

Seongwon Lee, Suhyeon Lee, Hongje Seong, Euntai KimCVPR 2023 · DOI

Abstract

Traditional global descriptors in image retrieval often fail to distinguish between objects with similar textures but different geometric structures. We present the Structural Embedding Network (SENet), which integrates a differentiable self-similarity module into standard backbone architectures. By capturing local geometric patterns through a novel convolutional lens, SENet produces global descriptors that are significantly more robust against "look-alike" distractors. Our approach achieves state-of-the-art performance on the Revisited Oxford and Paris benchmarks, particularly in the presence of large-scale distractors.

Introduction

In content-based image retrieval, the standard pipeline involves a fast initial search using global descriptors followed by a precise, but computationally expensive, local re-ranking stage. While global descriptors effectively capture visual properties like color and texture, they often lack the geometric awareness necessary to distinguish between objects with similar appearances but different structures. This leads to the "look-alike" problem, where a triangular roof might be confused with the Louvre Pyramid due to shared textural features.

Comparison of visual properties and self-similarity for a query image (Louvre), a positive match, and a negative distractor (CCTV building).

To address this, we revisit the classic concept of self-similarity. By focusing on how a pixel relates to its neighbors rather than its raw appearance, we can extract a geometric "fingerprint" that is more robust to domain shifts and textural distractors.

Methodology

SENet introduces three primary modules designed to augment standard backbone networks such as ResNet:

  1. Self-Similarity Computation (SSC): This module calculates a self-similarity map for every pixel by evaluating its relationship with its local neighborhood using a convolutional approach.
  2. Self-Similarity Encoder (SSE): To handle the high-dimensional and noisy nature of raw similarity maps, the SSE employs a sequence of 4D-like convolutions. These 3x3 kernels compress the information into a dense, learnable structural descriptor.
  3. Feature Fusion Module (FFM): This module harmoniously combines the new structural descriptor with original visual features. It utilizes a specific initialization strategy—setting Batch Normalization parameters to zero—to ensure that the structural cues do not disrupt the pre-trained knowledge of the base network.

The entire pipeline is end-to-end differentiable, allowing the network to learn the most discriminative structural patterns directly from data.

Experiments and Results

Datasets and Training

The model was trained on the Google Landmarks Dataset v2 (GLDv2-clean), comprising 1.5 million images across 81k landmarks. Evaluation was conducted on the Revisited Oxford (ROxf) and Revisited Paris (RPar) datasets, including the +1M distractor set to simulate real-world search conditions.

Performance

SENet achieved state-of-the-art (SOTA) results across multiple benchmarks:

  • Accuracy: On the ROxf "Hard" setup, SENet (R101) reached a 66.0 mAP, significantly outperforming previous global retrieval methods.
  • Robustness: The performance gap widened in the presence of distractors. On RPar-Hard+1M, SENet improved performance by up to 8.7% over standard baselines.
  • Efficiency: As a global single-pass method, SENet provides high precision without the computational overhead of traditional local-feature re-ranking.

Conclusion

SENet demonstrates that embedding structural information directly into global descriptors significantly enhances image retrieval robustness. By making self-similarity differentiable and learnable, we bridge the gap between fast global search and precise geometric verification. Future work will focus on improving the network's resilience to extreme viewpoint shifts and variations in image resolution.

Summary updated

BibTeX
@inproceedings{lee2023revisiting,
  author = {Seongwon Lee and Suhyeon Lee and Hongje Seong and Euntai Kim},
  title = {{Revisiting Self-Similarity: Structural Embedding for Image Retrieval}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02242},
}
Figure