Learning to rank approach for refining image retrieval in visual arts

Tetiana Yemelianenko, Iuliia Tkachenko, Tess Masclef, Mihaela Scuturici, Serge MiguetWorkshops @ ICCV 2023 · DOI

Abstract

Content-based image retrieval (CBIR) often fails to bridge the "intention gap" in specialized domains like art history, where expert criteria for similarity differ from general visual features. This paper proposes a Learning to Rank (LTR) framework using LambdaMART to re-rank initial search results. By simulating expert preferences based on color palette and object composition, the study demonstrates that retrieval systems can be effectively tailored to mimic expert behavior using small, specialized datasets, significantly improving retrieval accuracy as measured by NDCG.

Introduction

Standard content-based image retrieval (CBIR) systems, typically trained on general photographic datasets like ImageNet, excel at identifying broad visual similarities but frequently fall short in the context of visual arts. Art historians often prioritize non-obvious, domain-specific criteria such as specific iconographic objects, compositional structures, or nuanced color usage. Formalizing these expert preferences into a search algorithm is challenging, and retraining deep neural networks for every new criterion is computationally prohibitive. This research addresses the "intention gap" by refining general search results into specialized rankings that respect expert-defined criteria.

Methodology

The authors implement a two-stage retrieval pipeline. First, an initial set of $N=15$ images is retrieved using a ResNet50 model (fine-tuned for genre classification) to extract 512-dimensional feature vectors, followed by the ANNOY (Approximate Nearest Neighbors Oh Yeah) algorithm for efficient similarity search. Second, a Learning to Rank (LTR) refinement stage employs LambdaMART, a state-of-the-art gradient-boosted decision tree algorithm, to re-order these results based on specific expert criteria.

Comparison of Pointwise, Pairwise, and Listwise ranking approaches in Learning to Rank.

The LTR model was trained using simulated expert interactions based on two distinct criteria:

  • Color Palette Similarity: Ranking paintings by their dominant color proportions using the CIE76 formula.
  • Object Similarity: Ranking based on shared objects (e.g., "imaginary beings" or "human-like poses") using manual annotations from the DeArt dataset.

The study utilized the WikiArt dataset (~81,000 paintings) for color simulations and the DeArt dataset (>15,000 paintings) for object-based re-ranking, the latter providing 69 classes specific to cultural heritage.

Results and Discussion

Performance was evaluated using Normalized Discounted Cumulative Gain (NDCG), where a score of 1.0 indicates a perfect match to the expert ranking.

Evaluation of ranking quality using NDCG scores compared to an expert's ground truth.

In the color palette task, the LTR model achieved an NDCG@15 of 0.8848, outperforming the initial retrieval (0.8377). The improvement was more pronounced in the object similarity task, where the LTR model reached an NDCG@15 of 0.9552, compared to 0.8473 for the baseline. Furthermore, the LTR model placed the most relevant item at the top spot 86.77% of the time for object similarity, a substantial increase over the 59.41% achieved by standard retrieval.

These results suggest that while lower-level features like color are somewhat captured by initial deep learning models, higher-level semantic features (like object similarity) benefit immensely from LTR refinement. This approach allows systems to adapt to different user groups—from casual museum visitors to specialized researchers—without the need for expensive retraining of the underlying feature extraction models.

Conclusion

The proposed LTR framework effectively bridges the intention gap in art image retrieval by leveraging "shallow" models like LambdaMART to learn complex expert preferences from small datasets. While the current study relies on simulated criteria, future work will involve training on real-world rankings from art history specialists and integrating these capabilities into public-facing tools for museums and researchers.

Summary updated

BibTeX
@inproceedings{yemelianenko2023learning,
  author = {Tetiana Yemelianenko and Iuliia Tkachenko and Tess Masclef and Mihaela Scuturici and Serge Miguet},
  title = {{Learning to rank approach for refining image retrieval in visual arts}},
  booktitle = {Workshops @ ICCV},
  year = {2023},
  doi = {10.1109/iccvw60793.2023.00177},
}
Figure