Syntharch: Interactive Image Search with Attribute-Conditioned Synthesis

Zac Yu, Adriana Kovashka

Abstract

Interactive systems are promising for content-based image retrieval (CBIR), allowing users to iteratively refine search results. Conventional methods often rely on shared semantic knowledge, such as visual attributes, which introduces a "semantic gap" and requires labeled databases. We present Syntharch, a novel approach that elicits feedback exclusively through AI-synthesized images. By generating options that differ along isolated attribute dimensions, Syntharch eliminates the semantic gap and reduces user confusion caused by attribute correlation in real images. Experimental results on the UT-Zap50K dataset show that Syntharch achieves superior accuracy and consistency compared to retrieval-based baselines.

Introduction

Content-Based Image Retrieval (CBIR) involves locating a specific target image within a massive database. A primary obstacle is the "semantic gap"—the discrepancy between high-level human descriptors (e.g., "elegant" or "sporty") and low-level computational features. Interactive search systems attempt to bridge this gap through iterative feedback. However, traditional text-based queries are often ambiguous. Furthermore, using real images from the database for feedback is problematic; because attributes are naturally correlated (e.g., a "pointier" shoe might also be a different color), users may provide "noisy" feedback that hinders the search algorithm.

Syntharch addresses these challenges by utilizing attribute-conditioned synthesis for visual-only feedback. The system employs a RankCGAN generative backbone to learn a multidimensional latent attribute space.

Attribute-Conditioned Synthesis

Instead of retrieving real images, Syntharch synthesizes feedback options that vary along exactly one latent dimension. This allows the system to isolate specific visual properties, ensuring that the user's choice provides "clean" information about the target's location in the attribute space.

Examples of attribute-conditioned synthesis where specific traits like 'pointy' or 'sporty' are modified while preserving other features.

Range-Based Interaction and Re-ranking

The search process functions as a multidimensional binary search. In each iteration, the system synthesizes two images representing the 1/4 and 3/4 marks of the current search range for a specific attribute. Upon user selection, the system narrows the search window and employs a probabilistic model to re-rank the entire database, bringing the most relevant images to the top of the results.

Experimental Evaluation

The system was evaluated using the UT-Zap50K dataset, comprising 50,025 shoe images. The model was trained on four relative attributes: open, pointy, sporty, and comfort. A user study involving 10 participants across 300 search sessions compared Syntharch against two baselines: a retrieval-based method using real images ("Retrieved+Range") and an older synthesis method ("Synthesis+Pivot").

Results

Syntharch demonstrated superior performance and greater consistency than the baselines.

Comparison of target rank percentile across search iterations for Syntharch and baseline methods.

As shown in the performance graph, Syntharch achieved an average target rank percentile of 72.36% after 12 iterations, outperforming the "Retrieved+Range" baseline (70.89%) and significantly beating the "Synthesis+Pivot" method (57.13%). Notably, while retrieval-based methods often plateau or dip when the database lacks diverse feedback options, Syntharch's ability to generate precise variations leads to a consistent improvement in accuracy.

Conclusion

Syntharch effectively bridges the semantic gap by replacing ambiguous text with precise, AI-synthesized visual feedback. By isolating visual attributes through synthesis, the system reduces user confusion and improves search efficiency. Future work will focus on improving image realism in sparse regions of the attribute space and exploring unsupervised learning of visual attributes to generalize the method to unlabeled databases.

Summary updated

Abstract

The use of interactive systems has been found to be a promising approach for content-based image retrieval, the task of retrieving a specific image from a database based on its content. These systems allow the user to refine the set of results iteratively until the target is reached. In order to proceed with the search efficiently, conventional methods rely on some shared knowledge between the user and the system, such as semantic visual attributes of the images. Those approaches demand the images to be semantically labeled and introduce a semantic gap between the two parties' understanding. In this paper, we explore an alternative approach to interactive image search where feedback is elicited exclusively in visual forms, therefore eliminating the semantic gap and allowing for a generalized version of the method to operate on unlabeled databases. We present Syntharch, a novel interactive image search approach which uses synthesized images as options for feedback, instead of asking textual questions to gain information on the relative attribute values of the target image. We further demonstrate that by using synthesized images rather than real images retrieved from the database as feedback options, Syntharch causes less confusion to the user. Finally, we establish that our proposed search method performs similarly or better in comparison to the conventional approach.
BibTeX
@inproceedings{yu2020syntharch,
  author = {Zac Yu and Adriana Kovashka},
  title = {{Syntharch: Interactive Image Search with Attribute-Conditioned Synthesis}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  url = {https://openreview.net/forum?id=0q3qWCUMFcN},
}
Figure