Abstract
In this paper, we introduce an attribute-based interactive image search which can leverage human-in-the-loop feedback to iteratively refine image search results. We study active image search where human feedback is solicited exclusively in visual form, without using relative attribute annotations used by prior work which are not typically found in many datasets. In order to optimize the image selection strategy, a deep reinforcement model is trained to learn what images are informative rather than rely on hand-crafted measures typically leveraged in prior work. Additionally, we extend the recently introduced Conditional Similarity Network to incorporate global similarity in training visual embeddings, which results in more natural transitions as the user explores the learned similarity embeddings. Our experiments demonstrate the effectiveness of our approach, producing compelling results on both active image search and image attribute representation tasks.
Introduction
When searching for a specific item—like a particular pair of "sporty" sandals—users often have a clear mental picture but struggle to describe it in words. This is the "semantic gap" between low-level image features and high-level conceptual intent.
Existing interactive systems try to bridge this gap by asking users for feedback. However, these systems typically rely on relative attribute annotations (e.g., "Shoe A is more formal than Shoe B"). These labels are incredibly expensive to collect and are rarely found in standard datasets. The authors set out to build a system that uses only the categorical metadata already present in most e-commerce databases (like "heel height" or "material") while still providing a smooth, intuitive search experience.
Methods
The proposed system uses a two-pronged approach to bridge the semantic gap without requiring expensive manual labeling.
1. Globally-Consistent Attribute Embeddings
The model builds upon the Conditional Similarity Network (CSN), which learns to "mask" general image features to focus on specific attributes (e.g., just looking at the "color" or "sole type"). The authors introduce two key improvements:
- Global Compatibility: The loss function is modified to account for the total number of shared attributes between images. This encourages "smooth transitions" in the embedding space, ensuring that moving from one item to another feels like a logical progression of features.
- Normalization Constraints: L2 normalization and attribute masks that sum to one are applied to stabilize training and improve feature selection.
2. Reinforcement Learning for Image Selection
Instead of using hand-crafted rules to decide which image to show the user next, the system employs a Deep Q-Network (DQN). The DQN is trained to select the most "informative" image—the one that, based on the user's feedback, will most significantly reduce the search space and bring the target image to the top of the results.
Experiments and Results
The researchers evaluated their approach on two primary datasets: UT Zap-50K (50,000 shoe images) and OSR (Outdoor Scene Recognition).
Key findings include:
- Metadata Sufficiency: Categorical labels (binary or multi-class) can effectively substitute for expensive relative attribute labels. Simple "is this more like your target?" feedback is sufficient for high-quality search.
- Improved Accuracy: On UT Zap-50K, the enhanced CSN model improved triplet satisfaction by about 3% over the baseline.
- Search Efficiency: The DQN-based strategy reduced the number of steps needed to find a target image. On UT Zap-50K, it averaged 21.79 steps, compared to 26.40 steps for a standard nearest-neighbor baseline—a 17% improvement. On the OSR dataset, search steps were reduced from 6.21 to 4.54.
Conclusion
This work demonstrates that interactive image search can be significantly improved by combining globally-consistent embeddings with reinforcement learning for candidate selection. By leveraging existing categorical metadata, the system avoids the need for costly relative attribute annotations while providing a more efficient and intuitive user experience. Future work may explore hierarchical clustering and batch feedback modes to further accelerate the search process.
Summary updated