Abstract
User feedback helps an image search system refine its relevance predictions, tailoring the search towards the user's preferences. Existing methods simply take feedback at face value: clicking on an image means the user wants things like it, commenting that an image lacks a specific attribute means the user wants things that have it. However, we expect there is actually more information behind the user's literal feedback. In particular, a user's (possibly subconscious) search strategy leads him to comment on certain images rather than others, based on how any of the visible candidate images compare to the desired content. For example, he may be more likely to give negative feedback on an irrelevant image that is relatively close to his target, as opposed to bothering with one that is altogether different. We introduce novel features to capitalize on such implied feedback cues, and learn a ranking function that uses them to improve the system's relevance estimates. We validate the approach with real users searching for shoes, faces, or scenes using two different modes of feedback: binary relevance feedback and relative attributes-based feedback. The results show that retrieval improves significantly when the system accounts for the learned behaviors. We show that the nuances learned are domain-invariant, and useful for both generic user-independent search as well as personalized user-specific search.
Introduction
Traditional image search systems interpret user feedback literally. If a user clicks "not like this," the system simply filters for different visual features. However, this paper posits that the choice of which image a user decides to critique contains "implied feedback" about their internal search strategy. By learning these behavioral nuances, search engines can predict the target image much more accurately without requiring additional user effort.
When search results are imperfect, users provide feedback—such as binary relevance ("like this") or relative attributes ("I want something pointier than this shoe"). Humans are strategic: if shown a kitten, an elephant, and an angry dog while looking for a puppy, a user is more likely to critique the kitten (which is "close" to the target) than the elephant (which is obviously irrelevant). This choice reveals the user's mental model.
Methods and Feedback Modes
The authors developed a framework to learn a ranking function incorporating these implicit cues. They formulated the problem as a structural SVM-style ranking task, learning a weight vector $w$ to score the true target image higher than distractors based on observed feedback.
The system focuses on two common feedback modes:
- Binary Relevance: Users indicate "Like this" or "Not like this." The system measures how the selected image relates to other images the user chose not to comment on, identifying the "sweet spot" of informative images.
- Relative Attributes: Users specify "More/less [attribute] than this." A key insight is that the "non-comment" is informative: if a user says "less chubby than Person B" when Person A (who is even less chubby) is also visible, the system infers the target is likely not less chubby than A.
Experimental Setup
The approach was validated across three distinct domains using data gathered via Amazon Mechanical Turk:
- Outdoor Scene Recognition (OSR): 2,688 images (e.g., forests, coasts).
- Public Figures Face Database (PubFig): 900 images of 60 celebrities, focusing on attributes like "chubby."
- Shoes Dataset: 1,000 images, using attributes like "pointy" or "bright."
Key Results and Insights
The "implied feedback" model significantly outperformed standard baselines. On the Shoes dataset, the system improved the target image's rank by approximately 10 percentile points.
Crucially, user behavior nuances were found to be domain-invariant. A model trained on shoe searches successfully improved face searches, suggesting universal human strategies for communicating visual concepts. Furthermore, while general models are effective, the system can be personalized to learn an individual's specific "dialect" of feedback.
Limitations and Future Work
Current experiments primarily focused on single-turn feedback. Future research should explore how these strategies evolve over multi-turn sessions. Additionally, the reliance on pre-trained attribute predictors means noisy models could garble implied signals. Finally, while the study used a game-like interface, future work must evaluate these models in unconstrained, real-world search scenarios where user motivation may differ.
Summary updated