Towards Affordable Semantic Searching: Zero-Shot Retrieval via Dominant Attributes

Yang Long, Li Liu, Yuming Shen, Ling ShaoAAAI 2018 · OpenReview

Abstract

Instance-level retrieval has become an essential paradigm to index and retrieve images from large-scale databases. Conventional instance search requires at least an example of the query image to retrieve images that contain the same object instance. Existing semantic retrieval can only search semantically-related images, such as those sharing the same category or a set of tags, not the exact instances. Meanwhile, the unrealistic assumption is that all categories or tags are known beforehand. Training models for these semantic concepts highly rely on instance-level attributes or human captions which are expensive to acquire. Given the above challenges, this paper studies the Zero-shot Retrieval problem that aims for instance-level image search using only a few dominant attributes. The contributions are: 1) we utilise automatic word embedding to infer class-level attributes to circumvent expensive human labelling; 2) the inferred class-attributes can be extended into discriminative instance attributes through our proposed Latent Instance Attributes Discovery (LIAD) algorithm; 3) our method is not restricted to complete attribute signatures, query of dominant attributes can also be dealt with. On two benchmarks, CUB and SUN, extensive experiments demonstrate that our method can achieve promising performance for the problem. Moreover, our approach can also benefit conventional ZSL tasks.

Introduction

Traditional image retrieval systems typically require an example image to find similar instances. However, real-world scenarios often lack such visual queries—consider a witness describing a suspect or a researcher identifying a bird species based on a textual description. This "Semantic Searching" is hindered by three major challenges: the high cost of manual instance-level attribute labeling, the classification bias of existing Zero-Shot Learning (ZSL) models that fail to distinguish between instances of the same class, and the "query gap" where users provide only a few dominant traits rather than exhaustive attribute lists.

This paper introduces Zero-Shot Retrieval (ZSR), a framework designed to find specific image instances using only a few descriptive "dominant attributes" (e.g., "yellow wings" or "glass building"). By leveraging word embeddings and a novel latent attribute discovery algorithm, the proposed method eliminates the need for expensive manual labeling while outperforming traditional ZSL methods in instance-level search.

Methodology

The authors propose a two-pronged approach to make semantic searching both affordable and precise.

1. Augmented Attributes (AA) via Word Embeddings

To bypass the labeling bottleneck, the researchers utilize Word2Vec to automatically infer relationships between class names and attributes. By treating class names (e.g., "Cardinal") and attributes (e.g., "red beak") as vectors in a shared semantic space, the system can "guess" the attribute signature of a class based solely on its name. This process, termed Augmented Attribute Inference, significantly reduces the reliance on human-annotated tags.

2. Latent Instance Attributes Discovery (LIAD)

To address class-level bias, the LIAD algorithm discovers hidden, discriminative features that vary between instances of the same category.

  • Orthogonal Projection: The algorithm employs an orthogonal constrained projection to identify independent "latent" bases. This mathematical independence prevents redundant features and allows the model to capture unique visual nuances of individual images.
  • Sparse Querying: Designed for real-world utility, the system works with "dominant attributes." When a user specifies only a few traits, the model projects them into the latent space to reconstruct a full, instance-sensitive signature for retrieval.

Experimental Evaluation

Datasets

The model was evaluated on two standard computer vision benchmarks adapted for retrieval:

  • CUB (Caltech-UCSD Birds): 11,788 images of 200 bird species, representing a fine-grained dataset where instance differences are subtle.
  • SUN (Scene UNderstanding): 14,340 images of various scenes (e.g., offices, forests) with 102 attributes.

Key Results and Insights

The proposed method was compared against state-of-the-art ZSL models such as LatEm and ESZSL. In the "Image to Attributes" (I2A) task on the SUN dataset, the method achieved 28.7% @Rank1 accuracy, significantly outperforming LatEm (25.3%) and nearly tripling the performance of standard baseline methods like CCA.

Qualitative retrieval results demonstrating the model's ability to distinguish between instances using dominant attributes.

Qualitative results demonstrate the model's success in distinguishing between different instances of complex categories, such as "Electric indoor lighting," by correctly identifying secondary dominant attributes. Furthermore, the model showed high tolerance for "distractor" images from known classes in Generalized ZSR tests, proving its robustness beyond simple classification.

Conclusion

This research shifts the focus of Zero-Shot Learning from broad categorization to specific instance retrieval. By emphasizing the power of sparsity—showing that 10 to 20 well-chosen attributes are often sufficient—and the importance of orthogonality in latent spaces, the authors provide a scalable path toward affordable semantic searching. Future work will aim to improve Rank-1 accuracy and mitigate the impact of misleading or overly common dominant attributes.

Summary updated

Abstract

Instance-level retrieval has become an essential paradigm to index and retrieves images from large-scale databases. Conventional instance search requires at least an example of the query image to retrieve images that contain the same object instance. Existing semantic retrieval can only search semantically-related images, such as those sharing the same category or a set of tags, not the exact instances. Meanwhile, the unrealistic assumption is that all categories or tags are known beforehand. Training models for these semantic concepts highly rely on instance-level attributes or human captions which are expensive to acquire. Given the above challenges, this paper studies the Zero-shot Retrieval problem that aims for instance-level image search using only a few dominant attributes. The contributions are: 1) we utilise automatic word embedding to infer class-level attributes to circumvent expensive human labelling; 2) the inferred class-attributes can be extended into discriminative instance attributes through our proposed Latent Instance Attributes Discovery (LIAD) algorithm; 3) our method is not restricted to complete attribute signatures, query of dominant attributes can also be dealt with. On two benchmarks, CUB and SUN, extensive experiments demonstrate that our method can achieve promising performance for the problem. Moreover, our approach can also benefit conventional ZSL tasks.
BibTeX
@inproceedings{long2018towards,
  author = {Yang Long and Li Liu and Yuming Shen and Ling Shao},
  title = {{Towards Affordable Semantic Searching: Zero-Shot Retrieval via Dominant Attributes}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=HyFrJWO-S},
}
Figure