Generic Instance Search and Re-identification from One Example via Attributes and Categories

Ran Tao, Arnold W. M. Smeulders, Shih-Fu ChangarXiv 2016 · arXiv

Introduction: The Challenge of 3D Viewpoints

Traditional instance search algorithms are often "one-sided." They perform exceptionally well on near-planar objects like buildings or logos. For instance, if provided with a photo of a specific landmark, these systems can easily find other images of that same landmark taken from similar angles. However, generic 3D objects—such as shoes or cars—present a significantly harder challenge due to extreme viewpoint variability.

A shoe instance representing the complex 3D objects targeted by the proposed framework.

When a user provides a single query image of a shoe from the front, standard methods relying on local interest point matching (like SIFT) often fail. This is because the back or the sole of the shoe shares almost no visual features with the front. Humans overcome this by understanding the object's category and its specific attributes (e.g., "red leather high-heel"). This research introduces a framework that gives computers a similar ability to generalize from a single example across 3D rotations and occlusions.

Methodology: Bridging Pixels and Semantics

The proposed framework employs a two-pronged strategy to bridge the gap between low-level pixel matching and high-level semantics:

  1. Automatically Learned Attributes: Instead of manual labeling, the system learns "non-semantic" attributes from category-specific training sets. These attributes are optimized to be discriminative (distinguishing between instances), shareable (generalizing across similar instances), and non-redundant.
  2. Category-Level Integration: To ensure the search remains within the correct class (e.g., not confusing a "shiny shoe" with a "shiny car"), the system incorporates high-level features from a Deep Convolutional Neural Network (CNN) and a category classifier. This narrows the search space before applying instance-level attribute matching.

Notably, the authors treat person re-identification as a special case of generic instance search, applying the same logic to identify individuals across disparate camera views.

Evaluation and Results

The framework was evaluated across diverse datasets, ranging from traditional landmarks to challenging "in-the-wild" scenarios.

The Oxford5k dataset features landmarks that are relatively planar compared to 3D consumer goods.

Key Performance Metrics

  • Shoe Retrieval: On the CleanShoes dataset, which features extreme viewpoint changes, the attribute-based method achieved a Mean Average Precision (mAP) of 56.57%. This significantly outperformed state-of-the-art building retrieval methods like ExpVLAD, which managed only 16.14%.
  • Person Re-ID: On the VIPeR benchmark, the generic method reached a Rank-1 matching rate of 43.6%, proving competitive with specialized re-identification algorithms.
  • Cluttered Environments: In the StreetShoes test, combining Deep CNN features with a category classifier and attributes yielded the best result (30.45% mAP), highlighting the necessity of category-level filtering in complex scenes.

Conclusion

The research demonstrates that while traditional geometric verification works for buildings, it fails for 3D objects. By leveraging learned attributes—even those without human-readable names—the system captures visual patterns robust to viewpoint changes. This unified framework suggests that tasks like shoe retrieval and person re-identification, often treated as separate silos, can be addressed through a common "generic" approach to instance search. Future work aims to explore "zero-shot" attribute transfer between different categories to further reduce the need for category-specific training.

Summary updated

Abstract

This paper aims for generic instance search from one example where the instance can be an arbitrary object like shoes, not just near-planar and one-sided instances like buildings and logos. First, we evaluate state-of-the-art instance search methods on this problem. We observe that what works for buildings loses its generality on shoes. Second, we propose to use automatically learned category-specific attributes to address the large appearance variations present in generic instance search. Searching among instances from the same category as the query, the category-specific attributes outperform existing approaches by a large margin on shoes and cars and perform on par with the state-of-the-art on buildings. Third, we treat person re-identification as a special case of generic instance search. On the popular VIPeR dataset, we reach state-of-the-art performance with the same method. Fourth, we extend our method to search objects without restriction to the specifically known category. We show that the combination of category-level information and the category-specific attributes is superior to the alternative method combining category-level information with low-level features such as Fisher vector.
BibTeX
@misc{tao2016generic,
  author = {Ran Tao and Arnold W. M. Smeulders and Shih-Fu Chang},
  title = {{Generic Instance Search and Re-identification from One Example via Attributes and Categories}},
  year = {2016},
  eprint = {1605.07104},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1605.07104},
}
Figure