Discovering Details and Scene Structure with Hierarchical Iconoid Shift

Tobias Weyand, Bastian Leibe

Introduction

Landmark recognition has traditionally been a "flat" problem. If you take a photo of the Eiffel Tower, a standard recognition system identifies it as the Eiffel Tower. However, for a user standing in front of a complex landmark like Notre Dame de Paris, simply knowing the name of the building is often insufficient. They may be interested in a specific portal, a particular statue, or a unique relief.

Researchers have addressed this by developing Hierarchical Iconoid Shift (HIS), an algorithm that automatically discovers the fine-grained details of landmarks from massive collections of internet photos. By evolving standard clustering into a parameter-free, scale-aware hierarchy, the system organizes visual data from the broad building facade down to its smallest unique architectural features.

The Multi-Scale Challenge

The primary technical hurdle in landmark recognition is that architectural features exist at multiple scales. A single building facade might contain dozens of distinct sub-objects. Previous clustering methods typically required a "bandwidth" parameter—essentially a pre-defined search radius. If set for buildings, the system misses the statues; if set for statues, the building facade fragments into thousands of unrelated pieces.

Figure 1: Comparison between traditional flat recognition (Google Goggles) and the proposed hierarchical discovery which identifies specific architectural details of Notre Dame.

Methodology: HMS and HIS

The authors introduce two primary innovations to solve the scale problem:

  1. Hierarchical Medoid Shift (HMS): This is a new clustering algorithm inspired by scale-space theory. Unlike traditional methods that use a fixed window size, HMS performs a continuous search. It starts with a tiny window and gradually expands it; as the window grows, smaller clusters naturally merge into larger ones. This process is parameter-free and generates a dendrogram (a tree-like structure) that tracks these merges.
  2. Hierarchical Iconoid Shift (HIS): The researchers apply HMS to the "Iconoid Shift" framework. Here, the "distance" between images is determined by their geometric overlap (homography). An Iconoid represents a frontal, centered, and iconic view of an object. HIS organizes these Iconoids into a hierarchy where the leaves represent fine details and the root represents the entire landmark.

To ensure the results are useful for humans, the system includes Dendrogram Simplification. This step collapses redundant nodes where the visual change between a parent and child (such as a minor zoom or pan) is too small to be semantically meaningful.

Large-Scale Evaluation

The system was tested on a massive dataset of 802,129 images crawled from Flickr, covering 34 historic landmarks known for rich detail, including the Louvre, the Taj Mahal, and Aachen Cathedral. To validate the results, the researchers compared the discovered visual clusters against Wikipedia articles. They treated the text and images in Wikipedia as a "ground truth" of what humans consider important details of a landmark.

Key Results and Insights

  • Granular Discovery: HIS significantly outperformed previous state-of-the-art methods. By avoiding a fixed scale, it successfully identified very small-scale objects, such as specific gargoyles on Notre Dame or the "Adamen sorrera" relief on the Sagrada Familia.
  • Semantic Alignment: On average, the algorithm successfully matched approximately 50% of the details mentioned in Wikipedia articles across all 34 landmarks.
  • Efficiency: Despite the complexity of building a hierarchy, the algorithm maintains the same runtime and memory complexity as standard Medoid Shift and is easily parallelizable.
Figure 6: A discovered hierarchy of architectural details, demonstrating how the algorithm organizes images from the full building facade down to specific ornaments and portals.

Conclusion and Future Work

By treating landmark recognition as a hierarchical problem rather than a flat classification task, HIS provides a much richer understanding of architectural scenes. While the system currently struggles with weakly textured or highly non-planar surfaces (like rounded statues) due to its reliance on SIFT features, it opens new doors for mobile augmented reality. Future applications could include automated tourist guides that provide context for the exact detail a user is viewing in real-time.

Summary updated

Abstract

Current landmark recognition engines are typically aimed at recognizing building-scale landmarks, but miss interesting details like portals, statues or windows. This is because they use a flat clustering that summarizes all photos of a building facade in one cluster. We propose Hierarchical Iconoid Shift, a novel landmark clustering algorithm capable of discovering such details. Instead of just a collection of clusters, the output of HIS is a set of dendrograms describing the detail hierarchy of a landmark. HIS is based on the novel Hierarchical Medoid Shift clustering algorithm that performs a continuous mode search over the complete scale space. HMS is completely parameter-free, has the same complexity as Medoid Shift and is easy to parallelize. We evaluate HIS on 800k images of 34 landmarks and show that it can extract an often surprising amount of detail and structure that can be applied, e.g., to provide a mobile user with more detailed information on a landmark or even to extend the landmark's Wikipedia article.
BibTeX
@inproceedings{weyand2013discovering,
  author = {Tobias Weyand and Bastian Leibe},
  title = {{Discovering Details and Scene Structure with Hierarchical Iconoid Shift}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2013},
  doi = {10.1109/iccv.2013.432},
}
Figure