Combining Attention Model with Hierarchical Graph Representation for Region-Based Image Retrieval

Songhe Feng, De Xu, Bing LiIEICE Trans. Information and Systems 2008 · OpenReview

Abstract

The manifold-ranking algorithm has been successfully adopted in content-based image retrieval (CBIR) in recent years. However, while the global low-level features are widely utilized in current systems, region-based features have received little attention. In this paper, a novel attention-driven transductive framework based on a hierarchical graph representation is proposed for region-based image retrieval (RBIR). This approach can be characterized by two key properties: (1) Since the issue about region significance is the key problem in region-based retrieval, a visual attention model is chosen here to measure the regions' significance. (2) A hierarchical graph representation which combines region-level with image-level similarities is utilized for the manifold-ranking method. A novel propagation energy function is defined which takes both low-level visual features and regional significance into consideration. Experimental results demonstrate that the proposed approach shows the satisfactory retrieval performance compared to the global-based and the block-based manifold-ranking methods.

Introduction

Content-Based Image Retrieval (CBIR) systems have traditionally relied on global low-level features, such as color histograms or texture descriptors. However, these methods often fail to capture the semantic intent of a user—a phenomenon known as the "semantic gap." For instance, a search for a specific object might be confounded by background noise if only global features are considered.

Region-Based Image Retrieval (RBIR) attempts to bridge this gap by segmenting images into meaningful parts. Yet, a significant challenge remains: not all regions are created equal. If a user searches for a "tiger," the tiger itself is far more significant than the blurry grass in the background. This paper addresses the need to measure region significance (saliency) and integrate it into a mathematical framework that understands how images relate to one another in a large database.

Methodology

The proposed framework introduces an attention-driven approach to RBIR using a hierarchical graph representation.

1. Visual Attention and Segmentation

To identify significant regions, the system employs a visual attention model inspired by human biological perception. Saliency is calculated at the pixel level based on color, intensity, and orientation across multiple scales. Images are then segmented into homogenous regions using the Normalized-Cut (NC) algorithm. The saliency of a region is defined as the average saliency of the pixels within it.

Visual attention analysis and region segmentation for two sample images, showing original segments, saliency maps, and extracted salient regions.

2. Hierarchical Graph Construction

A two-tier graph structure is utilized to capture both local and global relationships:

  • Region-Level Graph: Vertices represent image segments. Edge weights are determined by combining visual features (HSV color and wavelet textures) with saliency scores. This ensures that similarity between highly salient regions carries more weight than similarity between background regions.
  • Image-Level Graph: Vertices represent entire images. The weight between two images is calculated as the sum of the weights of all region pairs between them.

3. Manifold Ranking

The system uses a transductive learning algorithm called manifold ranking. It treats the dataset as a network (the image-level graph) and "spreads" a relevance score from the query image to its neighbors. Because the graph is hierarchical and saliency-aware, the search "energy" flows more effectively through meaningful parts of the images.

Experimental Results

The researchers evaluated their method using a subset of the COREL collection, consisting of 5,000 images across 50 semantic categories (e.g., tigers, buses, mountains). The proposed "Hierarchical-MR" method was compared against three benchmarks: Global-MR (using global features), Block-MR (using fixed image blocks), and IRM (a well-known integrated region matching system).

Comparison of Average Precision between the proposed Hierarchical-MR and the baseline Global-MR across different numbers of nearest neighbors (k-NN).

Quantitative analysis demonstrates that Hierarchical-MR consistently achieves higher average precision than baseline methods. The inclusion of saliency as a weighting factor allows the system to prioritize focal objects over background elements, leading to more semantically relevant results. Furthermore, the system is computationally efficient for online retrieval, averaging approximately 0.65 seconds per query.

Conclusion and Future Work

By integrating human-centric visual attention with a hierarchical graph-based ranking, this research provides a robust solution for bridging the semantic gap in image retrieval. The core insight is that saliency should not just be an extra feature, but a weighting factor for similarity.

While the system is currently sensitive to the quality of the initial Normalized-Cut segmentation, it demonstrates the power of saliency-aware similarity measures. Future work will explore more advanced segmentation techniques and the incorporation of Multiple Instance Learning (MIL) to further refine how the system learns user intent through feedback.

Summary updated

Abstract

The manifold-ranking algorithm has been successfully adopted in content-based image retrieval (CBIR) in recent years. However, while the global low-level features are widely utilized in current systems, region-based features have received little attention. In this paper, a novel attention-driven transductive framework based on a hierarchical graph representation is proposed for region-based image retrieval (RBIR). This approach can be characterized by two key properties: (1) Since the issue about region significance is the key problem in region-based retrieval, a visual attention model is chosen here to measure the regions' significance. (2) A hierarchical graph representation which combines region-level with image-level similarities is utilized for the manifold-ranking method. A novel propagation energy function is defined which takes both low-level visual features and regional significance into consideration. Experimental results demonstrate that the proposed approach shows the satisfactory retrieval performance compared to the global-based and the block-based manifold-ranking methods.
BibTeX
@article{feng2008combining,
  author = {Songhe Feng and De Xu and Bing Li},
  title = {{Combining Attention Model with Hierarchical Graph Representation for Region-Based Image Retrieval}},
  journal = {IEICE Trans. Information and Systems},
  year = {2008},
  url = {https://openreview.net/forum?id=KSQpaOLk8f},
}
Figure