Region-based image retrieval using integrated color, shape, and location index

Bhanu Prasad, K. K. Biswas, Shivam GuptaComputer 2004 · DOI

TL;DR

This paper introduces a novel region-based image retrieval system that integrates color, shape, and location features into a single index. By combining these visual cues, the system achieves more effective and efficient image searching compared to methods relying on individual features, demonstrating improved retrieval accuracy and reduced search space.

What Problem Does This Paper Address?

Content-Based Image Retrieval (CBIR) systems aim to find images based on their visual content, moving beyond simple metadata. A significant challenge in CBIR is effectively representing and indexing complex visual information to match user queries accurately. While individual features like color, shape, and location have been explored, their integration into a cohesive indexing scheme that captures richer semantic meaning remains an active area of research. This paper addresses the need for a more robust and effective CBIR system by proposing an integrated index that leverages the complementary strengths of color, shape, and spatial location information for region-based image retrieval.

Methods

The proposed system employs a multi-stage approach:

  • Color Segmentation and Dominant Region Extraction: Images are first pre-processed using anisotropic diffusion to smooth color variations and preserve boundaries. Then, a color clustering approach in RGB space is used to extract up to three dominant color regions. This segmentation is described as "weak," sufficient for retrieval rather than full image recognition.
  • Shape Representation and Normalization: Dominant color regions are used as regions of interest for shape analysis. A region-based approach, inspired by existing work, is adopted. This involves representing shapes using a grid-based method that is invariant to translation, rotation, and scaling. Key steps include computing major and minor axes, rotation normalization to align the major axis with the x-axis, and scale normalization to a fixed size. A binary sequence is generated from an 8x8 grid overlay.
  • Location Indexing: The image space is divided into a 3x3 grid. The location index for a region is determined by the grid cell it maximally covers.
  • Integrated Indexing: The core contribution is the creation of composite indices:
  • Color-Shape Index (CSI): Combines color and shape indices.
  • Color-Shape-Location Index (CSLI): Integrates color, shape, and location indices into a single, low-dimensional index. This composite index is designed to be non-cascaded, meaning all features are considered simultaneously.
  • Hashing Structure: A hash table is used to store the combined indices, grouping similar images into buckets to reduce search space during retrieval.

Datasets

The paper evaluates its approach on several datasets:

  • Custom Database: Consists of 200 basic images of flags and 140 basic images of fruits, flowers, vegetables, and simulated objects. Approximately 900 images were generated by applying rotation and scaling transformations to the basic images.
  • MPEG-7 Core Experiments Testbed: A larger database of 1400 images, comprising 70 classes of various shapes, with 20 images per class. The authors used 46 of these classes for testing robustness to scaling, rotation, and noise.
  • MPEG-7 Shape Database: A publicly available database with 70 object classes, each containing multiple shapes. The authors used 46 classes, with 4 shapes per class, resulting in 184 base images. These were augmented with rotations and scaling to create a dataset of 920 images.

Key Insights and Ideas

  • Synergy of Features: The most significant insight is that combining color, shape, and location information provides a more comprehensive representation of image content than any single feature alone. This integrated approach leads to more accurate and discriminative image retrieval.
  • Non-Cascaded Indexing: The move from cascaded (sequential application of features) to non-cascaded (simultaneous consideration of features) indexing is crucial. This avoids the problem of missing relevant images that might be filtered out in early stages of cascaded retrieval.
  • Robustness through Normalization: Shape representation is made invariant to translation, rotation, and scaling through normalization techniques, which is essential for human-like visual perception and retrieval.
  • Weak Segmentation Sufficiency: For retrieval purposes, a "weak" segmentation that identifies dominant color regions is sufficient, avoiding the computational cost and complexity of perfect image segmentation.

Key Results

  • Improved Retrieval Effectiveness: The integrated color-shape-location index consistently outperformed retrieval based on individual color or shape indices across various databases.
  • Precision and Recall Gains: Tables 2, 3, 4, 5, 6, and 7 show significant improvements in precision and recall rates when using the combined indexing approach compared to individual feature indexing or previous methods (e.g., Lu and Sajjanhar [34]). For instance, on the flower/fruit/vegetable database, cascaded color-shape achieved an average precision of 0.845806, while the color-shape-location index showed even better performance in some scenarios.
  • Better Pruning: Figures 22 and 23 demonstrate that the composite color-shape-location index leads to better pruning of irrelevant images, retrieving results that are almost exact matches to the query.
  • Robustness: Experiments on the MPEG-7 shape database showed the approach's robustness to scaling and rotation.

Limitations and Future Work

  • Segmentation Dependency: The system's performance is inherently limited by the initial segmentation process. While anisotropic diffusion is used to improve robustness, obtaining truly homogenous color regions can still be challenging.
  • Disambiguities in Location: There may be ambiguities in precisely positioning regions within the defined sub-locations in the image space.
  • Extendibility: The authors suggest the scheme is easily extendible to domains with more colors, shapes, and locations to overcome current limitations.
  • Future Work: The authors are exploring storing extracted features in a relational database and using SQL for queries, enabling more complex query combinations (AND, OR, NOT) and interactive querying. They also plan to extend this to handle textual queries, integrating with metadata-based indexing paradigms like MPEG-7 and Dublin Core.

Summary updated

BibTeX
@inproceedings{prasad2004region,
  author = {Bhanu Prasad and K. K. Biswas and Shivam Gupta},
  title = {{Region-based image retrieval using integrated color, shape, and location index}},
  booktitle = {Computer},
  year = {2004},
  doi = {10.1016/j.cviu.2003.10.016},
}
Figure