Modèle de graphe et modèle de langue pour la reconnaissance de scènes visuelles

Eric Gaussier, Trong-Ton Pham, Loïc MaisonnasseDocument Numérique 2010 · DOI

TL;DR

This paper introduces a novel graph-based framework for image representation and retrieval that integrates visual concepts and their spatial relationships. By extending language modeling techniques to graph matching, the authors achieve improved performance in scene recognition and robot localization tasks compared to existing methods.

What Problem Does This Paper Address?

Content-Based Image Retrieval (CBIR) systems aim to search and organize images based on their visual content, but face a significant challenge in bridging the "semantic gap" between low-level visual features and high-level semantic understanding. Existing methods often rely on flat representations like bag-of-words, which struggle to capture the crucial spatial relationships between image elements. This paper addresses the need for a more comprehensive image representation that can effectively integrate diverse visual information and spatial context, leading to more robust and accurate image retrieval and recognition.

Methods

The proposed approach consists of three main stages:

  1. Image Processing: This stage involves extracting image regions (e.g., through segmentation or keypoint detection) and computing feature vectors (e.g., color, edge histograms, SIFT descriptors) for these regions.
  2. Graph Modeling: This is the core of the proposed method.
  3. Visual Concept Learning: Visually similar features are grouped into clusters using unsupervised learning (k-means clustering) to form "visual concepts."
    • Visual Graph Construction: Spatial relationships between these visual concepts are extracted, and an image is represented as a visual graph. This graph captures both the visual concepts and their spatial arrangements, providing a richer representation than traditional methods.
  4. Graph Retrieval: Inspired by language modeling in text retrieval, this stage extends the framework to match query graphs with document graphs. Images are then ranked based on the probability of generating the query graph from the document graph. This approach leverages the well-established theory of language modeling for a more reliable matching process.

Datasets

The proposed models were evaluated on two datasets:

  • STOIC-101: A collection of 3,849 tourist images of 101 landmarks in Singapore, captured under various conditions.
  • RobotVision'09 (IDOL2 database): A dataset of image sequences captured by a mobile robot in an indoor laboratory environment, featuring variations in illumination, object placement, and time.

Key Insights and Ideas

  • Graph as a Unified Representation: The paper argues that graphs are a natural and versatile way to represent image content by integrating diverse visual features and spatial relationships, effectively creating an intermediate layer to bridge the semantic gap.
  • Language Modeling for Graph Matching: Extending language modeling, a successful technique in text retrieval, to graph matching provides a principled and computationally efficient way to compare complex graph structures.
  • Spatial Relations Enhance Performance: The inclusion of spatial relationships between visual concepts significantly improves the accuracy of image recognition and localization tasks, demonstrating their importance beyond simple feature representation.
  • Flexibility of the Framework: The graph-based framework is shown to be adaptable to different image types and applications, as evidenced by its successful application to both scene recognition and robot localization.

Key Results

  • Scene Recognition (STOIC-101): The proposed visual graph models (VGM) outperformed standard language models (LM) and Support Vector Machine (SVM) methods by over 10% in accuracy. The integration of spatial relations further improved performance, especially when using multiple images for training and querying.
  • Robot Localization (RobotVision'09): The visual graph models demonstrated robustness and adaptability to challenging conditions like varying illumination, object occlusion, and temporal changes. The inclusion of spatial relations (VGMS→P) significantly boosted recognition scores compared to models without relations.
  • Comparison to SVM: In both applications, the proposed graph-based models consistently outperformed SVM methods, highlighting the effectiveness of the graph representation and language modeling-based matching.

Limitations and Future Work

  • Subjectivity of Relations: The authors acknowledge that the definition of spatial relations is currently manual and can vary depending on the image collection or application. Future work aims to explore more automated ways to define and extract these relations.
  • Choice of Visual Concepts: The number of visual concepts (clusters) was chosen empirically. Further study is needed to investigate the impact of this parameter on model quality and to explore more advanced clustering techniques (e.g., fuzzy c-means, EM clustering).
  • Integration of Textual Information: Combining textual information with the visual graph representation for multimedia retrieval and annotation is a promising direction.
  • Scalability and Real-time Performance: While the graph matching is more efficient than classical methods, further work on scalability for very large datasets and real-time performance for applications like video retrieval is suggested.
  • User Interface: Developing a sketchable user interface for graph retrieval is proposed as a way to enhance user interaction and allow for more intuitive querying.

Summary updated

BibTeX
@article{gaussier2010modele,
  author = {Eric Gaussier and Trong-Ton Pham and Loïc Maisonnasse},
  title = {{Modèle de graphe et modèle de langue pour la reconnaissance de scènes visuelles}},
  journal = {Document Numérique},
  year = {2010},
  doi = {10.3166/dn.13.1.211-227},
}
Figure