Weakly supervised construction of a repository of iconic images

Lydia Weiland, Wolfgang Effelsberg, Simone Paolo Ponzetto

TL;DR

This paper introduces a novel semi-automatic method for building a dataset of "iconic images" – visuals that evoke abstract concepts. By leveraging Wikipedia's encyclopedic entries and combining image similarity, clustering, and matching techniques, they create a valuable resource for computational modeling of visual communication.

What Problem Does This Paper Address?

Figurative language and imagery are crucial in human communication, especially in news and media. Iconic images, like a polar bear on an ice floe for global warming, are powerful tools that evoke emotions and associations with abstract topics. However, despite their prevalence, there's a lack of computational models and datasets to understand and utilize them. This paper tackles the challenge of creating such a dataset by developing a weakly supervised approach to automatically harvest iconic images from the web.

Methods

The authors propose a five-phase pipeline:

  1. Seed Selection: They start with human-selected "seed" images from encyclopedic resources like National Geographic Education, which explicitly use iconic images to represent topics. These seeds, along with their captions, are used to find similar images on Wikipedia.
  2. Text-based Image Search: Using the entities extracted from the initial captions, they query Flickr to gather a larger set of images and associated text.
  3. Image Clustering: To handle noise and "semantic mismatches" (where captions don't accurately describe the image), they cluster images based on visual similarity using HSV histograms and K-Means. This helps identify outliers.
  4. Image Filtering: A rule-driven filtering step is applied, first using face detection and then a more robust image matching approach.
  5. Image Matching: This is the core of their novel approach. They combine multiple image matching techniques (template matching, correlation coefficient matching, and SIFT-based features, along with contour matching) to identify visually similar regions between a cluster's "prototype" image and the original seed images. This aims to capture high-level semantic similarity while allowing for diversity in the objects depicted (e.g., different smokestacks for air pollution). A match is confirmed if at least two out of the three matching strategies agree.

Datasets

  • Seed Images: Initially sourced from National Geographic Education and then expanded using Wikipedia.
  • Flickr Dataset: Approximately 4,000 images and text pairs were retrieved from Flickr.
  • Newly Introduced Benchmark: The paper implicitly introduces a benchmark by creating a dataset of iconic images for various topics, which is then used for evaluation. This dataset is made available for research.

Key Insights and Ideas

  • Leveraging Encyclopedic Resources: Using curated encyclopedic entries as a starting point is an effective way to bootstrap the collection of iconic images, as these resources are designed for clear visual representation of concepts.
  • Multi-modal Matching for Diversity: Combining multiple image matching techniques is crucial for capturing iconic relationships. This allows for identifying semantically similar but visually distinct objects, promoting diversity within clusters.
  • Trade-off Between Precision and Recall: The evaluation highlights a significant trade-off. Requiring all matching methods to agree leads to higher precision but drastically reduces recall, indicating that overly strict filtering can miss relevant iconic images.
  • Topic Difficulty Varies: Some topics are inherently easier to find iconic representations for than others. Topics with more visually homogeneous seed images (like "greenhouse effect" with smokestacks) tend to yield better results than those requiring more diverse visual metaphors (like "biodiversity" or "climate refugee").

Key Results

  • The system successfully builds a repository of iconic images with minimal supervision.
  • Combining two image matching techniques achieved 59.5% recall and 68.5% precision.
  • Requiring all three matching techniques to match resulted in 46.9% precision and 14.3% recall, demonstrating the impact of stricter filtering.
  • The authors provide performance metrics for individual topics, showing variations in success rates. For example, "Adaption" and "Greenhouse Effect" showed perfect precision in some scenarios.

Limitations and Future Work

  • Error Analysis: The system suffers from false matches due to issues with contour detection and SIFT's performance on complex object similarity. A mobile phone being matched to "wildfire" is cited as an example of a semantic mismatch.
  • Parameter Tuning: The current image matching methods use default parameters, suggesting that in-domain tuning could improve performance.
  • Future Work:
    • Better integration of heterogeneous information from text and images, potentially using joint semantic representations like LDA-based topic models and visual features.
  • Exploring region-growing algorithms for detecting textured or homogeneous regions and segmenting them without relying solely on contours.
  • Expanding the dataset to cover more topics and improve the robustness of the iconic image detection.

Summary updated

BibTeX
@inproceedings{weiland2014weakly,
  author = {Lydia Weiland and Wolfgang Effelsberg and Simone Paolo Ponzetto},
  title = {{Weakly supervised construction of a repository of iconic images}},
  booktitle = {VL @ COLING},
  year = {2014},
  doi = {10.3115/v1/w14-5410},
}
Figure