Image-Image Search for Comparable Corpora Construction

Yu Hong, Liang Yao, Mengyi Liu, Tongtao Zhang, Wenxuan Zhou, Jianmin Yao, Heng JiHyTra @ COLING 2016 · OpenReview

TL;DR

This paper introduces a novel method for constructing bilingual comparable corpora by leveraging image similarity to find corresponding sentences across languages. Instead of relying solely on linguistic features, they use an image-image search engine to retrieve similar images and their associated captions, effectively bridging language barriers through visual content.

What Problem Does This Paper Address?

Building large-scale comparable corpora, which contain sentence pairs with similar meaning but in different languages, is crucial for training machine translation systems. Traditional methods often rely on complex cross-language information retrieval (CLIR) techniques, bilingual dictionaries, or statistical machine translation (SMT) to find these equivalents. However, these methods can be limited by the availability and quality of linguistic resources or the accuracy of translation models. This paper addresses the challenge of efficiently and reliably constructing such corpora by proposing a method that bypasses direct linguistic analysis and instead uses visual similarity as the primary link between languages.

Methods

The core of the proposed method is an Image-Image Search engine that acts as a bridge for Cross-Media Information Retrieval (CMIR). The process involves:

  1. Data Collection: Crawling images and their captions from news websites in both source (English) and target (Chinese) languages to build separate data banks (SDB and TDB).
  2. Image-Image Search: Using an image from the SDB as a query to search for visually similar images in the TDB. The similarity is measured using the Scale-Invariant Feature Transformation (SIFT) algorithm, which extracts keypoints from images that are robust to scale and rotation changes.
  3. Caption Retrieval: Once similar images are found in the TDB, their corresponding captions are retrieved.
  4. Comparable Corpus Construction: The retrieved captions from the target language are considered potential equivalents for the source caption. The ranking of these potential equivalents is based on the image similarity score.

The SIFT algorithm is detailed as a key component, focusing on its ability to detect local features and create scale-invariant keypoint descriptors. The paper also describes how image similarity is calculated using the Euclidean distance between matching keypoints, with a specific thresholding mechanism to filter out less relevant matches.

Datasets

  • Target Data Bank (TDB): Initially, 42,633 images and captions were crawled from Chinese news websites (China News, News of Sina, Xinhua Net). After filtering for single-sentence captions and a maximum length of 20 Chinese words, a TDB containing 10,371 image-caption pairs was created.
  • Source Data Bank (SDB): A smaller, "mini-sized" data bank was created similarly from English news websites (CNN, BBC, Xinhua Net). This SDB contains 52 pairs of source captions and images.
  • Ground-Truth Equivalents: To evaluate the system, the authors collected at least 5 ground-truth equivalents (similar images and comparable captions) for each of the 52 samples in the SDB, totaling 451 ground-truth equivalents. These were added to the TDB, making it no longer a "black box" for evaluation.

Key Insights and Ideas

  • Visual Similarity as a Bridge: The fundamental insight is that visually similar images often share semantically similar captions, even across different languages. This allows for a language-agnostic approach to finding bilingual sentence pairs.
  • SIFT for Robust Image Matching: The use of SIFT for image feature extraction is highlighted for its effectiveness in detecting local details and its scale-invariant properties, which are crucial for matching images taken from different perspectives or at different scales.
  • Ranking by Image Similarity: The method leverages the image similarity score to rank potential bilingual sentence pairs, assuming that higher image similarity implies higher caption comparability.
  • Automatic Comparability Measurement: The paper proposes an automatic method (C-measure) to assess the comparability of sentence pairs, combining content similarity, entity co-occurrence, and caption length ratio. This is crucial for evaluating the quality of the constructed corpora without relying solely on manual annotation.

Key Results

  • The proposed CMIR method achieved a precision of 0.85 in the top search results on 10,371 target captions.
  • In experiments using the mini-sized SDB as queries, the CMIR system achieved a macro-average precision of over 60% in the top 5 search results.
  • The CMIR method outperformed a state-of-the-art CLIR system, yielding nearly 2% performance gains at top 1 and top 2.
  • The distribution analysis of different comparability levels (Parallel, Comparable, Pseudo-comparable, Incomparable) over image-similarity rankings showed that higher-quality equivalents (Parallel and Comparable) were consistently ranked higher, indicating the method's ability to prioritize good matches.
  • The automatic C-measure was found to correlate highly with manual ratings (Pearson factor of 0.993), validating its reliability for evaluating comparability.

Limitations and Future Work

  • Reliability of Source Captions: A significant challenge is ensuring that the source captions selected for querying actually have equivalents in the target data bank. If not, the CMIR results will be incorrect, potentially degrading the quality of the constructed corpora.
  • Predicting Target Caption Numbers: Determining how many target captions to include in the final corpus is another issue. The current method of using the C-measure as a threshold might negatively impact efficiency.
  • Future Work:
    • Implement a full CMIR-based corpora constructor.
  • Develop methods to identify source captions that are likely to have equivalents in the target data bank, possibly by leveraging "break news" that are widely reported.
  • Investigate a joint model of textual comparability and image similarity for more robust filtering and selection of comparable sentence pairs.
  • Explore active learning techniques to enhance the evaluation process on larger datasets.

Summary updated

BibTeX
@inproceedings{hong2016image,
  author = {Yu Hong and Liang Yao and Mengyi Liu and Tongtao Zhang and Wenxuan Zhou and Jianmin Yao and Heng Ji},
  title = {{Image-Image Search for Comparable Corpora Construction}},
  booktitle = {HyTra @ COLING},
  year = {2016},
  url = {https://openreview.net/forum?id=rgk_gdrWFEb},
}
Figure