Backretrieval: An Image-Pivoted Evaluation Metric for Cross-Lingual Text Representations Without Parallel Corpora

Mikhail Fain, Niall Twomey, Danushka Bollegala

Abstract

Cross-lingual text representations have gained popularity lately and act as the backbone of many tasks such as unsupervised machine translation and cross-lingual information retrieval, to name a few. However, evaluation of such representations is difficult in the domains beyond standard benchmarks due to the necessity of obtaining domain-specific parallel language data across different pairs of languages. In this paper, we propose an automatic metric for evaluating the quality of cross-lingual textual representations using images as a proxy in a paired image-text evaluation dataset. Experimentally, Backretrieval is shown to highly correlate with ground truth metrics on annotated datasets, and our analysis shows statistically significant improvements over baselines. Our experiments conclude with a case study on a recipe dataset without parallel cross-lingual data. We illustrate how to judge cross-lingual embedding quality with Backretrieval, and validate the outcome with a small human study.

Introduction

Universal search and unsupervised machine translation rely on cross-lingual text representations—mathematical vectors where semantically similar terms in different languages (e.g., "apple" and "manzana") are positioned closely. The "gold standard" for evaluating these models is a cross-lingual retrieval task: given a sentence in Language A, can the model find its exact translation in Language B? However, this requires parallel corpora (exact translations), which are rare for specific domains like medical, legal, or culinary fields. Researchers often resort to machine-translating evaluation data, but this creates a "circular logic" bias where the evaluation favors models that behave like the translator used. Backretrieval circumvents this by using images as a universal semantic bridge.

Methodology: The Backretrieval Process

Backretrieval leverages images as a proxy for ground-truth translations. While languages vary, the visual world is a constant; a "chicken soup" recipe in any language should be linked to a visually similar image. The process follows four steps:

  1. Project: Take a source text (e.g., an English recipe) and its associated image. Use the model being tested to project the text into a cross-lingual embedding space.
  2. Retrieve Text: Find the most similar text in the target language (e.g., a Russian recipe) within that embedding space.
  3. Back-retrieve Image: Take the image associated with that target recipe and measure its visual similarity to the original source image using a pre-trained computer vision model.
  4. Rank: Rank the original source image among a pool based on its similarity to the retrieved image. The final metric (e.g., Recall@10) indicates how often the correct visual concept was maintained through the language hop.
The Backretrieval process using images as a semantic bridge.

Crucially, this method doesn't require the source and target texts to be translations of each other; it only requires that both languages have some text-image pairings available.

Datasets and Validation

The authors validated the metric on three public datasets containing images with independent descriptions in multiple languages: Multi30K (English/German), MS-COCO Stair (English/Japanese), and the IKEA-Dataset (English/French/German). They then applied it to a real-world "Cookpad" dataset of 50,000 recipes across five languages with no parallel translations, only images of the finished dishes.

Key Results and Insights

The core finding is the exceptionally high correlation between Backretrieval (BkR) and traditional ground-truth metrics. On the Multi30K and Stair datasets, Backretrieval scores were almost perfectly correlated (.97 to .99) with actual cross-lingual retrieval performance, proving that visual consistency is a reliable proxy for linguistic alignment.

Correlation between Backretrieval (BkR) and ground-truth metrics across various datasets.

Using Backretrieval, the authors identified that standard models like LaBSE struggled with specific language pairs in the recipe domain. They developed TripletXL, a model that uses a specialized loss function to ensure recipes are closer to their cross-lingual counterparts. In a blinded trial, TripletXL provided "very relevant" recipe suggestions 2.3x more often than the baseline, confirming the metric's real-world utility.

Conclusion

Backretrieval offers a robust, image-pivoted alternative for evaluating cross-lingual models in data-scarce domains. While dependent on the quality of the image similarity model, it provides a scalable way to judge embedding quality without expensive human translations. Future work aims to calibrate Backretrieval scores to directly estimate standard Recall@K scores, making the results more interpretable for researchers.

Summary updated

Abstract

Cross-lingual text representations have gained popularity lately and act as the backbone of many tasks such as unsupervised machine translation and cross-lingual information retrieval, to name a few. However, evaluation of such representations is difficult in the domains beyond standard benchmarks due to the necessity of obtaining domain-specific parallel language data across different pairs of languages. In this paper, we propose an automatic metric for evaluating the quality of cross-lingual textual representations using images as a proxy in a paired image-text evaluation dataset. Experimentally, Backretrieval is shown to highly correlate with ground truth metrics on annotated datasets, and our analysis shows statistically significant improvements over baselines. Our experiments conclude with a case study on a recipe dataset without parallel cross-lingual data. We illustrate how to judge cross-lingual embedding quality with Backretrieval, and validate the outcome with a small human study.
BibTeX
@inproceedings{fain2021backretrieval,
  author = {Mikhail Fain and Niall Twomey and Danushka Bollegala},
  title = {{Backretrieval: An Image-Pivoted Evaluation Metric for Cross-Lingual Text Representations Without Parallel Corpora}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2021},
  doi = {10.1145/3404835.3463027},
}
Figure