Abstract
This paper bridges the gap between Cross-Lingual Information Retrieval (CLIR) and Machine Translation (MT) by evaluating the "end-to-end" user experience, termed Translingual IR (TLIR). The authors demonstrate that MT errors significantly degrade retrieval performance (16-39% drop in precision) and introduce hybrid retrieval models and automatic post-editing techniques that successfully mitigate these "lost in translation" effects.
Introduction
In a perfect world, a user could search for documents in a foreign language and receive perfectly translated, relevant results. In reality, CLIR and MT are often studied in silos. CLIR researchers usually evaluate systems by looking at the original foreign-language documents to ignore MT noise, while MT researchers focus on fluency and metrics like BLEU without considering how those translations function in a search task.
This paper addresses the "under-studied gap" of how MT quality impacts the end user's ability to find and recognize information. The authors identify two specific failure modes:
- Lost in Retrieval: MT errors in the indexed documents prevent the search engine from finding them in the first place.
- Lost in Translation: The system finds a relevant document, but the translation is so poor (e.g., missing names or key verbs) that the user thinks it’s irrelevant.
Methodology
The researchers compared two baseline "pipeline" approaches and two novel hybrid models:
- Document Translation (DT): Translating the entire foreign corpus into the user's language before indexing.
- Query Translation (QT): Translating the user's query into the foreign language, searching the original corpus, and then translating only the results back.
- SMLIR (Simultaneous Multilingual IR): A hybrid model that indexes documents as bilingual pairs, allowing the system to match terms in both the query and document languages simultaneously.
- QT-rerank: A two-stage approach that retrieves results using Query Translation and then re-ranks them using the machine-translated versions of those results.
To improve translation quality, they tested two Automatic Post-Editors (APEs): a Rule-Based APE that uses part-of-speech tags to "patch" missing named entities, and a Feedback APE that feeds suggestions back into the MT system for better fluency.
The study utilized a specialized TLIR evaluation corpus based on the NIST MT08 Arabic-English test set. This dataset is unique because it includes human relevance judgments on the machine-translated output, allowing the authors to quantify exactly when a relevant document "became" irrelevant due to a bad translation.
Analysis and Results
A key insight of the study is that BLEU is an imperfect metric for Information Retrieval. Two MT systems (MT A and MT B) had nearly identical BLEU scores, yet MT B performed significantly better in the retrieval task because it was better at handling Named Entities, which are the backbone of search queries.
Quantitative results highlight the significant "MT Penalty":
- Performance Drop: MT errors caused a 16-39% decrease in Mean Average Precision (MAP) compared to a "ground truth" system using human translations.
- Vanishing Relevance: Between 5-19% of sentences that were actually relevant were judged as irrelevant by users because the MT output was too garbled to understand.
- Hybrid Success: The SMLIR and QT-rerank models outperformed the simple pipelines by leveraging both languages, effectively reducing "lost in retrieval" errors.
- Post-Editing Gains: The adequacy-oriented APEs improved end-to-end relevance by 4-28%, proving that even "black-box" MT systems can be improved for specific tasks without retraining the entire model.
Conclusion
The authors acknowledge that their evaluation corpus is small (roughly 1,300 sentences), a necessary trade-off for high-quality human relevance judgments. Future work includes integrating MT confidence estimation directly into search ranking and applying these techniques to more informal web genres where MT performance is traditionally lower than in structured newswire text.
Summary updated