Translation Corpus Source and Size in Bilingual Retrieval

Paul McNamee, James Mayfield, Charles NicholasNAACL 2009

TL;DR

This paper investigates how the source and size of parallel translation corpora impact cross-language information retrieval (CLIR). They found that corpus quality (alignment, genre) and tokenization method (character n-grams vs. words) are often more critical than sheer size, with character n-grams showing significant advantages, especially with limited data.

What Problem Does This Paper Address?

Cross-language information retrieval (CLIR) allows users to search for documents in a language different from their query language. A common approach to CLIR involves translating query terms using parallel corpora (texts available in multiple languages). However, there's a prevailing belief that simply having larger parallel corpora is always better for CLIR performance, and that corpora from domains closely related to the document collection are also superior. This paper aims to empirically test these assumptions by systematically comparing different parallel corpora of varying sizes and sources, and by examining the impact of tokenization methods on retrieval effectiveness.

Methods

The authors employed a statistical language model approach to CLIR. For each query, they identified source language documents containing query terms. These terms were then translated into the target language using a corpus-based translation method. The translation score for a candidate target language term was based on its relative document frequency within a local subset of documents and the overall corpus, weighted by its inverse document frequency. The translated query was then used to retrieve documents from the target language collection.

Two tokenization methods were compared:

  • Word-based: Standard word tokenization.
  • Character 5-grams: Using sequences of 5 characters as tokens. This method was explored for its potential to handle morphological variations and out-of-vocabulary words.

The experiments involved subsampling the largest corpus (Europarl) to systematically study the effect of corpus size on retrieval performance.

Datasets

The experiments used document collections from the Cross-Language Evaluation Forum (CLEF) in eight languages. English topic sets from 2000-2007 were used as queries.

Four parallel corpora were used for translation:

  • Bible: A religious text corpus, relatively small and with near-perfect verse-level alignments.
  • JRC-Acquis v3: A collection of EU laws, significantly larger than the Bible corpus, with sentence-level alignments.
  • Europarl v3: Transcribed parliamentary proceedings, also large, with high-quality sentence-level alignments.
  • Official Journal of the EU (Ojeu): A corpus constructed by the authors from EU official journals, large in size but with potential issues in text conversion and alignment quality.

Key Insights and Ideas

  • Quality over Quantity (Sometimes): While larger corpora generally perform better, the quality of alignments and the genre of the parallel text can be more crucial than sheer size. The Europarl corpus, despite not being the largest, consistently outperformed others.
  • Domain Matters, But Not Always: A closer domain match between the parallel corpus and the document collection is expected to improve performance. However, the Bible corpus, despite its specific genre, performed the worst, suggesting that its limitations in vocabulary (e.g., lacking terms like "nuclear" or "energy") significantly hindered its effectiveness for general CLIR topics.
  • Character N-grams Shine: Tokenizing text into character 5-grams significantly improved retrieval performance compared to word-based tokenization. This advantage was particularly pronounced when using smaller parallel corpora, suggesting n-grams are more robust to data sparsity.
  • Diminishing Returns with Size: The performance improvement from increasing parallel corpus size tends to plateau after a certain point, consistent with Heap's Law. Even small subsets of a good quality corpus can yield substantial gains.

Key Results

  • Europarl Dominance: The Europarl corpus consistently yielded the best results across all languages and tokenization methods, achieving up to 75% of monolingual effectiveness.
  • Bible's Weakness: The Bible corpus performed the worst, achieving only 39% of monolingual effectiveness on average.
  • 5-gram Advantage: Using character 5-grams for tokenization and translation resulted in significant improvements, with average gains of 30% to 47% compared to word-based methods, depending on the corpus used.
  • Subsampling Insights: Even a 2% subsample of the Europarl corpus (comparable in size to the Bible corpus) outperformed the Bible corpus, highlighting the importance of domain match and alignment quality. A 5% subsample of Europarl with 5-grams outperformed plain words with any amount of bitext.

Limitations and Future Work

  • Ojeu Data Issues: The authors acknowledge potential problems with converting the Ojeu data from PDF and issues with its alignments, which may have contributed to its suboptimal performance.
  • Limited Language Families: The study focused on Indo-European languages. Future work could explore other language families.
  • English as Source: English was the sole source language. Investigating other source languages would be beneficial.
  • Future Directions: The authors suggest exploring bidirectional translation models, applying query expansion techniques, and further investigating the impact of different language families and source languages.

Summary updated

BibTeX
@inproceedings{mcnamee2009translation,
  author = {Paul McNamee and James Mayfield and Charles Nicholas},
  title = {{Translation Corpus Source and Size in Bilingual Retrieval}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure