An Empirical Study on Web Mining of Parallel Data

Gumwon Hong, Chi-Ho Li, Ming Zhou, Hae-Chang RimCOLING 2010

TL;DR

This paper tackles the challenging task of automatically mining high-quality parallel corpora directly from the web, moving beyond traditional methods that rely on pre-existing comparable corpora. By developing novel web search query formulations and a robust document pair filtering mechanism, they demonstrate significant improvements in statistical machine translation (SMT) performance, even with smaller, web-mined parallel datasets.

What Problem Does This Paper Address?

Acquiring large, high-quality parallel corpora is crucial for training effective statistical machine translation (SMT) systems. Traditionally, researchers have relied on sources like parliamentary proceedings (e.g., Hansards) or mined comparable corpora from the web. Comparable corpora, however, often contain loosely related or non-parallel sentences, limiting their utility for SMT. Existing web mining approaches for comparable corpora often struggle with the vastness and noise of the open web, and their effectiveness for extracting truly parallel sentence pairs has been limited. This paper addresses the gap of efficiently and effectively mining genuine parallel data from the web, which is a much more challenging task than mining comparable data.

Methods

The authors propose a two-phase approach:

  • Document Pair Retrieval: This phase focuses on identifying candidate document pairs from the web that are likely to contain parallel sentences.
  • Novel Query Formulations: They move beyond simple keyword matching by developing sophisticated queries. This includes:
  • TF-IDF ranked keywords: Translating top TF-IDF keywords from the source language (SL) document.
  • MT-translated keywords: Using a machine translation (MT) system to translate the SL document and then extracting top TF-IDF keywords from the translated text. This leverages the MT system's ability to handle lexical ambiguity.
  • Named Entity (NE) focused queries: Leveraging named entities as strong indicators of parallel content.
  • Re-ranked Queries: A key innovation is using a "Learning to Rank" approach (specifically, Ranking SVM) to re-rank keywords. Features for re-ranking consider both the representativeness of a keyword to the document's topic (e.g., TF-IDF, title presence) and the confidence in its translation quality (e.g., source of translation, whether it's an original English word). This helps prioritize keywords that are more likely to be accurate translations.
  • Query Merging and Iterative Refinement: They merge results from multiple queries to boost recall and iteratively drop keywords if a query fails to return documents.
  • Sentence Pair Extraction: Once candidate document pairs are retrieved, this phase extracts actual parallel sentences.
  • For Comparable Corpora: They use a Maximum Entropy classifier based on alignment patterns from an HMM alignment model.
  • For Parallel Corpora: This is the paper's main contribution. They introduce a document pair filter based on IBM Model 1 alignment probabilities. This filter uses a strict threshold to ensure high precision, discarding noisy document pairs that might otherwise pass simpler filtering. This approach hypothesizes that the alignment probability of a document pair can serve as a strong indicator of its parallelism. Sentence alignment is then performed on the filtered pairs.

Datasets

  • Training Data:
    • NIST SMT training data was used to train dictionaries for keyword translation and the statistical dictionary for the IBM Model 1 alignment.
  • A dataset of 1,900 Chinese/English news document pairs (distinct from the main corpus) was used to train the keyword re-ranker.
  • Crawled Data: Approximately 11,000 Chinese news articles were crawled from Chinese news sites. These articles were selected based on cue phrases (e.g., "according to foreign media") suggesting English counterparts and a minimum length of 500 words to facilitate keyword formulation.
  • Evaluation Data:
    • NIST 2005 and NIST 2008 test sets were used for evaluating the SMT systems.
  • A baseline bilingual dataset (B1 section of NIST SMT training set) was used for comparison.

Key Insights and Ideas

  • Web mining for parallel corpora is significantly harder than for comparable corpora: The noise and scale of the web pose substantial challenges, and traditional methods are insufficient.
  • Sophisticated query formulation is key to web-based document retrieval: Simple keyword translation is not enough; leveraging MT and named entities, and especially re-ranking keywords based on translation confidence and document representativeness, dramatically improves retrieval.
  • Translation quality of keywords is a major bottleneck: The gap between simple queries and "oracle queries" (using target page keywords) highlights this. Re-ranking based on translation confidence is a promising way to mitigate this.
  • IBM Model 1 alignment probabilities are effective for filtering noisy document pairs: A strict threshold on Viterbi alignment probability can reliably identify genuinely parallel document pairs, even from noisy web sources.
  • Parallel corpora, even if smaller, can outperform comparable corpora for SMT: This suggests that quality (true parallelism) is more important than quantity when it comes to SMT training data, especially when using a strict filtering approach.
  • Incremental mining is necessary: The difficulty in retrieving older news documents suggests that bilingual news mining needs to be an ongoing process.

Key Results

  • The proposed re-ranked queries (especially QRANK-NE) significantly outperform simple queries in document pair retrieval recall, approaching the performance of "oracle queries" in some cases.
  • The document pair filter based on IBM Model 1 successfully identifies parallel corpora, which are substantially smaller than comparable corpora but yield better SMT performance.
  • SMT experiments showed that augmenting the baseline training data with mined parallel corpora led to significant improvements in BLEU scores, often outperforming the use of mined comparable corpora. For example, in Table 4, B1+parallel(all query) achieved a 2.43 BLEU point improvement over the baseline for Hierarchical PSMT on NIST 2008, while B1+comparable(all query) achieved only 1.72.
  • The proposed queries for comparable corpus mining also showed improvement over the Munteanu and Marcu (2005) method.

Limitations and Future Work

  • Limitations:
    • The performance of simple queries for comparable corpus mining was found to be unsatisfactory for web mining.
  • Retrieving older news documents proved more difficult, suggesting the need for incremental mining.
  • The current approach is primarily focused on news text, and extending it to other domains presents a significant challenge.
  • Future Work:
    • Applying the proposed approach to other language pairs.
  • Incorporating meta-information from SL documents (e.g., publication date, news agency) to improve retrieval precision.
  • Exploring mining of non-news text from the web, which will require careful identification and selection of cue phrases.
  • Investigating whether larger web-mined corpora can achieve performance comparable to manually prepared corpora, especially for out-of-domain test sets.

Summary updated

BibTeX
@inproceedings{hong2010empirical,
  author = {Gumwon Hong and Chi-Ho Li and Ming Zhou and Hae-Chang Rim},
  title = {{An Empirical Study on Web Mining of Parallel Data}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2010},
}
Figure