Bilingual Cluster Based Models for Statistical Machine Translation

Hirofumi Yamamoto, Eiichiro SumitaEMNLP 2007

TL;DR

This paper introduces a novel approach to statistical machine translation (SMT) by automatically clustering bilingual corpora into domain-specific sub-corpora. This allows for the creation of domain-specific language and translation models, which are then dynamically selected based on the input sentence, leading to significant improvements in translation quality.

What Problem Does This Paper Address?

Statistical machine translation (SMT) systems typically rely on large, general-domain training corpora. While effective, these systems often struggle when translating text from specialized domains (e.g., medical, legal, or travel). This is primarily due to two issues: data sparseness (not enough data for the specific domain) and the difficulty of identifying the domain of an incoming sentence. When the domain is unknown or changes dynamically, the system needs to predict both the domain and the translation simultaneously. This paper tackles the challenge of domain prediction and its integration into SMT for improved performance, especially in scenarios where domain knowledge is not explicitly provided.

Methods

The proposed method consists of two main phases: an off-line process for model creation and an on-line process for translation.

Off-line Process: 1. Bilingual Clustering: The core of the off-line process is an unsupervised clustering technique applied to a bilingual training corpus. This clustering is performed using an entropy reduction-based method, where each cluster is treated as a distinct domain. The goal is to group sentences that share similar linguistic characteristics. 2. Domain-Specific Model Creation: Once clusters are formed, domain-specific language models (for both source and target languages) and translation models are built using the data within each cluster. A general-domain model is also created for fallback. 3. Monolingual Corpus Clustering: Additional monolingual corpora are also clustered and assigned to the bilingual clusters to enhance language model training.

On-line Process: 1. Domain Prediction: For each incoming source sentence, the system predicts its domain. This is done by calculating the likelihood of the sentence belonging to each of the pre-defined clusters (domains). The domain that yields the highest probability for the source sentence is selected. This prediction is based on maximizing $P(D|f)$, which is approximated by maximizing $P(f|D)$ assuming constant prior probabilities for domains. 2. Domain-Specific Decoding: Once the domain is predicted, the system uses the corresponding domain-specific language and translation models for decoding the source sentence into the target language. This is formulated as maximizing $P(e|f, D)$, where $D$ is the predicted domain.

Datasets

  • Training Data: A travel conversation task corpus, specifically the subset of the BTEC corpus used in IWSLT06 (Japanese-English). This corpus was used for both training and development.
  • Evaluation Data: The IWSLT05 Japanese to English evaluation corpus was used for the primary evaluation. For comparisons with previous methods, the IWSLT06 Chinese to English open track corpora were used. For the monolingual corpus experiment, NIST MT06 evaluation data was utilized.

Key Insights and Ideas

  • Unsupervised Domain Discovery: The paper demonstrates that domain-specific models can be effectively created and utilized without prior human knowledge of domain labels, simply by clustering bilingual data.
  • Entropy Reduction for Clustering: Using entropy reduction as the criterion for clustering bilingual sentence pairs is a novel way to identify semantically coherent domains.
  • Dynamic Domain Selection: The ability to predict the domain of each source sentence on-the-fly and switch to domain-specific models is crucial for handling dynamic or unknown domains.
  • Combined LM+TM Benefit: The experiments show that using both domain-specific language models and translation models yields the greatest improvement, highlighting the synergistic effect of domain adaptation at both levels.
  • Robustness to ASR Errors: The method shows improved translation performance even when the input is noisy, such as direct output from an Automatic Speech Recognition (ASR) system.

Key Results

  • Japanese to English Translation: The proposed method achieved a significant improvement of 2.7 BLEU points (from 52.4 to 55.1) on the IWSLT05 Japanese to English evaluation corpus.
  • Comparison with Previous Methods: The proposed method outperformed two contemporary methods, Cluster Language Model (CLM) and Sentence Mixture Model (SMix), showing improvements in BLEU and NIST scores, while CLM and SMix degraded NIST scores.
  • Translation of ASR Output: The method demonstrated robustness, improving translation performance even with noisy ASR output.
  • Monolingual Corpus Integration: Incorporating an additional monolingual corpus further boosted performance across all evaluation measures.

Limitations and Future Work

  • Number of Clusters: The number of clusters (domains) needs to be fixed in advance. The paper suggests that ten clusters were found to be a good balance, as fewer clusters couldn't represent domain-specific characteristics well, and more clusters led to severe data sparseness issues in translation models. Determining the optimal number of clusters is an open question.
  • Approximation in Domain Prediction: The domain prediction relies on an approximation ($P(D|f) \approx P(f|D)$), which assumes constant prior probabilities for domains. While confirmed to be reasonable in experiments, it's an approximation.
  • Future Work: The paper implicitly suggests further exploration into optimal cluster number selection and potentially more sophisticated domain prediction models. The success with monolingual corpus integration also points towards further research in combining various data sources.

Summary updated

BibTeX
@inproceedings{yamamoto2007bilingual,
  author = {Hirofumi Yamamoto and Eiichiro Sumita},
  title = {{Bilingual Cluster Based Models for Statistical Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2007},
}
Figure