A Statistical Approach to Language Translation

P. Brown, J. Cocke, S. Della Pietra, V. Della Pietra, F. Jelinek, R. Mercer, P. RoossinCOLING 1988

TL;DR

This paper introduces a novel statistical approach to machine translation, moving away from traditional rule-based systems. By leveraging large parallel corpora (like the Canadian Hansard), the authors propose a method to automatically build translation glossaries and generate translations based on probabilistic models of word and phrase correspondences. This work lays the groundwork for modern statistical machine translation systems.

What Problem Does This Paper Address?

The authors tackle the long-standing challenge of automatic language translation. Traditional approaches often relied on complex linguistic rules and grammars, which were difficult to create, maintain, and adapt to different language pairs. The core problem was how to systematically and automatically derive translation knowledge from existing translated texts, rather than relying on human-crafted linguistic expertise. They aim to bridge the gap between source and target languages by learning statistical relationships from data.

Methods

The paper outlines a three-stage statistical approach:

  1. Partitioning the source text: Breaking down the source sentence into "fixed locutions" (words or phrases that have a consistent translation).
  2. Glossary-based selection: Using a statistically derived glossary and contextual information to select corresponding locutions in the target language.
  3. Arranging target words: Ordering the words of the selected target locutions to form a coherent target sentence.

Key to their method is the automatic creation of a statistical glossary. This involves:

  • Initial word-to-word probability estimation: Using parallel corpora to estimate the probability $P(e|f)$ of an English word $e$ translating a French word $f$. They initially use a naive approach where all words in the target sentence are considered potential translates, then refine this by considering mutual information to identify stronger associations.
  • A generative model: They propose a probabilistic model where French words generate primary English words, and these can further generate secondary English words (accounting for length differences). This model uses probabilities $P(e|f)$, $Q(e_j|e_i)$ (secondary generation), and $R(k|e_i)$ (number of secondary words).
  • Iterative re-estimation (EM-like): To estimate the parameters of the generative model, they use an iterative process. They start with initial guesses for probabilities, find the most likely "generation pattern" for each sentence pair, and then re-estimate probabilities based on these patterns. This process is repeated until the estimates stabilize.
  • Locution identification: They extend the concept to multi-word locutions by analyzing mutual information between word pairs derived from the secondary generation probabilities.

Datasets

The primary dataset used is the bi-lingual Hansard corpus of proceedings of the Canadian parliament. This corpus contains approximately 30 million words of parallel English and French text. They also mention using a large English corpus for estimating trigram probabilities for sentence re-arrangement.

Key Insights and Ideas

  • The power of parallel corpora: The availability of large, aligned translated texts is the cornerstone of their statistical approach, enabling the learning of translation probabilities.
  • Moving beyond literal translation: The concept of "fixed locutions" acknowledges that translation isn't always word-for-word and that phrases need to be translated as units.
  • Probabilistic modeling of generation: The generative model provides a principled way to account for variations in sentence length and the generation of secondary words, which is crucial for realistic translation.
  • Iterative refinement is key: The EM-like algorithm for parameter estimation is essential for bootstrapping the learning process from initial guesses to more accurate probabilities.
  • Mutual information as a guide: Using mutual information helps identify strong word and phrase correspondences, even when simple frequency counts might be misleading (e.g., for common function words).
  • Statistical sentence re-arrangement: Even if the correct target words are generated, their order matters. They show that statistical models (like trigrams) can help re-arrange words to form more natural-sounding sentences.

Key Results

While the paper focuses on outlining the methodology and presenting intermediate results, they report:

  • Encouraging glossary creation: They present lists of high mutual information French-English word pairs (Figure 2) and partial glossary lists with probabilities (Figure 3), demonstrating the effectiveness of their statistical methods for identifying likely translations.
  • Promising sentence re-arrangement: In an experiment using trigram probabilities on English sentences, they found that in 63% of sentences with 10 words or less, the most likely statistical arrangement was the original sentence. Furthermore, this arrangement preserved the meaning in 79% of cases. This suggests that statistical ordering can be effective.

Limitations and Future Work

  • No full French/English translation examples: The authors explicitly state that their project was less than a year old and they were not yet able to provide complete French/English translation examples.
  • Crude grammar: They acknowledge that their approach eschews conventional grammars and that the degree of sophistication required is uncertain. They suggest that preliminary experiments indicate only a "very crude grammar" might be needed.
  • Computational complexity: The proposed methods, especially for identifying complex locutions and maximizing generation patterns, are computationally intensive. They mention that some aspects are computationally excessive and rely on truncated modifications.
  • Initial parameter selection: The initial selection of probabilities for the generative model and the method for finding the maximizing pattern are noted as areas requiring careful consideration.
  • Future work: They plan to refine the model to better reflect reality, include more complex locutions, and provide a complete mathematical specification for their approach. They also suggest that further research is needed on appropriate equivalence classification schemes for contextual glossaries.

Summary updated

BibTeX
@inproceedings{brown1988statistical,
  author = {P. Brown and J. Cocke and S. Della Pietra and V. Della Pietra and F. Jelinek and R. Mercer and P. Roossin},
  title = {{A Statistical Approach to Language Translation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1988},
}
Figure