Capitalizing Machine Translation

Wei Wang, Kevin Knight, Daniel MarcuNAACL 2006

TL;DR

This paper introduces a novel bilingual capitalization model for machine translation (MT) outputs. By leveraging information from both the source and target languages, their Conditional Random Field (CRF) based approach significantly outperforms traditional monolingual methods, especially for challenging language pairs and smaller datasets. This work is crucial for improving the readability and usability of MT systems.

What Problem Does This Paper Address?

Machine translation systems often output text in all lowercase. While this simplifies the training of translation models by reducing vocabulary size, it sacrifices the natural capitalization that makes text readable. Capitalization, or "truecasing," is important for legibility and can even influence downstream NLP tasks like named entity recognition. Traditional capitalization models are "monolingual," meaning they only consider the target language. However, this is insufficient for MT outputs, which can be ungrammatical or preserve case information from the source language. For instance, a sentence that should be all caps in the target language might be missed if the model only looks at the lowercased target text. This paper aims to bridge this gap by developing a capitalization model that effectively uses bilingual information.

Methods

The core of this paper is a bilingual capitalization model formulated as a Conditional Random Field (CRF). CRFs are a type of discriminative model well-suited for sequence labeling tasks. Unlike previous approaches that relied solely on monolingual language models or maximum entropy Markov models (MEMMs), this CRF model explicitly incorporates information from:

  • Source Sentence (F): The original sentence before translation.
  • Target Sentence (e): The lowercased output from the MT system.
  • Alignment (A): The correspondence between source and target words/phrases, typically provided by the MT system.

The model defines a set of "coarse-grained" feature functions that capture various aspects of capitalization, including:

  • Monolingual Language Model: Standard n-gram probabilities for the target sentence.
  • Capitalized Translation Model: Features that leverage the probability of a target word being translated from a source word, considering their cases. This helps preserve case information from the source.
  • Capitalization Tag Translation: Features that model the probability of a target capitalization tag given a source capitalization tag.
  • Upper-case Translation: A feature to encourage all-uppercase translations when the source is also all uppercase.
  • Initial Capitalization & Punctuation Features: Heuristics for sentence-initial words and words following punctuation.

The CRF framework allows these diverse features to be combined and trained jointly to find the optimal capitalization for the target sentence. The paper also details a "naïve phrase aligner" (NPA) to generate phrase alignments for training data, which is computationally less expensive than full phrase alignment.

Datasets

The experiments were conducted on three language pairs:

  • English-to-French (E→F): Information Technology (IT) domain.
  • French-to-English (F→E): General news domain.
  • Chinese-to-English (C→E): General news domain.

Each language pair utilized a training corpus, a development corpus, and two test sets: Test-Precision (for evaluating capitalization accuracy on well-formed text) and Test-BLEU (for evaluating the impact on end-to-end MT performance). The paper notes that the Chinese source language has no case information, making it a particularly interesting test case.

Key Insights and Ideas

  • Bilingual Information is Key: The most significant insight is that leveraging source language case information is crucial for accurate capitalization of MT outputs, especially when the target sentence might be ungrammatical or have unusual case patterns.
  • CRFs for Integrated Capitalization: CRFs provide a flexible and powerful framework to integrate various sources of information (monolingual, bilingual, alignment) into a single capitalization model.
  • Coarse-Grained Features are Efficient: The use of "coarse-grained" features allows for efficient training on large datasets, enabling rapid tuning and good performance without the computational burden of fine-grained features.
  • Chinese as a Test Case: The experiments with Chinese highlight that even when the source language lacks case information, the bilingual model can still perform well, demonstrating the robustness of the approach.

Key Results

  • Significant Improvement over Baselines: The CRF-based bilingual capitalizer significantly outperformed strong monolingual baselines (unigram and trigram language models) on both capitalization precision and end-to-end MT performance (BLEU scores).
  • BLEU Score Gains: For European language pairs, the bilingual model achieved more than a 1 BLEU point improvement on MT translations.
  • Error Rate Reduction: The model led to a 34% relative reduction in capitalization error rate for French-to-English and a 42% reduction for English-to-French.
  • Performance on Small Datasets: The bilingual capitalizer showed a particularly strong advantage when training corpus sizes were small, with the difference between bilingual and monolingual models decreasing as corpus size increased.

Limitations and Future Work

  • Mixed-Cased Translation Models: The authors suggest that estimating mixed-cased translation models (where case is preserved during translation training) is an interesting topic for future study.
  • Real Phrase Aligners: While the NPA is efficient, the authors believe that using a real phrase aligner might lead to even higher quality phrase alignments and, consequently, better capitalization.
  • Fine-Grained Features: The paper acknowledges that fine-grained features might offer better accuracy but require significantly more training data and computational resources. Future work could explore more efficient ways to utilize such features.

Summary updated

BibTeX
@inproceedings{wang2006capitalizing,
  author = {Wei Wang and Kevin Knight and Daniel Marcu},
  title = {{Capitalizing Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
}
Figure