Abstract
Statistical Machine Translation (SMT) systems often struggle when translating informal text because they are predominantly trained on formal documents like parliamentary proceedings. This research demonstrates that using simple, rule-based transformations to "normalize" informal French (e.g., converting informal tu to formal vous) before translation significantly improves quality. This run-time pre-processing approach outperforms the alternative of generating artificial training data.
The Problem: Register Mismatch
The "data hunger" of Statistical Machine Translation is a well-documented challenge, but the type of data is just as critical as the quantity. Most high-quality, bilingual corpora originate from formal sources like the European Parliament (Europarl) or the United Nations. However, real-world applications—such as the ACCEPT project—often require the translation of informal content from online technical forums.
This creates a "register mismatch." In French, this is particularly pronounced. Formal French utilizes the pronoun vous and specific verb inflections, whereas informal French uses tu. Furthermore, question formation differs significantly between these registers. When an SMT system trained on formal text encounters an informal forum post, it often encounters "out-of-vocabulary" verb forms or produces awkward, literal translations (e.g., translating the question-starter est-ce que literally as "is it that").
Experimental Approaches
The researchers evaluated two strategies to bridge this register gap using source-language transformations:
- Artificial Training Data (Bottom-Up): Formal Europarl data was "informalized" using rules—changing vous to tu and adjusting verbs accordingly—to retrain the SMT model on a more diverse dataset.
- Run-time Pre-processing (Top-Down): Instead of modifying the model, the input was transformed. Informal sentences were "formalized" (e.g., tu becomes vous) before being processed by a standard SMT engine trained on formal data.
To handle complex verb inflections, the researchers utilized the MMORPH lexicon, a database of over 2 million French word forms. Transformations were implemented using a custom Prolog-based transduction notation, combining regular expression matching with lexical lookups.
Evaluation and Findings
Because standard metrics like BLEU often show only marginal gains in these scenarios, the authors employed human evaluation via Amazon Mechanical Turk. Judges compared translations side-by-side to determine preference.
The results highlighted a clear distinction between "local" and "global" transformations. Pronoun transformations (tu to vous) were highly successful because the changes are contained within a small grammatical window. Human judges significantly preferred the translations using these transformations over the baseline, as shown in the aggregated judgements for the pronoun-specific test set.
Conversely, transformations for question formation (est-ce que) were statistically insignificant. Unlike pronouns, question formation is "global" and context-dependent; simple rules often fail to determine correct verb placement, sometimes leading to translations that were less accurate than the literal baseline.
Conclusion and Future Work
The study concludes that transforming input at run-time is more effective than retraining models with synthetic data. It is fundamentally easier to make a single sentence resemble the training data than to teach a model a new dialect via artificial examples.
While the rule-based approach is effective for certain grammatical features, its "ad hoc" nature and struggle with long-distance dependencies suggest a need for more integrated "automatic editing" pipelines. Future research may explore using sophisticated syntactic parsing to handle complex sentence restructuring across both source and target languages.
Summary updated