Controlling Formality in Low-Resource NMT with Domain Adaptation and Re-Ranking: SLT-CDT-UoS at IWSLT2022

Sebastian Vincent, Loïc Barrault, Carolina ScartonACL 2022

TL;DR

This paper tackles the challenge of controlling formality in machine translation, especially in low-resource scenarios. The authors propose a novel approach combining domain adaptation to extract formality-annotated data and a hypothesis re-ranking method to boost formality control, achieving near-perfect accuracy in many settings.

What Problem Does This Paper Address?

Formality in language is a nuanced aspect that significantly impacts how a message is perceived. In machine translation (MT), generating translations that match the desired formality level (e.g., formal vs. informal) is crucial for effective communication. However, achieving this control is particularly difficult in low-resource language pairs where large amounts of parallel data are scarce. Existing MT systems often default to a dominant formality, failing to cater to specific user needs. This paper addresses the need for robust formality control in MT, especially for languages with limited training data, by developing methods to adapt existing resources and refine translation outputs.

Methods

The core of the proposed solution lies in two main strategies:

  • Data Engineering for Formality Annotation: The authors developed language-independent methods to extract formal and informal sentence pairs from existing corpora. They used English as a "pivot" language to transfer formality annotations to languages where such data was not directly available. This involved:
  • Domain Adaptation: Leveraging the MuST-C corpus (out-of-domain) and the IWSLT formality-annotated corpus (in-domain) to identify similar sentence pairs. They used sentence-level perplexity differences with language models trained on formal and informal target sentences to rank pairs.
  • INFEREASY and INFERFULL: Two approaches for inferring formality. INFEREASY used simple positional ranking, while INFERFULL introduced a more sophisticated method considering the relative difference in ranking positions to better capture formality.
  • Zero-Shot Adaptation: For language pairs with no direct formality-annotated data (e.g., English-to-Russian), they adapted data inferred from other pairs (e.g., English-to-German/Spanish) by using English as a pivot.
  • Hypothesis Re-ranking (FORMALITYRERANK): Even when the best translation hypothesis from a standard MT model is incorrect in terms of formality, the correct formality might be present in the top-k hypotheses. This method re-ranks these hypotheses using a custom objective function that prioritizes formality. This involves:
  • Relative Frequency Model: Calculating term weights based on their occurrence counts in formal vs. informal corpora.
  • Objective Function: A modified translation objective that incorporates the formality of the generated hypothesis.

Additionally, they employed BESTACCAVERAGING for model selection, averaging checkpoints that yielded the highest accuracy on the development set, rather than simply averaging the last few.

Datasets

The paper utilizes several datasets:

  • MuST-C (v1.2): A large multilingual speech translation corpus used for pre-training.
  • IWSLT formality-annotated corpus: The primary source of direct formality annotations, used for fine-tuning and evaluation. This corpus contains sentence pairs with formal and informal target translations.
  • ParaCrawl, WMT Corpora (News Translation task), Common Crawl, WikiMatrix, WikiTitles, Europarl, UN, Tilde Rapid, Yandex: Various large-scale parallel corpora used for data augmentation in the unconstrained track.

The authors' data engineering efforts effectively created larger, formality-annotated datasets from these sources, particularly for the constrained track.

Key Insights and Ideas

  • Formality is not always a dominant trait: For many language pairs, MT models tend to learn one formality (e.g., formal for EN-DE, informal for EN-ES) much better than the other. This highlights the challenge of achieving balanced control.
  • English as a pivot is powerful but imperfect: Using English to transfer formality annotations to zero-shot languages worked reasonably well, but the quality of inferred data for the re-ranking model was crucial and could be a bottleneck.
  • Re-ranking is a game-changer for formality: Even if the initial best hypothesis is wrong, the correct formality is often found within the top-k hypotheses. FORMALITYRERANK effectively leverages this, significantly boosting accuracy.
  • Data augmentation is key for unconstrained settings: Expanding the training data with diverse corpora dramatically improves performance, especially for formality control.
  • Formality expression varies across languages and contexts: The paper notes that the same English sentence can have different formality expressions in German and Spanish, even within the same discourse situation, underscoring the complexity of the task.

Key Results

  • Constrained Track:
    • Achieved near-ideal accuracy (e.g., .961 to 1.000) for dominant formalities with supervised models.
  • For non-dominant formalities, accuracy was still impressive (average .880 for EN-DE/ES).
  • Zero-shot models for EN-{RU,IT} improved by .193 accuracy points over the baseline.
  • Unconstrained Track:
    • Achieved an impressive average accuracy of .992 for dominant formalities.
  • Performance for non-dominant formalities in EN-{DE,ES} improved significantly to an average of .992.
  • The unconstrained zero-shot model achieved the best accuracy for zero-shot pairs (.659).
  • FORMALITYRERANK: Showed significant improvements, pushing accuracy up to .959 in oracle experiments and achieving near-oracle accuracy in practice. For k=100 hypotheses, it achieved an average improvement of .102 accuracy.
  • BESTACCAVERAGING: Helped achieve an average accuracy score of .961 without impacting translation quality.

Limitations and Future Work

  • Zero-shot formality propagation: While English as a pivot was useful, the quality of the inferred data for the re-ranking model was a limitation. Future work aims to find more robust solutions for propagating formality via a source language to extract training data for other language pairs.
  • Complexity of phenomena: The authors suggest that their methods might be more challenging to apply to more complex phenomena, such as grammatical expression of gender, compared to simpler token-based formality cues (like German "Sie").
  • Data quality for re-ranking: The inferred data used for the relative frequency model in the unconstrained zero-shot track was not as high quality as expected, potentially limiting performance.

The paper's code is publicly available, encouraging further research in this area.

Summary updated

BibTeX
@inproceedings{vincent2022controlling,
  author = {Sebastian Vincent and Loïc Barrault and Carolina Scarton},
  title = {{Controlling Formality in Low-Resource NMT with Domain Adaptation and Re-Ranking: SLT-CDT-UoS at IWSLT2022}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure