System Description for Transperfect

Wiktor Stribiżew, Fred Bane, José Conceição, Anna ZaretskayaACL 2021

TL;DR

Transperfect's participation in the 2021 Workshop on Asian Translation (JPC2 patent task) explored a "one-size-fits-all" approach to machine translation for patent documents across six language pairs. Despite using a uniform methodology and a single base Transformer model, their results showed significant performance variance across language pairs, highlighting the ongoing challenge of achieving consistent high performance in specialized domains.

What Problem Does This Paper Address?

The paper addresses the challenge of applying cutting-edge machine translation (MT) techniques, particularly Transformer models, to real-world industrial settings like patent translation. While academic research often focuses on pushing performance boundaries with extensive customization, companies face constraints related to deployment costs, scalability, explainability, and the complexity of optimizing hyperparameters for numerous language pairs. This paper investigates whether a standardized, simplified approach can yield competitive results across multiple language pairs in a specialized domain, specifically patent translation.

Methods

The authors employed a standardized data preparation and model training pipeline using a single base Transformer architecture. They utilized the Marian NMT framework and trained unidirectional models for each of the six language pairs. Crucially, they kept hyperparameters such as label smoothing, dropout, learning rate, batch size, and model architecture fixed across all language pairs to test the efficacy of a uniform approach. They also experimented with different subword vocabulary sizes (8,000 and 32,000) and configurations (shared vs. split vocabularies). Data augmentation through back-translation was also applied.

Datasets

The primary dataset used was the JPC2 patent task data, provided by the Japan Patent Office (JPO). This dataset consisted of parallel corpora for three language pairs: English ↔ Japanese, Korean ↔ Japanese, and Chinese ↔ Japanese. For each pair, training sets contained one million sentence pairs, drawn from four domains: chemistry, electricity, mechanical engineering, and physics.

Key Insights and Ideas

  • The "One-Size-Fits-All" Challenge: The core insight is that despite a uniform methodology and model architecture, performance varied significantly across language pairs. This strongly suggests that a single, optimized configuration is unlikely to be universally effective for all language pairs, even within a specialized domain.
  • Hyperparameter Uniformity vs. Performance Variance: The paper highlights a disconnect between the uniformity of their experimental setup and the diversity of their results. This implies that the underlying linguistic characteristics of each language pair, or their interaction with the patent domain, play a crucial role in determining optimal translation settings.
  • Back-Translation is Universally Beneficial: The inclusion of back-translated data consistently improved validation scores across all tested language pairs, underscoring its value as a data augmentation technique in NMT.
  • Vocabulary Size Matters Differently: The size of the subword vocabulary had a notable impact, but its effect varied significantly across language pairs. This suggests that vocabulary optimization is a critical, language-pair-specific tuning parameter.

Key Results

The paper reports BLEU scores for various configurations. For instance, the English → Japanese pair achieved a BLEU score of 27.1 with a shared 8K vocabulary and back-translation. In contrast, the Korean → Japanese pair saw scores around 46.6-46.8 with similar configurations. The Korean → Japanese pair also showed a significant drop when using a shared vocabulary without back-translation (45.6). The authors note that their local BLEU scores do not match those reported by the organizers due to processing differences.

Limitations and Future Work

The authors acknowledge that their "one-size-fits-all" approach precluded achieving optimal performance for individual language pairs. They also state that they are unaware of any theoretical framework to predict how hyperparameters interact to produce such varied results across language pairs, suggesting that iterative experimentation is currently the primary method for optimization.

Future work could explore additional steps that were not included to maintain simplicity, such as:

  • Ensembling models for decoding.
  • Using larger model sizes.
  • Performing word segmentation prior to vocabulary creation.
  • Employing curriculum learning (ordering training data using a language model).
  • Using an additional model for right-to-left re-ranking.

The paper concludes that despite advances in MT, achieving optimal performance still requires careful, language-pair-specific hyperparameter selection.

Summary updated

BibTeX
@inproceedings{stribizew2021system,
  author = {Wiktor Stribiżew and Fred Bane and José Conceição and Anna Zaretskaya},
  title = {{System Description for Transperfect}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure