NICT's Neural Machine Translation Systems for the WAT21 Restricted Translation Task

Zuchao Li, Masao Utiyama, Eiichiro Sumita, Hai ZhaoWAT @ ACL 2021 · DOI

Abstract

Restricted translation, where specific terminology must be included in the output, presents a significant challenge for standard Neural Machine Translation (NMT) models. This paper describes the system developed by NICT and Shanghai Jiao Tong University for the WAT’21 restricted translation task. By introducing Sampled Constraints as Concentration (SCC), the authors bridge the gap between training and inference. When integrated with sparse Transformer architectures and efficient constrained decoding, the system achieves substantial improvements, reaching gains of nearly 10 BLEU points over standard baselines.

Introduction

While Neural Machine Translation (NMT) has achieved high levels of fluency, it often operates as a "black box," making it difficult to enforce the inclusion of specific technical terms or brand names. This requirement is known as restricted translation. Traditional approaches rely on Constrained Decoding (CD) at inference time to force the inclusion of specific tokens. However, a fundamental mismatch exists: models are typically trained on unconstrained text but are forced to adhere to fixed anchors during testing. This "training-inference gap" often results in suboptimal phrasing and reduced translation quality as the model struggles with unexpected constraints.

Methodology

The proposed system addresses the training-inference gap through a multi-pronged approach:

Sampled Constraints as Concentration (SCC)

SCC is the core innovation of this work. During the training phase, words are randomly sampled from the target reference sentence and provided to the decoder as additional context. To ensure the model learns to utilize these constraints, a penalty factor ($\gamma$) is added to the loss function at these sampled positions. This "concentrates" the model's attention on the required words, preparing its internal representations for the hard constraints it will encounter during inference. Notably, positional encodings are removed for these constraints to maintain order independence, as user-provided keywords may not match the final output order.

PRIME Transformer

To handle long-range dependencies and avoid over-concentration on single tokens, the team utilized the PRIME Transformer (referred to as Sparse Transformer in experimental results). This architecture employs a multi-scale sparse attention mechanism and dynamic convolutions, allowing the model to better capture both local and global context compared to standard Transformers.

Vectorized Dynamic Beam Allocation

To overcome the computational bottlenecks of traditional constrained decoding, the system employs a vectorized version of dynamic beam allocation. This allows for efficient batch processing on GPUs, making the inclusion of hard constraints feasible for practical applications.

Experimental Setup

The models were evaluated using the Asian Scientific Paper Excerpt Corpus (ASPEC) for both English-to-Japanese (En→Ja) and Japanese-to-English (Ja→En) tasks. The training pipeline involved pre-training on ParaCrawl-v5.1 (10.12M sentences) and Wiki Titles v2 (3.64M sentences), followed by fine-tuning on the ASPEC dataset (3.01M sentences).

Results

The experimental results demonstrate significant improvements across all primary metrics: BLEU, RIBES, and AMFM.

English-to-Japanese (En→Ja)

In the En→Ja task, the baseline Transformer-big model achieved a BLEU score of 41.82 (using the kyt tokenizer). The integration of SCC training and Constrained Decoding (CD) increased this score to 49.24. The final ensemble system, which combines vanilla and sparse Transformers, reached a peak BLEU of 51.32.

Results for the English-to-Japanese (En→Ja) translation task across different tokenizers and metrics.

Japanese-to-English (Ja→En)

Similar trends were observed in the Ja→En task. The baseline BLEU score of 28.18 was improved to 35.26 with the SCC + CD approach. The ensemble model further pushed the performance to 37.01 BLEU.

Results for the Japanese-to-English (Ja→En) translation task.

Conclusion

The success of the NICT system at WAT’21 highlights the importance of aligning training objectives with inference-time constraints. By training the model to expect and incorporate constraints through SCC, the researchers effectively mitigated the training-inference gap. The combination of this strategy with architectural improvements like the PRIME Transformer and efficient decoding algorithms sets a new benchmark for restricted translation tasks. Future work will focus on optimizing sampling strategies and further reducing the computational overhead of constrained decoding.

Summary updated

BibTeX
@inproceedings{li2021nict,
  author = {Zuchao Li and Masao Utiyama and Eiichiro Sumita and Hai Zhao},
  title = {{NICT's Neural Machine Translation Systems for the WAT21 Restricted Translation Task}},
  booktitle = {WAT @ ACL},
  year = {2021},
  doi = {10.18653/v1/2021.wat-1.4},
}
Figure