Abstract
This paper details the NAIST-NICT machine translation system developed for the WAT2017 English-Japanese Scientific Paper Translation Task. The system utilizes a language-independent tokenizer and an attentional encoder-decoder neural machine translation (NMT) model. Official results demonstrate that this relatively simple architecture achieves higher translation accuracy than more complex systems from previous campaigns, establishing a robust baseline for the task.
Introduction
In the rapidly evolving field of Neural Machine Translation (NMT), there is often a trend toward increasing architectural complexity to gain marginal performance improvements. However, the NAIST-NICT team demonstrated at the 4th Workshop on Asian Translation (WAT2017) that a "simple and strong" approach can outperform sophisticated, language-specific systems. By focusing on clean, unsupervised tokenization and optimized decoding heuristics, their system achieved state-of-the-art results on the ASPEC (Asian Scientific Paper Excerpt Corpus) English-Japanese task without the need for complex morphological analyzers or heavy ensembling.
Methodology
The system is built upon a standard attentional encoder-decoder framework, refined with several key components designed for robustness and efficiency.
Architecture and Attention
The model employs a 2-layer stacked bidirectional LSTM for the encoder and a 2-layer LSTM for the decoder. To improve the flow of information between these components, the authors implemented a "dense bridge" connection, which initializes the decoder's state using the encoder's final hidden states. The attention mechanism follows the Bahdanau-style multi-layer perceptron approach, supplemented by "input feeding," where the previous attention context is fed back into the decoder at the next step to ensure better tracking of the translated content.
SentencePiece Tokenization
A significant departure from traditional Japanese NMT pipelines is the use of SentencePiece. Instead of relying on language-specific morphological analyzers to segment text, SentencePiece treats the input as a raw stream of characters and learns subword units in an unsupervised manner. This makes the system entirely language-independent and effectively eliminates the "out-of-vocabulary" (OOV) problem by ensuring all text can be decomposed into known subwords.
Decoding with Word Penalty
During the inference phase, the system uses beam search. To address the tendency of NMT models to generate shorter-than-optimal sentences, the authors introduced a "word penalty" heuristic. This constant factor is applied during decoding to encourage the model to match the natural length distribution of the target language.
Results and Performance
The effectiveness of these simple refinements is most evident in the performance jump between the raw "One-best" output and the "Adjusted" output (where beam width and word penalty were tuned).
As shown in the evaluation tables, the "Adjusted" single model achieved a BLEU score of 38.25, a significant improvement over the previous year's best of 36.19. Notably, the system took 1st place overall in the AM-FM metric, which measures semantic fidelity, and ranked 1st among all single-model submissions in human evaluation.
Discussion: Single Models vs. Ensembles
While the single adjusted model is remarkably strong, the study also highlights the persistent gap between single models and ensembles. The authors compared their single model against an 8-model ensemble and the previous year's 3-model ensemble.
The distribution of human adequacy scores shows that while the single model is highly competitive, the 8-model ensemble significantly increases the frequency of "5-star" (perfect) translations. However, the fact that a single, simple model can achieve 1st place in specific metrics like AM-FM suggests that many NMT models possess latent capabilities that can be unlocked through better decoding strategies rather than just adding more parameters.
Conclusion
The NAIST-NICT system for WAT2017 serves as a reminder that foundational components—tokenization and decoding—are as critical as the neural architecture itself. By utilizing SentencePiece and tuning decoding hyperparameters, the authors established a high-performance baseline that is easier to maintain and adapt than language-specific alternatives. While the field has since transitioned toward Transformer-based architectures, the core lessons regarding subword units and search heuristics remain fundamental to modern NLP engineering.
Summary updated