Challenges of Neural Machine Translation for Short Texts

Yu Wan, Baosong Yang, Derek Fai Wong, Lidia S. Chao, Liang Yao, Haibo Zhang, Boxing Chen

Abstract

Common intuition in Natural Language Processing suggests that short sentences should be inherently easier for AI to translate than long, complex ones. However, this research reveals that modern Neural Machine Translation (NMT) models struggle significantly with short texts (STs), often producing over-translations or mistranslations. This paper identifies two primary causes for this phenomenon: data imbalance in training sets and a lack of contextual information. Through targeted interventions such as data upsampling and the integration of translation memory, the authors demonstrate that these performance gaps can be substantially mitigated.

Introduction

In the development of Neural Machine Translation, research has historically prioritized the "long-range dependency" problem—the challenge models face in maintaining coherence across lengthy sequences. Consequently, short texts (defined as 10 tokens or fewer) were largely considered a solved problem.

Despite their brevity, short texts are ubiquitous in digital communication, appearing in search queries, chat dialogues, and entity names. The authors observed that state-of-the-art NMT engines, including the Transformer architecture, frequently fail on these snippets. This study investigates the architectural and statistical reasons why STs remain a hurdle for NMT and proposes practical solutions to improve their reliability.

Methodology

The researchers employed a multi-faceted diagnostic approach to analyze "short text syndrome":

  1. Fine-Grained Error Analysis: Translation errors were categorized into four specific types: Over-translation (adding redundant words), Under-translation (omitting content), Mistranslation (incorrect meaning), and Misorder (syntactic errors).
  2. Inference Expected Calibration Error (Inf-ECE): This metric was used to quantify "miscalibration," or the discrepancy between a model’s predicted confidence and its actual translation accuracy.
  3. Model Uncertainty via MCDS: Monte Carlo Dropout Sampling (MCDS) was applied to the Transformer’s attention layers (Encoder, Decoder, and Cross-Attention) to measure the model's "hesitation" during the translation of short sequences.
  4. Proposed Interventions: Two primary techniques were tested:
  5. Data Upsampling: Increasing the frequency of short sentences during training to correct the length-distribution imbalance.
  6. Translation Memory (TM): Augmenting the model with similar sentence pairs from an external database to provide necessary semantic context.

Experimental Setup

The study utilized two major benchmarks to evaluate performance: the WMT’14 English-to-German (En⇒De) dataset and the WMT’17 Chinese-to-English (Zh⇒En) dataset. To isolate the performance on short texts, the authors extracted specific ST test sets from adjacent years' data.

Table 1: Statistics of the datasets used in the study, highlighting the specific subsets for Short Texts (ST).

Analysis and Key Insights

The "Easy" Myth Dispelled

One of the most striking findings is that while older Statistical Machine Translation (SMT) systems performed best on short sentences, NMT models follow a bell curve, performing best on medium-length sentences. For NMT, short texts are statistical outliers that the architecture is not naturally optimized to handle.

Figure 1: Comparison of Translation Edit Rate (TER) across different sentence lengths for PBSMT and NMT models.

The Imbalance Trap and Contextual Poverty

The researchers identified two distinct failure modes:

  • Calibration Issues: Because training data is dominated by medium-to-long sentences, the model's internal mechanism for determining when to stop generating tokens is miscalibrated. This leads to "over-translation," where the model continues to generate words beyond the source's meaning.
  • Uncertainty Issues: Short sentences lack the "semantic anchors" found in longer contexts. The Cross-Attention (CAN) mechanism, which bridges the source and target languages, showed the highest uncertainty for STs. Without sufficient context, the model is forced to "guess," leading to mistranslations.

Results

The proposed interventions yielded significant improvements across both language pairs:

  • Upsampling Results: By balancing the training distribution, the researchers reduced Inf-ECE and significantly decreased over-translation errors. This resulted in a BLEU score increase of approximately 1.6 points on the En⇒De task.
  • Translation Memory Results: Providing additional context via TM reduced model uncertainty in the cross-attention layers. This led to gains of nearly 1.0 BLEU point on En⇒De and 0.7 points on Zh⇒En.
  • Human Validation: In blind evaluations, human judges rated the upsampled model's output at 3.11/4.0, a marked improvement over the baseline's 2.58/4.0.

Conclusion

While short texts are often overlooked in NMT research, they present unique challenges related to data distribution and semantic sparsity. This study demonstrates that simple yet effective strategies like upsampling and translation memory can bridge the performance gap. Future work may involve more sophisticated data augmentation techniques and deeper investigations into why specific attention heads struggle with brevity, potentially leading to more robust NMT architectures.

Summary updated

Abstract

Short texts (STs) present in a variety of scenarios, including query, dialog, and entity names. Most of the exciting studies in neural machine translation (NMT) are focused on tackling open problems concerning long sentences rather than short ones. The intuition behind is that, with respect to human learning and processing, short sequences are generally regarded as easy examples. In this article, we first dispel this speculation via conducting preliminary experiments, showing that the conventional state-of-the-art NMT approach, namely, Transformer (Vaswani et al. 2017), still suffers from over-translation and mistranslation errors over STs. After empirically investigating the rationale behind this, we summarize two challenges in NMT for STs associated with translation error types above, respectively: (1) the imbalanced length distribution in training set intensifies model inference calibration over STs, leading to more over-translation cases on STs; and (2) the lack of contextual information forces NMT to have higher data uncertainty on short sentences, and thus NMT model is troubled by considerable mistranslation errors. Some existing approaches, like balancing data distribution for training (e.g., data upsampling) and complementing contextual information (e.g., introducing translation memory) can alleviate the translation issues in NMT for STs. We encourage researchers to investigate other challenges in NMT for STs, thus reducing ST translation errors and enhancing translation quality.
BibTeX
@article{wan2022challenges,
  author = {Yu Wan and Baosong Yang and Derek Fai Wong and Lidia S. Chao and Liang Yao and Haibo Zhang and Boxing Chen},
  title = {{Challenges of Neural Machine Translation for Short Texts}},
  journal = {Comput. Linguistics},
  year = {2022},
  doi = {10.1162/coli\_a\_00435},
}
Figure