Abstract
Common intuition in Natural Language Processing suggests that short sentences should be inherently easier for AI to translate than long, complex ones. However, this research reveals that modern Neural Machine Translation (NMT) models struggle significantly with short texts (STs), often producing over-translations or mistranslations. This paper identifies two primary causes for this phenomenon: data imbalance in training sets and a lack of contextual information. Through targeted interventions such as data upsampling and the integration of translation memory, the authors demonstrate that these performance gaps can be substantially mitigated.
Introduction
In the development of Neural Machine Translation, research has historically prioritized the "long-range dependency" problem—the challenge models face in maintaining coherence across lengthy sequences. Consequently, short texts (defined as 10 tokens or fewer) were largely considered a solved problem.
Despite their brevity, short texts are ubiquitous in digital communication, appearing in search queries, chat dialogues, and entity names. The authors observed that state-of-the-art NMT engines, including the Transformer architecture, frequently fail on these snippets. This study investigates the architectural and statistical reasons why STs remain a hurdle for NMT and proposes practical solutions to improve their reliability.
Methodology
The researchers employed a multi-faceted diagnostic approach to analyze "short text syndrome":
- Fine-Grained Error Analysis: Translation errors were categorized into four specific types: Over-translation (adding redundant words), Under-translation (omitting content), Mistranslation (incorrect meaning), and Misorder (syntactic errors).
- Inference Expected Calibration Error (Inf-ECE): This metric was used to quantify "miscalibration," or the discrepancy between a model’s predicted confidence and its actual translation accuracy.
- Model Uncertainty via MCDS: Monte Carlo Dropout Sampling (MCDS) was applied to the Transformer’s attention layers (Encoder, Decoder, and Cross-Attention) to measure the model's "hesitation" during the translation of short sequences.
- Proposed Interventions: Two primary techniques were tested:
- Data Upsampling: Increasing the frequency of short sentences during training to correct the length-distribution imbalance.
- Translation Memory (TM): Augmenting the model with similar sentence pairs from an external database to provide necessary semantic context.
Experimental Setup
The study utilized two major benchmarks to evaluate performance: the WMT’14 English-to-German (En⇒De) dataset and the WMT’17 Chinese-to-English (Zh⇒En) dataset. To isolate the performance on short texts, the authors extracted specific ST test sets from adjacent years' data.
Analysis and Key Insights
The "Easy" Myth Dispelled
One of the most striking findings is that while older Statistical Machine Translation (SMT) systems performed best on short sentences, NMT models follow a bell curve, performing best on medium-length sentences. For NMT, short texts are statistical outliers that the architecture is not naturally optimized to handle.
The Imbalance Trap and Contextual Poverty
The researchers identified two distinct failure modes:
- Calibration Issues: Because training data is dominated by medium-to-long sentences, the model's internal mechanism for determining when to stop generating tokens is miscalibrated. This leads to "over-translation," where the model continues to generate words beyond the source's meaning.
- Uncertainty Issues: Short sentences lack the "semantic anchors" found in longer contexts. The Cross-Attention (CAN) mechanism, which bridges the source and target languages, showed the highest uncertainty for STs. Without sufficient context, the model is forced to "guess," leading to mistranslations.
Results
The proposed interventions yielded significant improvements across both language pairs:
- Upsampling Results: By balancing the training distribution, the researchers reduced Inf-ECE and significantly decreased over-translation errors. This resulted in a BLEU score increase of approximately 1.6 points on the En⇒De task.
- Translation Memory Results: Providing additional context via TM reduced model uncertainty in the cross-attention layers. This led to gains of nearly 1.0 BLEU point on En⇒De and 0.7 points on Zh⇒En.
- Human Validation: In blind evaluations, human judges rated the upsampled model's output at 3.11/4.0, a marked improvement over the baseline's 2.58/4.0.
Conclusion
While short texts are often overlooked in NMT research, they present unique challenges related to data distribution and semantic sparsity. This study demonstrates that simple yet effective strategies like upsampling and translation memory can bridge the performance gap. Future work may involve more sophisticated data augmentation techniques and deeper investigations into why specific attention heads struggle with brevity, potentially leading to more robust NMT architectures.
Summary updated