Abstract
Non-Autoregressive Translation (NAT) models offer significant speedups by predicting tokens in parallel, but they frequently suffer from multi-modality issues leading to information redundancy. While recent advancements have mitigated obvious continuous repetitions, this paper identifies more subtle forms of redundancy—discontinuous and synonym-based—that persist in state-of-the-art models. We propose a new taxonomy and automatic metrics to quantify these errors, revealing that modern NAT models often trade local repetitions for global redundancies.
Introduction
In Machine Translation, the trade-off between speed and quality is a central challenge. Non-Autoregressive Translation (NAT) models achieve high inference speeds by generating all words in a sentence simultaneously. However, this independence assumption often leads to the "multi-modality" problem: the model may capture multiple valid translation paths at once, resulting in redundant outputs.
Historically, this was most visible as continuous token repetition (e.g., "I ate ate pizza"). While architectural improvements like the Directed Acyclic Transformer (DAT) have nearly eliminated these obvious errors, the underlying multi-modality issue has not been solved; it has merely evolved into more complex patterns that standard metrics fail to capture.
A Taxonomy of Redundancy
To better understand these errors, we categorize information redundancy into four distinct types based on continuity and semantic relationship:
- Continuous Repetition: Identical tokens appearing back-to-back.
- Discontinuous Repetition: Identical tokens separated by other words.
- Continuous Synonym: Semantically similar tokens (synonyms) appearing back-to-back.
- Discontinuous Synonym: Synonyms separated by other words.
To automate the detection of these errors, we developed metrics using mBART word embeddings to identify semantic similarity. We introduce the Continuous Redundancy Ratio (CRR) and the Discontinuous Redundancy Ratio (DRR). The DRR specifically filters for stopwords and words naturally repeated in the source text to ensure accuracy.
Experimental Results
We evaluated several prominent NAT architectures—CMLM, GLAT, and DAT—on the WMT20 datasets (English ↔ Chinese and English ↔ German).
The "Whack-a-Mole" Effect
Our analysis reveals a significant trend: as models become better at suppressing local repetitions, they often produce more global redundancies. For instance, the Directed Acyclic Transformer (DAT) achieves a 0% continuous repetition rate on raw data but exhibits the highest rate of discontinuous redundancy (over 7%).
As shown in the table above, while Knowledge Distillation (KD) reduces redundancy across all models, it does not eliminate the structural differences between architectures. DAT consistently maintains a higher "Other Redundancy" profile compared to earlier models like CMLM, even when its BLEU score is superior.
Key Insights
- Multi-modality Drivers: Synonym redundancy is primarily driven by lexical multi-modality (uncertainty between word choices), whereas discontinuous repetition is linked to reordering multi-modality (uncertainty regarding word placement).
- Metric Limitations: Standard BLEU scores can be misleading in NAT research. A model might achieve a high BLEU score by generating multiple high-probability but redundant tokens. Reporting CRR and DRR alongside BLEU is essential for a holistic evaluation of translation quality.
Conclusion and Future Work
This study highlights that information redundancy remains a critical hurdle for NAT. While "obvious" bugs have been fixed, "subtle" ones like discontinuous synonyms have taken their place. Future research should focus on training objectives that specifically penalize these non-local redundancies. Additionally, improving synonym detection through more robust linguistic resources will further refine our ability to benchmark NAT progress accurately.
Summary updated