On the Information Redundancy in Non-Autoregressive Translation

Zhihao Wang, Longyue Wang, Jinsong Su, Junfeng Yao, Zhaopeng TuarXiv 2024 · DOI · arXiv

Abstract

Non-Autoregressive Translation (NAT) models offer significant speedups by predicting tokens in parallel, but they frequently suffer from multi-modality issues leading to information redundancy. While recent advancements have mitigated obvious continuous repetitions, this paper identifies more subtle forms of redundancy—discontinuous and synonym-based—that persist in state-of-the-art models. We propose a new taxonomy and automatic metrics to quantify these errors, revealing that modern NAT models often trade local repetitions for global redundancies.

Introduction

In Machine Translation, the trade-off between speed and quality is a central challenge. Non-Autoregressive Translation (NAT) models achieve high inference speeds by generating all words in a sentence simultaneously. However, this independence assumption often leads to the "multi-modality" problem: the model may capture multiple valid translation paths at once, resulting in redundant outputs.

Historically, this was most visible as continuous token repetition (e.g., "I ate ate pizza"). While architectural improvements like the Directed Acyclic Transformer (DAT) have nearly eliminated these obvious errors, the underlying multi-modality issue has not been solved; it has merely evolved into more complex patterns that standard metrics fail to capture.

A Taxonomy of Redundancy

To better understand these errors, we categorize information redundancy into four distinct types based on continuity and semantic relationship:

  1. Continuous Repetition: Identical tokens appearing back-to-back.
  2. Discontinuous Repetition: Identical tokens separated by other words.
  3. Continuous Synonym: Semantically similar tokens (synonyms) appearing back-to-back.
  4. Discontinuous Synonym: Synonyms separated by other words.
Examples of the four types of redundancy identified in NAT outputs.

To automate the detection of these errors, we developed metrics using mBART word embeddings to identify semantic similarity. We introduce the Continuous Redundancy Ratio (CRR) and the Discontinuous Redundancy Ratio (DRR). The DRR specifically filters for stopwords and words naturally repeated in the source text to ensure accuracy.

Experimental Results

We evaluated several prominent NAT architectures—CMLM, GLAT, and DAT—on the WMT20 datasets (English ↔ Chinese and English ↔ German).

The "Whack-a-Mole" Effect

Our analysis reveals a significant trend: as models become better at suppressing local repetitions, they often produce more global redundancies. For instance, the Directed Acyclic Transformer (DAT) achieves a 0% continuous repetition rate on raw data but exhibits the highest rate of discontinuous redundancy (over 7%).

Comparison of BLEU scores and redundancy error rates across different NAT models and data types.

As shown in the table above, while Knowledge Distillation (KD) reduces redundancy across all models, it does not eliminate the structural differences between architectures. DAT consistently maintains a higher "Other Redundancy" profile compared to earlier models like CMLM, even when its BLEU score is superior.

Key Insights

  • Multi-modality Drivers: Synonym redundancy is primarily driven by lexical multi-modality (uncertainty between word choices), whereas discontinuous repetition is linked to reordering multi-modality (uncertainty regarding word placement).
  • Metric Limitations: Standard BLEU scores can be misleading in NAT research. A model might achieve a high BLEU score by generating multiple high-probability but redundant tokens. Reporting CRR and DRR alongside BLEU is essential for a holistic evaluation of translation quality.

Conclusion and Future Work

This study highlights that information redundancy remains a critical hurdle for NAT. While "obvious" bugs have been fixed, "subtle" ones like discontinuous synonyms have taken their place. Future research should focus on training objectives that specifically penalize these non-local redundancies. Additionally, improving synonym detection through more robust linguistic resources will further refine our ability to benchmark NAT progress accurately.

Summary updated

Abstract

Token repetition is a typical form of multi-modal problem in fully non-autoregressive translation (NAT). In this work, we revisit the multi-modal problem in recently proposed NAT models. Our study reveals that these advanced models have introduced other types of information redundancy errors, which cannot be measured by the conventional metric - the continuous repetition ratio. By manually annotating the NAT outputs, we identify two types of information redundancy errors that correspond well to lexical and reordering multi-modality problems. Since human annotation is time-consuming and labor-intensive, we propose automatic metrics to evaluate the two types of redundant errors. Our metrics allow future studies to evaluate new methods and gain a more comprehensive understanding of their effectiveness.
BibTeX
@misc{wang2024information,
  author = {Zhihao Wang and Longyue Wang and Jinsong Su and Junfeng Yao and Zhaopeng Tu},
  title = {{On the Information Redundancy in Non-Autoregressive Translation}},
  year = {2024},
  eprint = {2405.02673},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.02673},
}
Figure