AdMix: A Mixed Sample Data Augmentation Method for Neural Machine Translation

Chang Jin, Shigui Qiu, Nini Xiao, Hao JiaIJCAI 2022 · DOI

TL;DR

This paper introduces AdMix, a novel data augmentation technique for Neural Machine Translation (NMT) that doesn't require additional training data. AdMix creates synthetic training pairs by introducing subtle noise to existing sentence pairs and then mixing them with their originals, leading to significant improvements in translation quality.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) models, like other deep learning systems, benefit greatly from large amounts of training data. However, acquiring vast parallel corpora (sentence pairs in two languages) can be challenging and expensive. While data augmentation techniques have been successful in other domains like image classification, their application in Natural Language Processing (NLP) is more complex due to the discrete nature of language. Existing methods for NMT, such as back-translation, often rely on additional monolingual data or can be computationally intensive. Other noising-based methods, while simpler, can sometimes lead to a loss of semantic meaning or limited diversity in the augmented data. This paper aims to address this gap by proposing a simple yet effective data augmentation method for NMT that enhances translation performance without needing extra data or significantly altering the original sentence's meaning.

Methods

AdMix consists of two main stages:

  1. Discrete Noise Introduction: The method first applies subtle noise operations to both the source and target sentences of an existing training pair. These operations include:
  2. Word Replacement (WR): Replacing a word with another from the vocabulary.
  3. Word Dropping (WD): Randomly removing words from the sentence.
  4. Word Swapping (WS): Swapping the positions of words within the sentence. The amount of noise is controlled by a hyperparameter $\gamma$, which determines the proportion of words affected by WR/WS or the probability of dropping a word for WD.

  5. Data Mixing: The noisy augmented sentences are then mixed with their original counterparts. This mixing is done in two steps:

  6. Soft Mixing: The embeddings of the augmented sentences are mixed using element-wise convex combinations, with mixing weights sampled from a Dirichlet distribution.
  7. Residual Connection: The softly mixed augmented embeddings are then combined with the original sentence embeddings using a residual connection. This final combination is also a convex mix, with the coefficient sampled from a Beta distribution. This ensures that the final synthetic sample retains a strong connection to the original sentence, preserving semantic meaning.

The augmented data is then used to train the NMT model with a combined loss function that includes the standard cross-entropy loss on original samples and a Jensen-Shannon (JS) divergence consistency loss. This JS divergence loss encourages the model to produce similar output distributions for both original and augmented sentence pairs, further promoting semantic consistency.

Datasets

The paper evaluates AdMix on three standard machine translation benchmarks:

  • IWSLT14 German-English (De-En): A smaller dataset, preprocessed with BPE (Byte-Pair Encoding) with 10K joint operations.
  • LDC Chinese-English (Zh-En): A larger dataset, also preprocessed with BPE (32K merge operations).
  • WMT14 English-German (En-De): Another large dataset, preprocessed with BPE (32K merge operations).

Key Insights and Ideas

  • Semantic Preservation is Key: The core idea behind AdMix is to introduce diversity without sacrificing the original meaning. The two-stage mixing process, especially the residual connection, is crucial for achieving this.
  • Discrete Noise is Sufficient: Unlike some methods that might require complex adversarial generation, AdMix demonstrates that simple discrete noise operations, when combined effectively, can significantly boost NMT performance.
  • JS Divergence for Consistency: The use of Jensen-Shannon divergence as a consistency loss is highlighted as a way to ensure that the model learns robust representations that are invariant to the introduced augmentations, leading to better generalization.
  • Complementary to Other Methods: AdMix is shown to be effective not only on its own but also when combined with other augmentation techniques like back-translation, suggesting its versatility.

Key Results

AdMix consistently outperforms strong Transformer baselines across all tested datasets, achieving significant improvements:

  • Zh-En: 1.70 BLEU points improvement.
  • En-De: 0.96 BLEU points improvement.
  • De-En: 2.67 BLEU points improvement.

The paper also shows that AdMix is particularly effective on smaller datasets. When compared to other data augmentation methods like Swap, WordDrop, Switchout, and SeqMix, AdMix demonstrates superior performance. For instance, it outperforms SwitchOut by 0.8 BLEU points on Zh-En and SeqMix by 1.61 points on De-En. Ablation studies confirm the importance of each component of AdMix, with word swapping being particularly impactful for diversity. The method also shows improved robustness against noisy inputs. Combining AdMix with back-translation further boosts performance, yielding an additional 0.36 BLEU improvement on De-En.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper explores the effect of hyperparameters $\lambda$ (for JS divergence) and $\gamma$ (noise fractions), finding optimal values might still require careful tuning for different datasets. The paper suggests that $\lambda=10$ and $\gamma=0.1$ are good defaults.
  • Future Applications: The authors express interest in exploring AdMix's application beyond machine translation, specifically for natural language understanding (NLU) tasks, such as those in the GLUE benchmark. They also suggest investigating more sophisticated noise operations or mixing strategies.

Summary updated

BibTeX
@inproceedings{jin2022admix,
  author = {Chang Jin and Shigui Qiu and Nini Xiao and Hao Jia},
  title = {{AdMix: A Mixed Sample Data Augmentation Method for Neural Machine Translation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/579},
}
Figure