Improving Text Generation with Dynamic Masking and Recovering

Zhidong Liu, Junhui Li, Muhua ZhuIJCAI 2021 · DOI

TL;DR

This paper introduces a novel, universal approach to enhance text generation models by dynamically masking tokens in the decoder's input during training and employing a masked language modeling (MLM) auxiliary task to recover these masked tokens. This method improves robustness and performance across various text generation tasks without requiring task-specific modifications.

What Problem Does This Paper Address?

Text generation models, often built on encoder-decoder architectures, face challenges in robustness and generalization. Existing methods to improve these models are frequently input-specific, limiting their applicability across diverse tasks like machine translation, AMR-to-text generation, and image captioning. A key issue is the discrepancy between training (using ground truth history) and inference (where history is generated), which can lead to performance degradation. Furthermore, training robust models often requires extensive data, and finding ways to improve performance with limited data is a persistent goal.

Methods

The authors propose a two-pronged approach applied to the decoder of an encoder-decoder framework:

  • Dynamic Masking: During training, a percentage of tokens in the decoder's input sequence (the "history context") are randomly masked. Instead of using the ground truth history, the model learns to predict the next token using this corrupted version. This introduces randomness and helps bridge the gap between training and inference. Crucially, the masking pattern is generated anew for each training instance in every epoch, preventing the model from relying on fixed corrupted versions.
  • Masked Language Modeling (MLM) Auxiliary Task: To further enhance the model's ability to handle corrupted inputs, an auxiliary task is introduced. This task trains the model to predict the original, masked tokens from the corrupted history. This is inspired by BERT's MLM but adapted for the seq2seq generation setting, where the masking occurs on the decoder side.

These two components are integrated into a multi-task learning framework, where the decoder layers are shared, but the final prediction layers are task-specific. The overall loss function combines the standard text generation loss with the MLM loss.

Datasets

The proposed approach was evaluated on several standard datasets across different text generation tasks:

  • Machine Translation:
    • WMT14 English-German (EN-DE)
    • IWSLT14 German-English (DE-EN)
  • AMR-to-text Generation:
    • AMR2.0 dataset
  • Image Captioning:
    • MSCOCO 2014 dataset (using the Karpathy splitting)

Key Insights and Ideas

  • Universality of the Approach: The method is designed to be input-agnostic, working with various encoder types and requiring no task-specific techniques beyond the standard encoder-decoder structure. This makes it a broadly applicable enhancement.
  • Bridging the Train-Inference Gap: Dynamic masking directly addresses the common issue where models perform differently during training (with perfect history) and inference (with generated history). By training with imperfect history, the model becomes more resilient.
  • Synergy of Masking and MLM: Combining dynamic masking with the MLM auxiliary task proves more effective than either technique alone. The MLM task helps the model learn to "fill in the blanks," reinforcing the learning from the dynamic masking.
  • Faster Convergence: The addition of the MLM task not only improves performance but also appears to accelerate the convergence of the training process, as evidenced by learning curves.
  • Data Efficiency: The approach shows significant improvements even on large datasets, suggesting it can enhance robustness and performance without necessarily requiring more data.

Key Results

The proposed approach demonstrated significant improvements across all tested tasks:

  • Machine Translation: Achieved improvements of 0.90 and 1.01 BLEU scores over the Transformer baseline on WMT14 EN-DE and IWSLT14 DE-EN, respectively. Dynamic masking alone provided a 0.61 and 0.66 BLEU score improvement.
  • AMR-to-text Generation: Showed a significant improvement of 1.39 BLEU over the state-of-the-art, with dynamic masking alone contributing 0.87 BLEU and the MLM task adding a further 0.52 BLEU.
  • Image Captioning: Reported improvements of 1.5, 0.5, and 0.9 in CIDEr, SPICE, and BLEU-4 scores, respectively, on the MSCOCO test set.

Limitations and Future Work

  • Hyperparameter Sensitivity: The effectiveness of the MLM task is influenced by a hyperparameter ($\alpha$) that balances the text generation and MLM losses. While the paper explores this, finding the optimal $\alpha$ might require tuning. The paper notes that the optimal value is generally less than 0.7 and varies across tasks.
  • Sampled Token Ratio: For dynamic masking, the ratio of tokens to be masked is a hyperparameter. Experiments suggest that 15% is optimal for AMR-to-text generation, indicating this ratio might need tuning for different tasks.
  • Future Work: The paper implicitly suggests that further exploration of different masking strategies and their integration with various generation architectures could be beneficial. The universality of the approach also opens doors for application to other sequence-to-sequence tasks.

Summary updated

BibTeX
@inproceedings{liu2021improving,
  author = {Zhidong Liu and Junhui Li and Muhua Zhu},
  title = {{Improving Text Generation with Dynamic Masking and Recovering}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/534},
}
Figure