TL;DR
While most data augmentation for sequence-to-sequence models focuses on the source input, this paper introduces SSTIA (Soft Sequence-level Target-side Inputs Augmentation), a method that augments the target-side history. By feeding "soft pseudo-tokens" (a mix of ground truth and model predictions) into the decoder during training, the authors achieve significant performance gains across translation, summarization, and dialog tasks without adding any parameters or inference latency.
What Problem Does This Paper Address?
In autoregressive (AR) sequence generation—the backbone of modern LLMs and translation tools—the model predicts the next word based on two things: the source input (e.g., a French sentence) and the words it has already generated (the target-side history).
Most data augmentation research focuses on the source side, such as swapping words in the input or using back-translation. However, the target-side history is equally critical. During training, models usually see perfect "ground truth" history (Teacher Forcing), but during inference, they must rely on their own potentially flawed previous predictions. This discrepancy is known as exposure bias.
While some methods like "Scheduled Sampling" try to bridge this gap by occasionally feeding model predictions back into the decoder, they often do so at a crude, token-level scale. This paper aims to provide a more robust, sequence-level augmentation for that target-side history.
The SSTIA Method
The authors propose SSTIA (Soft Sequence-level Target-side Inputs Augmentation). The process works in three main steps during training:
- Generate Soft Pseudo-Tokens: Instead of just using the hard "one-hot" ground truth word, the model feeds the ground truth into the decoder to get a probability distribution over the whole vocabulary. This distribution is multiplied by the word embeddings to create a "soft" representation of what the model thinks might come next.
- Mixup: Inspired by the "mixup" technique in computer vision, they blend these soft pseudo-tokens with the original ground truth embeddings using a ratio ($\gamma$). This creates a synthetic, "noisy" version of the target history.
- Dual-Stream Training with Consistency Loss: The model is trained to predict the next token using both the original ground truth history and this new synthetic history. Crucially, they add a Consistency Loss ($\mathcal{L}_c$) that forces the model's predictions from both streams to be similar, which stabilizes training and improves generalization.
Importantly, this is a training-only modification. At inference time, the model behaves exactly like a standard Transformer, meaning there is zero extra computational cost for the user.
Datasets and Evaluation
The researchers evaluated their method across three distinct tasks using five widely recognized benchmarks:
- Dialog Generation: DailyDialog (single-turn) and Persona-Chat (multi-turn).
- Machine Translation: IWSLT’14 (English $\leftrightarrow$ German, small scale) and WMT’14 (English $\to$ German, large scale).
- Abstractive Summarization: CNN/DM (news summaries), using the pre-trained BART-Large model as a foundation.
Key Insights and Ideas
- Target-Side Augmentation is a "Hidden Gem": The paper demonstrates that even without touching the source input, improving how the model perceives its own history leads to massive gains.
- Soft is Better than Hard: Unlike previous methods that randomly swap words (hard sampling), using soft probability distributions provides a smoother training signal that helps the model explore the "neighborhood" of the correct sequence.
- The Temperature-Vocabulary Connection: The authors found a linear correlation between the optimal softmax temperature ($T$) and the size of the vocabulary. Larger vocabularies require lower temperatures to keep the probability distributions from becoming too "flat" or noisy.
- Complementary Nature: SSTIA isn't a replacement for source-side augmentation. When combined with Back Translation, the performance increased even further, suggesting that source and target augmentations provide independent benefits.
Key Results
The results were consistently positive across all benchmarks:
- Dialog: On the DailyDialog dataset, SSTIA achieved a +3.57 BLEU-4 improvement over a standard Transformer baseline and significantly outperformed other augmentation methods like SeqMix.
- Translation: On WMT'14 (EN-DE), the method reached 29.1 BLEU, matching or beating much more complex architectures and specialized training routines.
- Summarization: When applied to the already powerful BART model on CNN/DM, it still managed to squeeze out improvements in ROUGE scores, proving the method works even for large-scale pre-trained models.
- Human Evaluation: In blind tests, human evaluators preferred the responses generated by the SSTIA-trained model, noting higher quality in both dialog and summarization tasks.
Limitations and Future Work
- Hyperparameter Tuning: The method introduces a few new hyperparameters (temperature $T$, mixup ratio $\alpha$, and consistency weight $\beta$). While the authors provide some heuristics (like the temperature/vocabulary correlation), these still require tuning for new datasets.
- Multi-round Diminishing Returns: While the method can be applied iteratively (using the augmented model to create even better soft tokens), the authors noted that the gains drop off sharply after the first round.
- Non-Autoregressive Models: This paper focuses strictly on autoregressive models. The authors mention that applying these ideas to non-autoregressive (parallel) generation is a task for future research.
Summary updated