Abstract
BLISS (Robust Sequence-to-Sequence Learning via Self-Supervised Input Representation) is a novel framework designed to enhance the robustness of Sequence-to-Sequence (Seq2Seq) models, such as the Transformer. By integrating self-supervised denoising tasks directly into the encoder, BLISS forces the model to reconstruct original tokens and their positions from perturbed inputs. This multi-task learning approach leads to richer internal representations and state-of-the-art performance across machine translation, text summarization, and grammatical error correction. Notably, BLISS demonstrates significantly higher resilience to noisy data at inference time compared to standard data augmentation techniques.
Introduction
Data Augmentation (DA) is a critical component in modern Natural Language Processing (NLP), used to prevent overfitting and improve model generalization. Common techniques involve perturbing input sentences—such as shuffling word order or replacing tokens—to teach models to handle variability. However, standard Seq2Seq architectures often underutilize this perturbed data; the encoder merely passes a degraded signal to the decoder without explicitly learning the nature of the noise. BLISS addresses this limitation by requiring the encoder to actively "restore" the input, transforming it from a passive conduit into a robust feature extractor.
The BLISS Framework
The BLISS framework introduces two primary innovations to the standard training pipeline: a controlled data augmentation strategy and auxiliary self-supervised objectives.
1. Smooth Augmented Data Generator
Rather than applying random noise, BLISS utilizes a Smoothness Controller. The number of perturbed tokens is sampled from a geometric distribution, ensuring a balanced mix of "easy" samples (near-original) and "hard" samples (highly diverse). This prevents the model from being overwhelmed by excessive noise while still providing enough challenge to foster learning. The primary perturbations used are shuffling (reordering words) and replacement (substituting words with random vocabulary tokens).
2. Self-Supervised Input Representation
The core of BLISS lies in its auxiliary classification heads attached to the encoder. These heads perform two specific tasks:
- Token Classifier: Predicts the original identity of tokens that were replaced.
- Position Classifier: Predicts the original absolute position of tokens that were shuffled.
By training with a combined loss—the standard task-specific loss plus these reconstruction losses—the encoder develops a deeper understanding of syntax and semantics, creating hidden states that are significantly more informative for the decoder.
Experimental Results
The authors evaluated BLISS across five benchmarks, including WMT translation tasks, CoNLL14 for Grammatical Error Correction (GEC), and CNN/Daily Mail for summarization.
Performance Gains
- Machine Translation: BLISS outperformed the vanilla Transformer by an average of +0.7 BLEU points, surpassing other competitive baselines like SeqMix and SwitchOut.
- GEC: The model achieved a +2.0 F0.5 improvement on the CoNLL14 task, demonstrating its effectiveness in error-heavy domains.
- Summarization: BLISS yielded a +0.5 Rouge-L improvement on the CNN/DM dataset.
Robustness to Inference Noise
A key finding is the model's resilience to "messy" inputs at test time. While vanilla Transformers experience sharp performance degradation when faced with typos or scrambled word orders, BLISS maintains high accuracy.
Linguistic probing further confirmed these results, showing that BLISS-trained encoders possess superior understanding of sentence length, word content, and syntactic depth compared to standard models.
Conclusion
BLISS represents a significant step forward in making Seq2Seq models more robust and capable. By giving the encoder the "homework" of reconstructing its own input, the framework ensures that the model learns fundamental linguistic structures rather than just surface-level mapping. Future work may extend this denoising logic to other modalities, such as speech recognition or multi-modal dialogue systems.
Summary updated