BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representation

Zheng Zhang, Liang Ding, Dazhao Cheng, Xuebo Liu, Min Zhang, Dacheng TaoarXiv 2022 · DOI · arXiv

Abstract

BLISS (Robust Sequence-to-Sequence Learning via Self-Supervised Input Representation) is a novel framework designed to enhance the robustness of Sequence-to-Sequence (Seq2Seq) models, such as the Transformer. By integrating self-supervised denoising tasks directly into the encoder, BLISS forces the model to reconstruct original tokens and their positions from perturbed inputs. This multi-task learning approach leads to richer internal representations and state-of-the-art performance across machine translation, text summarization, and grammatical error correction. Notably, BLISS demonstrates significantly higher resilience to noisy data at inference time compared to standard data augmentation techniques.

Introduction

Data Augmentation (DA) is a critical component in modern Natural Language Processing (NLP), used to prevent overfitting and improve model generalization. Common techniques involve perturbing input sentences—such as shuffling word order or replacing tokens—to teach models to handle variability. However, standard Seq2Seq architectures often underutilize this perturbed data; the encoder merely passes a degraded signal to the decoder without explicitly learning the nature of the noise. BLISS addresses this limitation by requiring the encoder to actively "restore" the input, transforming it from a passive conduit into a robust feature extractor.

The BLISS Framework

The BLISS framework introduces two primary innovations to the standard training pipeline: a controlled data augmentation strategy and auxiliary self-supervised objectives.

1. Smooth Augmented Data Generator

Rather than applying random noise, BLISS utilizes a Smoothness Controller. The number of perturbed tokens is sampled from a geometric distribution, ensuring a balanced mix of "easy" samples (near-original) and "hard" samples (highly diverse). This prevents the model from being overwhelmed by excessive noise while still providing enough challenge to foster learning. The primary perturbations used are shuffling (reordering words) and replacement (substituting words with random vocabulary tokens).

2. Self-Supervised Input Representation

The core of BLISS lies in its auxiliary classification heads attached to the encoder. These heads perform two specific tasks:

  • Token Classifier: Predicts the original identity of tokens that were replaced.
  • Position Classifier: Predicts the original absolute position of tokens that were shuffled.
The BLISS framework architecture

By training with a combined loss—the standard task-specific loss plus these reconstruction losses—the encoder develops a deeper understanding of syntax and semantics, creating hidden states that are significantly more informative for the decoder.

Experimental Results

The authors evaluated BLISS across five benchmarks, including WMT translation tasks, CoNLL14 for Grammatical Error Correction (GEC), and CNN/Daily Mail for summarization.

Performance Gains

  • Machine Translation: BLISS outperformed the vanilla Transformer by an average of +0.7 BLEU points, surpassing other competitive baselines like SeqMix and SwitchOut.
  • GEC: The model achieved a +2.0 F0.5 improvement on the CoNLL14 task, demonstrating its effectiveness in error-heavy domains.
  • Summarization: BLISS yielded a +0.5 Rouge-L improvement on the CNN/DM dataset.

Robustness to Inference Noise

A key finding is the model's resilience to "messy" inputs at test time. While vanilla Transformers experience sharp performance degradation when faced with typos or scrambled word orders, BLISS maintains high accuracy.

Robustness to shuffle and replace noise

Linguistic probing further confirmed these results, showing that BLISS-trained encoders possess superior understanding of sentence length, word content, and syntactic depth compared to standard models.

Conclusion

BLISS represents a significant step forward in making Seq2Seq models more robust and capable. By giving the encoder the "homework" of reconstructing its own input, the framework ensures that the model learns fundamental linguistic structures rather than just surface-level mapping. Future work may extend this denoising logic to other modalities, such as speech recognition or multi-modal dialogue systems.

Summary updated

Abstract

Data augmentations (DA) are the cores to achieving robust sequence-to-sequence learning on various natural language processing (NLP) tasks. However, most of the DA approaches force the decoder to make predictions conditioned on the perturbed input representation, underutilizing supervised information provided by perturbed input. In this work, we propose a framework-level robust sequence-to-sequence learning approach, named BLISS, via self-supervised input representation, which has the great potential to complement the data-level augmentation approaches. The key idea is to supervise the sequence-to-sequence framework with both the \textit{supervised} ("input$\rightarrow$output") and \textit{self-supervised} ("perturbed input$\rightarrow$input") information. We conduct comprehensive experiments to validate the effectiveness of BLISS on various tasks, including machine translation, grammatical error correction, and text summarization. The results show that BLISS outperforms significantly the vanilla Transformer and consistently works well across tasks than the other five contrastive baselines. Extensive analyses reveal that BLISS learns robust representations and rich linguistic knowledge, confirming our claim. Source code will be released upon publication.
BibTeX
@misc{zhang2022bliss,
  author = {Zheng Zhang and Liang Ding and Dazhao Cheng and Xuebo Liu and Min Zhang and Dacheng Tao},
  title = {{BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representation}},
  year = {2022},
  eprint = {2204.07837},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.07837},
}
Figure