On Sparsifying Encoder Outputs in Sequence-to-Sequence Models

Biao Zhang, Ivan Titov, Rico Sennrich

TL;DR

This paper introduces LODROP, a novel method to sparsify encoder outputs in sequence-to-sequence models, allowing them to discard up to 70% of source information without significant quality loss. This sparsification not only reduces computational load, leading to decoding speedups of up to 1.65x, but also provides insights into which types of information are most crucial for generation.

What Problem Does This Paper Address?

Sequence-to-sequence (Seq2Seq) models, widely used in tasks like machine translation and text summarization, typically process all encoder outputs to generate target sequences. However, this approach might be inefficient, as not all source information is equally important for generation. Previous work has shown that many source words receive minimal attention during translation (Figure 1). This redundancy suggests that Seq2Seq models might be able to achieve comparable performance while processing a sparser representation of the source input. The challenge is to identify and discard this redundant information effectively without compromising the quality of the generated output and to understand what information is truly essential.

Methods

The core contribution of this paper is the LODROP (LO-regularized Neural Sparsity Layer) layer. This differentiable layer is inserted between the encoder and decoder of a standard Transformer model. LODROP introduces a set of "gates" (one for each encoder output) that are learned during training. These gates are regularized using an L0 penalty, encouraging them to become exactly zero, effectively masking out corresponding encoder outputs. This process is end-to-end differentiable, allowing the model to learn which encoder states to prune.

The paper also proposes a modified decoding algorithm (Algorithm 1) that leverages the learned sparsity to speed up inference by only attending to the non-pruned encoder outputs. Furthermore, the authors explore rule-based sparsity patterns, inspired by their analysis of LODROP's learned behavior, to manually mask out specific types of encoder outputs.

Datasets

The authors evaluate their method on several standard datasets:

  • Machine Translation:
    • WMT14 English-German (En-De)
    • WMT18 Chinese-English (Zh-En)
  • Document Summarization:
    • CNN/Daily Mail
    • WikiSum
  • Character-based Machine Translation: The paper mentions using character-based models for this task, implying datasets suitable for it.

Key Insights and Ideas

  • Compressibility of Encoder Outputs: The paper strongly supports the hypothesis that encoder outputs are highly compressible. A significant portion of source information can be discarded without a substantial drop in generation quality.
  • Task and Language Dependency: The degree of compressibility and the impact of sparsity on quality vary across tasks and, to some extent, languages. Summarization tasks appear to be more tolerant to sparsity than machine translation.
  • Information Hierarchy: LODROP learns to prioritize certain types of information. It tends to prune punctuation and function words (e.g., determiners, prepositions) while retaining content words (verbs, nouns). This suggests a learned hierarchy of importance for different linguistic elements.
  • Subword vs. Character Level: While subword-based models benefit from LODROP's efficiency gains, character-based models also see improvements, especially with longer sequences, indicating LODROP's utility in handling long inputs.
  • Rule-based Sparsity: Manually defined sparsity patterns based on linguistic features (like POS tags or word frequency) can achieve similar compression rates and quality to LODROP on specific tasks, but they lack the generalization ability of the data-driven LODROP.

Key Results

  • Sparsity and Quality: LODROP can prune around 40-70% of source encodings with only a marginal decrease in generation quality (e.g., -0.5 BLEU or -0.1 ROUGE-L).
  • Decoding Speedup: LODROP yields significant decoding speedups, particularly for longer sequences:
  • Up to 1.65x on document summarization (WikiSum).
  • Up to 1.20x on character-based machine translation.
  • Attention Re-routing: Analysis shows that when encoder outputs are pruned, the Transformer model adapts by spreading attention more evenly among the retained encodings and by encoding more context information into these representations.
  • Rule-based Patterns: Rule-based patterns achieved comparable results to LODROP on WMT14 En-De (e.g., POS Pattern with 46.7% sparsity and 27.11 BLEU vs. LODROP's 46.7% sparsity and 27.06 BLEU), but performed poorly on summarization tasks.

Limitations and Future Work

  • Sparsity Control: The authors note that the sparsity induced by LODROP is highly task-dependent and difficult to manipulate directly. They suggest developing novel algorithms to make sparsity induction more controllable.
  • Generalization of Rule-based Patterns: While rule-based patterns can mimic LODROP's behavior on specific tasks, they do not generalize well to different tasks, highlighting the advantage of data-driven approaches like LODROP.
  • Early Encoder Sparsification: Inserting LODROP layers earlier in the encoder stack did not yield significant benefits and even led to a slight performance drop, suggesting that lower-level information might be crucial for initial understanding.
  • Future Work: The authors aim to develop more controllable sparsity induction methods and further explore the interplay between sparsity and model understanding.

Summary updated

BibTeX
@inproceedings{zhang2021sparsifying,
  author = {Biao Zhang and Ivan Titov and Rico Sennrich},
  title = {{On Sparsifying Encoder Outputs in Sequence-to-Sequence Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=xF_VX9oo0OC},
}

Loading related content...

Figure