On Sparsifying Encoder Outputs in Sequence-to-Sequence Models

Biao Zhang, Ivan Titov, Rico Sennrich

Abstract

Sequence-to-sequence models usually transfer all encoder outputs to the decoder for generation. In this work, by contrast, we hypothesize that these encoder outputs can be compressed to shorten the sequence delivered for decoding. We take Transformer as the testbed and introduce a layer of stochastic gates in-between the encoder and the decoder. The gates are regularized using the expected value of the sparsity-inducing $L_0$ penalty, resulting in completely masking-out a subset of encoder outputs. In other words, via joint training, the $L_0$DROP layer forces Transformer to route information through a subset of its encoder states. We investigate the effects of this sparsification on two machine translation and two summarization tasks. Experiments show that, depending on the task, around 40-70% of source encodings can be pruned without significantly compromising quality. The decrease of the output length endows $L_0$DROP with the potential of improving decoding efficiency, where it yields a speedup of up to 1.65x on document summarization tasks against the standard Transformer. We analyze the $L_0$DROP behaviour and observe that it exhibits systematic preferences for pruning certain word types, e.g., function words and punctuation get pruned most. Inspired by these observations, we explore the feasibility of specifying rule-based patterns that mask out encoder outputs based on information such as part-of-speech tags, word frequency and word position.

Introduction

Modern sequence-to-sequence models, particularly Transformers, operate under the assumption that the decoder must attend to every token representation generated by the encoder. While effective, this creates a significant computational bottleneck. For a source sequence of length $N$ and a target sequence of length $M$, the cross-attention mechanism incurs $O(NM)$ complexity.

However, not all words carry equal weight for the generation process. Punctuation or common function words often carry redundant information. This paper introduces $L_0$DROP, a method to identify and prune these redundant encoder outputs, thereby reducing the effective sequence length and accelerating decoding without sacrificing performance.

Methodology: The $L_0$DROP Layer

The core innovation is a Neural Sparsity Layer inserted between the encoder and decoder. This layer employs stochastic gates ($g_i$) for each encoder output. If $g_i = 1$, the representation is passed to the decoder; if $g_i = 0$, it is discarded.

To optimize these gates, the researchers utilized the Hard Concrete distribution, which allows for differentiable, nearly binary gating. By adding an $L_0$ penalty to the loss function, the model is incentivized to maximize sparsity (closing gates) while maintaining task performance. During inference, the pruned tokens are skipped entirely, enabling a modified attention algorithm that yields significant speedups.

Comparison of attention patterns between Vanilla, Sparse, and $L_0$DROP models.

Experimental Setup

The $L_0$DROP approach was evaluated across diverse tasks:

  • Machine Translation: WMT14 English-German (En-De) and WMT18 Chinese-English (Zh-En).
  • Document Summarization: CNN/Daily Mail and WikiSum. WikiSum, involving multi-document inputs truncated to 2048 subwords, served as a primary test for long-sequence efficiency.

Results and Analysis

Experiments demonstrate that a substantial portion of encoder outputs can be pruned with minimal impact on quality. In translation tasks, approximately 40% of outputs were removed with a loss of only ~0.5 BLEU. Summarization tasks proved even more compressible, with up to 71.5% pruning possible on WikiSum while maintaining strong ROUGE-L scores.

Performance metrics (BLEU and ROUGE-L) across different sparsity rates for translation and summarization tasks.

Information Bottleneck and Word Types

Analysis of the pruning patterns revealed that the model systematically targets specific word types. Punctuation and function words (e.g., "the", "a") are typically the first to be pruned. In contrast, content words like nouns and verbs are retained longer. Furthermore, when using Byte Pair Encoding (BPE), the model tends to keep the "head" subword and prune subsequent pieces, suggesting it learns to consolidate meaning into the initial token representation.

Efficiency Gains

The reduction in sequence length translated directly into decoding speedups. The $L_0$DROP layer achieved a 1.21× speedup on CNN/Daily Mail and a 1.65× speedup on the long-form WikiSum task. The benefits are most apparent in these "long-form" tasks where the $O(NM)$ complexity is most taxing.

Conclusion

$L_0$DROP demonstrates that sequence-to-sequence models can maintain high performance while ignoring a large fraction of encoder outputs. While the administrative overhead of the gating mechanism can offset gains for very short sequences, the benefits for document-level tasks are substantial. Future work may explore applying these pruning techniques earlier in the encoder stack to further optimize the encoding phase itself.

Summary updated

Abstract

Sequence-to-sequence models usually transfer all encoder outputs to the decoder for generation. In this work, by contrast, we hypothesize that these encoder outputs can be compressed to shorten the sequence delivered for decoding. We take Transformer as the testbed and introduce a layer of stochastic gates in-between the encoder and the decoder. The gates are regularized using the expected value of the sparsity-inducing L0penalty, resulting in completely masking-out a subset of encoder outputs. In other words, via joint training, the L0DROP layer forces Transformer to route information through a subset of its encoder states. We investigate the effects of this sparsification on two machine translation and two summarization tasks. Experiments show that, depending on the task, around 40-70% of source encodings can be pruned without significantly compromising quality. The decrease of the output length endows L0DROP with the potential of improving decoding efficiency, where it yields a speedup of up to 1.65x on document summarization tasks against the standard Transformer. We analyze the L0DROP behaviour and observe that it exhibits systematic preferences for pruning certain word types, e.g., function words and punctuation get pruned most. Inspired by these observations, we explore the feasibility of specifying rule-based patterns that mask out encoder outputs based on information such as part-of-speech tags, word frequency and word position.
BibTeX
@inproceedings{zhang2021sparsifying,
  author = {Biao Zhang and Ivan Titov and Rico Sennrich},
  title = {{On Sparsifying Encoder Outputs in Sequence-to-Sequence Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=xF_VX9oo0OC},
}
Figure