Input-length-shortening and text generation via attention values

Neset Özkan Tan, Alex Yuxuan Peng, Joshua Bensemann, Qiming Bao, Tim Hartill, Mark Gahegan, Michael WitbrockarXiv 2023 · DOI · OpenReview

Introduction

Transformer-based models, such as BERT, have revolutionized natural language processing but suffer from a significant computational bottleneck: quadratic complexity. As input sequences grow, memory and processing requirements increase at a squared rate, typically capping input lengths at 512 tokens. While many architectural solutions have been proposed, this research explores a data-centric alternative: can we identify and retain only the most essential tokens using the model's own internal attention mechanisms?

Methodology

The researchers investigated two primary methods for input compression and one for text reconstruction:

  1. Attention-Based Pruning: By extracting attention weights from a pre-trained BERT model and summing scores across all 12 heads in a specific layer, tokens were assigned relevance scores. This allowed for the removal of "noise" tokens while preserving those critical for classification.
  2. Similarity-Based Filtering: As a baseline, Sentence-BERT (SBERT) was used to identify and prune semantically redundant sentences.
  3. Conditional Generation: To test the information density of the pruned tokens, GPT-2 was fine-tuned to reconstruct full reviews using only the sentiment label and the top 10-20% of tokens identified by BERT.

The "Early Bird" Advantage

A pivotal finding of the study is that not all layers are created equal for the purpose of filtering. Contrary to the intuition that deeper layers contain more semantic nuance, the researchers discovered that Layer 1 provides the most reliable signal for identifying critical tokens.

Accuracy vs. Tokens Kept across different BERT layers, showing the superior performance of the 1st layer for filtering.

As shown in the figure above, filtering based on the first layer's attention weights maintains significantly higher accuracy as the number of tokens is reduced compared to later layers. This suggests that the initial attention mechanism effectively captures the "DNA" of the text's meaning.

Key Results and Efficiency

The experiments across IMDB, ArXiv, and FEVER datasets demonstrated remarkable redundancy in natural language for classification tasks:

  • Extreme Compression: On the IMDB sentiment dataset, retaining just 6% of the original tokens still yielded an accuracy of 86.5%.
  • Scientific Abstracts: Using SciBERT on the ArXiv dataset, cutting the input length by 50% resulted in a negligible accuracy drop (from 92.03% to 91.34%).

Synthetic Text Generation

The study further proved that these high-attention tokens are sufficient for high-quality text generation. GPT-2 was able to generate synthetic reviews that human evaluators found as cohesive as the originals. Crucially, these synthetic reviews preserved the original sentiment so well that a fine-tuned BERT model could classify them with nearly the same accuracy as the original human-written text.

Classification accuracy of BERT on original vs. synthetic (imitated) reviews, demonstrating sentiment preservation.

Conclusion

This research demonstrates that attention-based filtering, particularly using the first layer of BERT, is a highly effective method for reducing the computational burden of Transformer models. By treating attention values as a "control knob," researchers can achieve significant input compression and even guide generative models to produce high-fidelity synthetic data. Future work may explore whether these findings scale to modern Large Language Models (LLMs) and if this pruning can be integrated into the pre-training phase to save massive computational resources.

Summary updated

Abstract

Identifying words that impact a task's performance more than others is a challenge in natural language processing. Transformers models have recently addressed this issue by incorporating an attention mechanism that assigns greater attention (i.e., relevance) scores to some words than others. Because of the attention mechanism's high computational cost, transformer models usually have an input-length limitation caused by hardware constraints. This limitation applies to many transformers, including the well-known bidirectional encoder representations of the transformer (BERT) model. In this paper, we examined BERT's attention assignment mechanism, focusing on two questions: (1) How can attention be employed to reduce input length? (2) How can attention be used as a control mechanism for conditional text generation? We investigated these questions in the context of a text classification task. We discovered that BERT's early layers assign more critical attention scores for text classification tasks compared to later layers. We demonstrated that the first layer's attention sums could be used to filter tokens in a given sequence, considerably decreasing the input length while maintaining good test accuracy. We also applied filtering, which uses a compute-efficient semantic similarities algorithm, and discovered that retaining approximately 6\% of the original sequence is sufficient to obtain 86.5\% accuracy. Finally, we showed that we could generate data in a stable manner and indistinguishable from the original one by only using a small percentage (10\%) of the tokens with high attention scores according to BERT's first layer.
BibTeX
@misc{tan2023input,
  author = {Neset Özkan Tan and Alex Yuxuan Peng and Joshua Bensemann and Qiming Bao and Tim Hartill and Mark Gahegan and Michael Witbrock},
  title = {{Input-length-shortening and text generation via attention values}},
  year = {2023},
  eprint = {2303.07585},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.07585},
}
Figure