Introduction
Transformer-based models, such as BERT, have revolutionized natural language processing but suffer from a significant computational bottleneck: quadratic complexity. As input sequences grow, memory and processing requirements increase at a squared rate, typically capping input lengths at 512 tokens. While many architectural solutions have been proposed, this research explores a data-centric alternative: can we identify and retain only the most essential tokens using the model's own internal attention mechanisms?
Methodology
The researchers investigated two primary methods for input compression and one for text reconstruction:
- Attention-Based Pruning: By extracting attention weights from a pre-trained BERT model and summing scores across all 12 heads in a specific layer, tokens were assigned relevance scores. This allowed for the removal of "noise" tokens while preserving those critical for classification.
- Similarity-Based Filtering: As a baseline, Sentence-BERT (SBERT) was used to identify and prune semantically redundant sentences.
- Conditional Generation: To test the information density of the pruned tokens, GPT-2 was fine-tuned to reconstruct full reviews using only the sentiment label and the top 10-20% of tokens identified by BERT.
The "Early Bird" Advantage
A pivotal finding of the study is that not all layers are created equal for the purpose of filtering. Contrary to the intuition that deeper layers contain more semantic nuance, the researchers discovered that Layer 1 provides the most reliable signal for identifying critical tokens.
As shown in the figure above, filtering based on the first layer's attention weights maintains significantly higher accuracy as the number of tokens is reduced compared to later layers. This suggests that the initial attention mechanism effectively captures the "DNA" of the text's meaning.
Key Results and Efficiency
The experiments across IMDB, ArXiv, and FEVER datasets demonstrated remarkable redundancy in natural language for classification tasks:
- Extreme Compression: On the IMDB sentiment dataset, retaining just 6% of the original tokens still yielded an accuracy of 86.5%.
- Scientific Abstracts: Using SciBERT on the ArXiv dataset, cutting the input length by 50% resulted in a negligible accuracy drop (from 92.03% to 91.34%).
Synthetic Text Generation
The study further proved that these high-attention tokens are sufficient for high-quality text generation. GPT-2 was able to generate synthetic reviews that human evaluators found as cohesive as the originals. Crucially, these synthetic reviews preserved the original sentiment so well that a fine-tuned BERT model could classify them with nearly the same accuracy as the original human-written text.
Conclusion
This research demonstrates that attention-based filtering, particularly using the first layer of BERT, is a highly effective method for reducing the computational burden of Transformer models. By treating attention values as a "control knob," researchers can achieve significant input compression and even guide generative models to produce high-fidelity synthetic data. Future work may explore whether these findings scale to modern Large Language Models (LLMs) and if this pruning can be integrated into the pre-training phase to save massive computational resources.
Summary updated