Summary
Adaptive Retention is an architecture-agnostic mechanism that enables Transformer models to learn which tokens are critical for a given task at each layer. By employing differentiable probabilistic gates to prune less informative tokens under a strict memory budget, the method maintains over 95% of baseline performance while reducing peak memory usage by up to 45% and nearly doubling throughput.
Introduction: The Quadratic Scaling Challenge
The primary bottleneck in standard Transformer architectures is the quadratic scaling of memory and computational requirements relative to sequence length ($O(n^2)$). This scaling makes the processing of long-form content—such as legal documents, scientific manuscripts, and medical records—prohibitively expensive for many real-world applications.
Existing solutions often rely on fixed attention patterns (e.g., sliding windows) or static heuristics (e.g., keeping only 'heavy hitter' tokens). However, these approaches lack the flexibility to adapt to specific task requirements or different layers of the network. Adaptive Retention addresses this by treating token selection as a learnable, layer-wise optimization problem.
Methodology: Adaptive Retention
The authors propose a probabilistic framework for dynamic token selection that can be integrated into existing Transformer encoders without modifying the underlying attention mechanism.
1. Retention Predictor and Gating
At each Transformer block, a lightweight Retention Predictor (a small MLP) processes token representations to generate an importance score ($p_t$). This score represents the probability that a token should be retained for subsequent layers.
2. Differentiable Training via Hard-Concrete Relaxation
To enable end-to-end training, the model must backpropagate through the discrete decision of keeping or dropping a token. The authors utilize a Hard-Concrete relaxation of the Bernoulli distribution. This technique provides a differentiable approximation of binary gates ($z_t$), allowing the model to learn importance weights through standard gradient descent.
3. Budget-Aware Optimization
To enforce memory efficiency, the training objective includes a Lagrange multiplier penalty. This mathematical constraint forces the model to minimize the number of retained tokens to meet a predefined global memory budget ($M$). If the model exceeds this budget, the loss increases, driving the selection mechanism to become more selective.
Experimental Evaluation
The framework was evaluated using DistilBERT and Longformer-base backbones across a variety of benchmarks:
- Short-Context Tasks: SST-2 (sentiment), IMDb (reviews), and CUAD (legal classification).
- Long-Context Tasks: ArXiv (summarization), QASPER (QA), and PubMed RCT (medical abstracts).
Results and Comparative Analysis
Adaptive Retention demonstrates a superior trade-off between performance and efficiency compared to both heuristic pruning and specialized sparse architectures.
Performance Retention
Empirical results show that keeping only 30–50% of tokens preserves at least 95% of the full Transformer's performance. Notably, the method often matches or exceeds the performance of specialized models like BigBird and Longformer, despite being a simpler modification to the standard pipeline.
Efficiency Gains
- Memory: Peak memory usage was reduced by 35–45%.
- Throughput: The method achieved up to 1.8× improvement in throughput. On standard hardware (e.g., a 12 GB GPU), the optimized model processed batches nearly twice as fast as the dense baseline.
The 'Funnel' Effect
Analysis of the learned gates reveals a natural 'funnel' behavior: the model tends to retain more tokens in early layers to capture broad context and becomes increasingly selective in deeper layers. This suggests the model learns to distill information progressively as it approaches the final task head.
Conclusion and Future Directions
Adaptive Retention provides a robust, learnable alternative to static pruning heuristics. Its architecture-agnostic nature allows it to be 'dropped in' to most Transformer-based encoders to achieve significant efficiency gains with minimal performance loss.
Future work remains to extend this framework to autoregressive decoding (e.g., GPT-style models), which would require managing dynamic Key-Value (KV) caches. Additionally, exploring the stability of these probabilistic gates in massive-scale models (billions of parameters) is a promising avenue for further research.
Summary updated