Abstract
Transformer attention scales quadratically with sequence length $O(n^2)$, limiting long-context use. We propose Adaptive Retention, a probabilistic, layer-wise token selection mechanism that learns which representations to keep under a strict global budget $M$. Retention is modeled with Bernoulli gates trained via a Hard-Concrete/variational relaxation and enforced with a simple top-M rule at inference, making the method differentiable and drop-in for standard encoders. Across classification, extractive QA, and long-document summarization, keeping only 30-50% of tokens preserves $\ge$ 95% of full-model performance while cutting peak memory by ~35-45% and improving throughput by up to ~1.8x. This architecture-agnostic approach delivers practical long-context efficiency without modifying base attention or task heads.
Introduction
The quadratic scaling of the self-attention mechanism remains the primary bottleneck for Transformer-based models. As input sequence length ($n$) increases, memory and computational requirements grow by $n^2$, making the processing of long-form content—such as legal contracts, scientific literature, or technical manuals—prohibitively expensive.
Existing efficiency solutions often fall into two categories: sparse attention patterns (e.g., Longformer) or heuristic-based pruning (e.g., H2O). However, these methods frequently rely on fixed patterns or human-designed rules that do not adapt to the specific semantic requirements of a task. This research introduces Adaptive Retention, a mechanism that allows a model to learn exactly which tokens are critical for a specific objective under a strict memory budget, effectively "learning what to remember."
Methodology
Adaptive Retention functions as a layer-wise token selection mechanism that can be integrated into standard Transformer encoders. The process is defined by four key components:
1. Probabilistic Gating
For every token at each layer, a lightweight scorer network predicts the probability that the token should be retained. This allows the model to assign importance scores dynamically based on the hidden representations at that specific depth.
2. The Hard-Concrete Relaxation
Token selection is inherently discrete (a token is either kept or dropped), which typically prevents the use of backpropagation. To overcome this, the authors employ a Hard-Concrete relaxation. This mathematical technique provides a continuous, differentiable approximation of Bernoulli gates during training. During inference, these "fuzzy" probabilities are converted back into hard binary decisions.
3. Lagrangian Budgeting
To prevent the model from simply retaining all tokens, a global memory budget ($M$) is enforced using a Lagrangian optimization objective. This acts as a constraint that penalizes the model if the total number of retained tokens across all layers exceeds the specified limit. This global approach allows the model to allocate its "memory budget" flexibly—spending more on information-dense layers and pruning more aggressively in redundant ones.
4. Top-M Inference
Once trained, the model utilizes a simple Top-M selection rule. At each layer, only the $M$ tokens with the highest retention scores are passed to the next layer. This results in a sequence length that effectively shrinks as data moves deeper into the model, leading to significant reductions in peak memory and computational overhead.
Experimental Setup
The efficacy of Adaptive Retention was evaluated using DistilBERT for short-context tasks and Longformer for long-document tasks. The benchmarks spanned six diverse datasets:
- Sentiment Analysis: SST-2 and IMDb.
- Scientific Document Processing: ArXiv and PubMed (summarization).
- Question Answering: QASPER (research paper QA).
- Legal Analysis: CUAD (contract clause classification).
Key Results and Insights
Performance and Efficiency
The experiments demonstrate that Adaptive Retention is highly effective at maintaining model utility while reducing resource consumption:
- High Fidelity: Keeping only 30–50% of tokens preserved $\ge$ 95% of the original full-model performance.
- Memory Reduction: Peak memory usage was reduced by 35–45%.
- Throughput Gains: The method achieved up to a 1.8x increase in throughput, allowing for faster inference on the same hardware.
Learned Importance vs. Heuristics
Unlike fixed pruning methods, Adaptive Retention discovers contextual importance. The model learns to discard "filler" tokens and redundant information that do not contribute to the final task. Interestingly, the researchers found that even with only 30% token retention, the model outperformed zero-shot results from significantly larger models like Llama 2 and Mistral on several benchmarks, suggesting that efficient memory management can be more impactful than raw parameter count.
Discussion and Limitations
While Adaptive Retention offers a robust, architecture-agnostic solution for encoders, there are several areas for future exploration:
- Decoder Architectures: This study focused on encoder-only tasks. Applying this to autoregressive decoders (like GPT-style models) is more complex due to the need for managing a dynamic Key-Value (KV) cache during generation.
- Scaling: The current experiments were conducted on models up to ~150M parameters. Further research is required to determine if these retention patterns hold for "frontier" models exceeding 70B parameters.
- Hyperparameter Tuning: The performance is sensitive to the "temperature" and "stretch" parameters of the Hard-Concrete relaxation, necessitating careful tuning during the training phase.
Conclusion
Adaptive Retention provides a practical path toward making Transformers more efficient without requiring specialized attention kernels or modifications to task-specific heads. By treating memory as a learnable budget, models can process longer contexts with significantly lower hardware requirements, bridging the gap between high-performance Transformers and resource-constrained environments.
Summary updated