Abstract
REFORM is a novel inference framework designed to enable Transformers to process extremely long contexts (up to 1M tokens) with high fidelity and minimal overhead. By combining recurrent chunked processing with a targeted "gather-and-recompute" strategy, REFORM achieves over 50% performance gains on standard long-context benchmarks while simultaneously reducing inference latency by 30%.
Introduction
As Large Language Models (LLMs) are increasingly deployed for repository-level coding, extensive document analysis, and multi-modal processing, the context window has emerged as a critical bottleneck. Standard Transformer architectures suffer from quadratic computational complexity $O(L^2)$, making the processing of million-token sequences prohibitively expensive.
Existing approaches typically fall into two categories: recurrent/compression methods, which suffer from information loss ("forgetting"), and random access methods, which incur massive memory and latency overhead by maintaining the full Key-Value (KV) cache. REFORM introduces a "middle way" that maintains the efficiency of recurrence without the permanent information loss of standard eviction strategies.
Methodology
REFORM (REcurrent chunked Forwarding with On-demand cache RecoMputation) operates in two distinct phases to balance memory efficiency with retrieval accuracy.
Phase 1: Encoding with Early Exit
The model processes input in manageable chunks (e.g., 32k tokens). During this phase, the system performs KV cache compression using eviction strategies to keep only "heavy hitter" tokens active. Crucially, REFORM extracts lightweight, token-level Cross-Layer Context Embeddings from specific "retrieval heads" identified as being superior at capturing relevant information. Because these retrieval-capable embeddings are primarily located in the lower and middle layers, REFORM employs an Early Exit strategy, skipping upper-layer processing during encoding to save computation.
Phase 2: On-Demand Recomputation
Upon receiving a query, REFORM enters its second phase:
- Gather: The query's embeddings are compared against the stored context embeddings using cosine similarity.
- Identify: The system identifies the most relevant historical tokens (significant inputs).
- Recompute: REFORM recomputes the full, high-fidelity KV cache for only these selected tokens across all layers.
This approach ensures the model has "perfect memory" for the specific context segments required for the current query, avoiding the staleness of compressed caches.
Experimental Results
The authors evaluated REFORM across a diverse suite of benchmarks, including synthetic stress tests (RULER, BABILong), realistic long-context tasks ($\infty$-Bench, RepoEval), and multi-modal tests (MM-NIAH).
REFORM demonstrated significant improvements over state-of-the-art baselines. On the RULER benchmark at a 1M context length, REFORM achieved a 52% performance gain over the best-performing baselines. On the BABILong multi-hop reasoning task, it saw a 34% gain. Furthermore, REFORM reduced inference time by 30% and peak memory usage by 5% compared to methods like InfLLM.
Key Insights
- Retrieval Head Specialization: Not all attention heads are equally capable of retrieval. By targeting specific heads for embedding extraction, REFORM achieves higher accuracy than methods using standard hidden states.
- Efficiency of Targeted Recomputation: While recomputation is often viewed as a bottleneck, performing it for a small, targeted subset of tokens is significantly more efficient than the latency penalties associated with CPU-offloading or the accuracy loss of permanent compression.
- Layer Specialization: The discovery that retrieval embeddings are concentrated in the lower 70% of the model allows for the Early Exit strategy, providing a computational speedup without sacrificing retrieval precision.
Conclusion and Future Work
REFORM successfully bridges the gap between memory-efficient recurrence and high-fidelity random access. While highly effective, the system's accuracy remains dependent on the initial "Gather" stage's ability to identify relevant tokens. Future work may involve integrating these processes into optimized kernels like Flash Attention and expanding the framework to handle more diverse data modalities such as audio and video.
Summary updated