Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers

Woomin Song, Sai Muralidhar Jayanthi, Srikanth Ronanki, Kanthashree Mysore Sathyendra, Jinwoo Shin, Aram Galstyan, Shubham Katiyar, Sravan Babu Bodapati

Abstract

REFORM is a novel inference framework designed to enable Transformers to process extremely long contexts (up to 1M tokens) with high fidelity and minimal overhead. By combining recurrent chunked processing with a targeted "gather-and-recompute" strategy, REFORM achieves over 50% performance gains on standard long-context benchmarks while simultaneously reducing inference latency by 30%.

Introduction

As Large Language Models (LLMs) are increasingly deployed for repository-level coding, extensive document analysis, and multi-modal processing, the context window has emerged as a critical bottleneck. Standard Transformer architectures suffer from quadratic computational complexity $O(L^2)$, making the processing of million-token sequences prohibitively expensive.

Existing approaches typically fall into two categories: recurrent/compression methods, which suffer from information loss ("forgetting"), and random access methods, which incur massive memory and latency overhead by maintaining the full Key-Value (KV) cache. REFORM introduces a "middle way" that maintains the efficiency of recurrence without the permanent information loss of standard eviction strategies.

Methodology

REFORM (REcurrent chunked Forwarding with On-demand cache RecoMputation) operates in two distinct phases to balance memory efficiency with retrieval accuracy.

Phase 1: Encoding with Early Exit

The model processes input in manageable chunks (e.g., 32k tokens). During this phase, the system performs KV cache compression using eviction strategies to keep only "heavy hitter" tokens active. Crucially, REFORM extracts lightweight, token-level Cross-Layer Context Embeddings from specific "retrieval heads" identified as being superior at capturing relevant information. Because these retrieval-capable embeddings are primarily located in the lower and middle layers, REFORM employs an Early Exit strategy, skipping upper-layer processing during encoding to save computation.

Phase 2: On-Demand Recomputation

Upon receiving a query, REFORM enters its second phase:

  1. Gather: The query's embeddings are compared against the stored context embeddings using cosine similarity.
  2. Identify: The system identifies the most relevant historical tokens (significant inputs).
  3. Recompute: REFORM recomputes the full, high-fidelity KV cache for only these selected tokens across all layers.
Figure 1: The REFORM framework overview, illustrating the two-phase process of context encoding and on-demand cache recomputation.

This approach ensures the model has "perfect memory" for the specific context segments required for the current query, avoiding the staleness of compressed caches.

Experimental Results

The authors evaluated REFORM across a diverse suite of benchmarks, including synthetic stress tests (RULER, BABILong), realistic long-context tasks ($\infty$-Bench, RepoEval), and multi-modal tests (MM-NIAH).

REFORM demonstrated significant improvements over state-of-the-art baselines. On the RULER benchmark at a 1M context length, REFORM achieved a 52% performance gain over the best-performing baselines. On the BABILong multi-hop reasoning task, it saw a 34% gain. Furthermore, REFORM reduced inference time by 30% and peak memory usage by 5% compared to methods like InfLLM.

Figure 5: Performance comparison on RULER and BABILong benchmarks across various context lengths, showing REFORM's superior accuracy.

Key Insights

  • Retrieval Head Specialization: Not all attention heads are equally capable of retrieval. By targeting specific heads for embedding extraction, REFORM achieves higher accuracy than methods using standard hidden states.
  • Efficiency of Targeted Recomputation: While recomputation is often viewed as a bottleneck, performing it for a small, targeted subset of tokens is significantly more efficient than the latency penalties associated with CPU-offloading or the accuracy loss of permanent compression.
  • Layer Specialization: The discovery that retrieval embeddings are concentrated in the lower 70% of the model allows for the Early Exit strategy, providing a computational speedup without sacrificing retrieval precision.

Conclusion and Future Work

REFORM successfully bridges the gap between memory-efficient recurrence and high-fidelity random access. While highly effective, the system's accuracy remains dependent on the initial "Gather" stage's ability to identify relevant tokens. Future work may involve integrating these processes into optimized kernels like Flash Attention and expanding the framework to handle more diverse data modalities such as audio and video.

Summary updated

BibTeX
@inproceedings{song2025compress,
  author = {Woomin Song and Sai Muralidhar Jayanthi and Srikanth Ronanki and Kanthashree Mysore Sathyendra and Jinwoo Shin and Aram Galstyan and Shubham Katiyar and Sravan Babu Bodapati},
  title = {{Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=UJSaY7p53L},
}
Figure