Abstract
Standard Transformer architectures face a quadratic computational bottleneck, limiting their effectiveness for long-document processing. We present Recurrent Attention Networks (RAN), a novel architecture that integrates window-level recurrence with self-attention. By utilizing a Global Perception Cell (GPC) to maintain state across non-overlapping windows and a Memory Review mechanism for global information consolidation, RAN achieves state-of-the-art performance across language modeling and long-text classification tasks while maintaining high computational efficiency. Our results demonstrate that RAN significantly outperforms existing chunking and linear attention models in both accuracy and training speed.
1. Introduction
The self-attention mechanism, while powerful, suffers from a "quadratic bottleneck" where computational costs grow with the square of the sequence length ($O(n^2)$). This limitation effectively restricts most models to a 512-token context window, making them ill-suited for long-form content such as legal filings, scientific reports, or literature.
Existing solutions generally fall into two categories: linear attention approximations and chunking strategies. Linear attention attempts to achieve $O(n)$ complexity but often results in performance degradation. Chunking strategies break text into independent segments, which either loses global context or, if using traditional token-level RNNs to link segments, destroys the model's ability to compute in parallel. RAN addresses these issues by introducing a window-level recurrent strategy that preserves the parallel processing power of Transformers while maintaining the long-range memory of Recurrent Neural Networks (RNNs).
2. Methodology
The core innovation of the Recurrent Attention Network (RAN) is its hierarchical approach to sequence modeling, combining local attention with global state persistence.
2.1 Window-based Processing and the GPC
Instead of processing a document as a single sequence or a series of isolated tokens, RAN divides text into non-overlapping windows (e.g., 256 tokens). Within each window, the model employs a Global Perception Cell (GPC). This learnable vector acts as the model's memory; it is concatenated with the input tokens of a window, allowing the self-attention mechanism to facilitate interactions between local tokens and the global state. Once a window is processed, the updated GPC is passed to the subsequent window via a residual connection and layer normalization, serving as a "contextual bridge."
2.2 Positional Encoding and Memory Review
To maintain spatial awareness across long sequences, RAN utilizes Rotary Positional Embeddings (RoPE), which offer greater flexibility for varying sequence lengths than absolute positional encodings.
Furthermore, RAN introduces a Memory Review Mechanism. After all windows have been processed, the model performs a cross-attention operation where the final sequence output "reviews" the intermediate GPC vectors generated at each window step. This mechanism mimics human cognitive processes, where a reader consolidates "highlights" from an entire text to form a final understanding.
3. Experimental Evaluation
RAN was evaluated across a diverse suite of NLP tasks, including long-text classification (AGNews, 20NewsGroups, Arxiv, EURLEX-57K), language modeling (WikiText-103), and named entity recognition (OntoNotesV5.0).
3.1 Performance and Efficiency
In classification benchmarks, RAN outperformed BERT, Longformer, and ToBERT on five out of six datasets. Notably, on the Arxiv scientific paper dataset, RAN achieved a 2.2% improvement over the previous state-of-the-art chunking model. In language modeling, RAN reached a perplexity of 22.76 on WikiText-103, surpassing Transformer-XL.
Computational efficiency is a primary advantage of the RAN architecture. By restricting token-to-token attention to local windows while mediating global context through the GPC, the model achieves significant speedups. On the Arxiv dataset, RAN (96M parameters) required only 5,393 seconds per epoch, compared to 26,536 seconds for Longformer—representing a nearly 5x improvement in training throughput.
3.2 Ablation Studies
Ablation analysis confirmed that the Memory Review mechanism is critical for document-level tasks; removing this step resulted in a 2.51% performance drop. This suggests that simple state-forwarding is insufficient for complex long-text modeling; the model requires a dedicated mechanism to revisit historical context.
4. Conclusion and Future Work
Recurrent Attention Networks provide a scalable and efficient solution for long-text processing by bridging the gap between the parallelization of Transformers and the recurrent memory of RNNs. While the model currently limits direct token-to-token interaction to the window size, its ability to maintain global context through the GPC and Memory Review makes it a powerful tool for large-scale document analysis. Future research will focus on extending RAN to multi-modal inputs and exploring deeper architectures to further enhance its long-range dependency handling.
Summary updated