Recurrent Attention Networks for Long-text Modeling

Xianming Li, Zongxi Li, Xiaotian Luo, Haoran Xie, Xing Lee, Yingbin Zhao, Fu Lee Wang, Qing Li

Abstract

Standard Transformer architectures face a quadratic computational bottleneck, limiting their effectiveness for long-document processing. We present Recurrent Attention Networks (RAN), a novel architecture that integrates window-level recurrence with self-attention. By utilizing a Global Perception Cell (GPC) to maintain state across non-overlapping windows and a Memory Review mechanism for global information consolidation, RAN achieves state-of-the-art performance across language modeling and long-text classification tasks while maintaining high computational efficiency. Our results demonstrate that RAN significantly outperforms existing chunking and linear attention models in both accuracy and training speed.

1. Introduction

The self-attention mechanism, while powerful, suffers from a "quadratic bottleneck" where computational costs grow with the square of the sequence length ($O(n^2)$). This limitation effectively restricts most models to a 512-token context window, making them ill-suited for long-form content such as legal filings, scientific reports, or literature.

Existing solutions generally fall into two categories: linear attention approximations and chunking strategies. Linear attention attempts to achieve $O(n)$ complexity but often results in performance degradation. Chunking strategies break text into independent segments, which either loses global context or, if using traditional token-level RNNs to link segments, destroys the model's ability to compute in parallel. RAN addresses these issues by introducing a window-level recurrent strategy that preserves the parallel processing power of Transformers while maintaining the long-range memory of Recurrent Neural Networks (RNNs).

2. Methodology

The core innovation of the Recurrent Attention Network (RAN) is its hierarchical approach to sequence modeling, combining local attention with global state persistence.

2.1 Window-based Processing and the GPC

Instead of processing a document as a single sequence or a series of isolated tokens, RAN divides text into non-overlapping windows (e.g., 256 tokens). Within each window, the model employs a Global Perception Cell (GPC). This learnable vector acts as the model's memory; it is concatenated with the input tokens of a window, allowing the self-attention mechanism to facilitate interactions between local tokens and the global state. Once a window is processed, the updated GPC is passed to the subsequent window via a residual connection and layer normalization, serving as a "contextual bridge."

2.2 Positional Encoding and Memory Review

To maintain spatial awareness across long sequences, RAN utilizes Rotary Positional Embeddings (RoPE), which offer greater flexibility for varying sequence lengths than absolute positional encodings.

Furthermore, RAN introduces a Memory Review Mechanism. After all windows have been processed, the model performs a cross-attention operation where the final sequence output "reviews" the intermediate GPC vectors generated at each window step. This mechanism mimics human cognitive processes, where a reader consolidates "highlights" from an entire text to form a final understanding.

3. Experimental Evaluation

RAN was evaluated across a diverse suite of NLP tasks, including long-text classification (AGNews, 20NewsGroups, Arxiv, EURLEX-57K), language modeling (WikiText-103), and named entity recognition (OntoNotesV5.0).

3.1 Performance and Efficiency

In classification benchmarks, RAN outperformed BERT, Longformer, and ToBERT on five out of six datasets. Notably, on the Arxiv scientific paper dataset, RAN achieved a 2.2% improvement over the previous state-of-the-art chunking model. In language modeling, RAN reached a perplexity of 22.76 on WikiText-103, surpassing Transformer-XL.

Computational efficiency is a primary advantage of the RAN architecture. By restricting token-to-token attention to local windows while mediating global context through the GPC, the model achieves significant speedups. On the Arxiv dataset, RAN (96M parameters) required only 5,393 seconds per epoch, compared to 26,536 seconds for Longformer—representing a nearly 5x improvement in training throughput.

3.2 Ablation Studies

Ablation analysis confirmed that the Memory Review mechanism is critical for document-level tasks; removing this step resulted in a 2.51% performance drop. This suggests that simple state-forwarding is insufficient for complex long-text modeling; the model requires a dedicated mechanism to revisit historical context.

4. Conclusion and Future Work

Recurrent Attention Networks provide a scalable and efficient solution for long-text processing by bridging the gap between the parallelization of Transformers and the recurrent memory of RNNs. While the model currently limits direct token-to-token interaction to the window size, its ability to maintain global context through the GPC and Memory Review makes it a powerful tool for large-scale document analysis. Future research will focus on extending RAN to multi-modal inputs and exploring deeper architectures to further enhance its long-range dependency handling.

Summary updated

Abstract

Self-attention-based models have achieved remarkable progress in short-text mining. However, the quadratic computational complexities restrict their application in long text processing. Prior works have adopted the chunking strategy to divide long documents into chunks and stack a self-attention backbone with the recurrent structure to extract semantic representation. Such an approach disables parallelization of the attention mechanism, significantly increasing the training cost and raising hardware requirements. Revisiting the self-attention mechanism and the recurrent structure, this paper proposes a novel long-document encoding model, Recurrent Attention Network (RAN), to enable the recurrent operation of self-attention. Combining the advantages from both sides, the well-designed RAN is capable of extracting global semantics in both token-level and document-level representations, making it inherently compatible with both sequential and classification tasks, respectively. Furthermore, RAN is computationally scalable as it supports parallelization on long document processing. Extensive experiments demonstrate the long-text encoding ability of the proposed RAN model on both classification and sequential tasks, showing its potential for a wide range of applications.
BibTeX
@inproceedings{li2023recurrent,
  author = {Xianming Li and Zongxi Li and Xiaotian Luo and Haoran Xie and Xing Lee and Yingbin Zhao and Fu Lee Wang and Qing Li},
  title = {{Recurrent Attention Networks for Long-text Modeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.188},
}
Figure