Abstract
Long document re-ranking has been a challenging problem for neural re-rankers based on deep language models like BERT. Early work breaks the documents into short passage-like chunks. These chunks are independently mapped to scalar scores or latent vectors, which are then pooled into a final relevance score. These encode-and-pool methods however inevitably introduce an information bottleneck: the low dimension representations. In this paper, we propose instead to model full query-to-document interaction, leveraging the attention operation and modular Transformer re-ranker framework. First, document chunks are encoded independently with an encoder module. An interaction module then encodes the query and performs joint attention from the query to all document chunk representations. We demonstrate that the model can use this new degree of freedom to aggregate important information from the entire document. Our experiments show that this design produces effective re-ranking on two classical IR collections Robust04 and ClueWeb09, and a large-scale supervised collection MS-MARCO document ranking.
Introduction
In modern Information Retrieval (IR), a two-stage pipeline is standard: a fast retriever identifies a candidate set of documents, followed by a powerful neural re-ranker that sorts them by relevance. While Transformer-based models like BERT have set new benchmarks for this task, they face a significant structural limitation: a fixed input length, typically capped at 512 tokens.
For long documents, the traditional solution is to "chunk" the text into smaller segments, score them independently, and aggregate these scores through max or mean pooling. This approach creates a significant information bottleneck. By reducing a rich, multi-paragraph document into a single scalar or low-dimensional vector before considering the query's full context, granular signals are often lost. To address this, we introduce MORES+, a modular re-ranking framework designed to handle long documents through a joint query-to-all-chunk attention mechanism.
The MORES+ Architecture
MORES+ builds upon the Modular Re-ranker (MORES) framework to decouple document encoding from query interaction. The architecture is divided into two primary components:
1. Independent Document Encoding
The document is partitioned into chunks (e.g., 512 tokens each). Each chunk is processed by a Transformer encoder independently of the query. This design offers a major efficiency advantage: document representations can be pre-computed and stored offline, significantly reducing the computational load during the online re-ranking phase.
2. Joint Query-to-All-Chunk Attention
The core innovation of MORES+ is the Interaction Module. Rather than scoring chunks in isolation, this module takes the query and performs cross-attention across the representations of all document chunks simultaneously.
To implement this effectively, we utilize BART, a sequence-to-sequence model. We employ BART’s encoder for the document chunks and its decoder—which is natively designed for cross-attention—as the interaction module. This allows the query to "scan" the entire document, performing a "soft" selection of relevant information scattered across different sections. This mimics a human reader who might refer back to various parts of a text to synthesize an answer to a complex question.
Experimental Setup
We evaluated MORES+ on three major IR benchmarks characterized by documents that exceed the standard 512-token limit:
- Robust04: A classic collection with an average of 0.7K tokens per document.
- ClueWeb09: A web crawl dataset with an average of 3.3K tokens per document.
- MS-MARCO Document Ranking: A large-scale supervised dataset based on real-world Bing searches, averaging 1.3K tokens per document.
Testing was conducted using both "Title" queries (short keyword-based) and "Description" queries (longer, natural language sentences) to assess the model's versatility.
Key Results and Insights
State-of-the-Art Performance
MORES+ achieved state-of-the-art results on Robust04 and ClueWeb09, particularly for natural language "Description" queries. It outperformed both traditional "encode-and-pool" methods and more recent sparse-attention models. On the competitive MS-MARCO Document Ranking leaderboard, MORES+ (integrated with a dense retriever) achieved the second-best global performance at the time of the study.
Effective Information Aggregation
A critical finding of our ablation studies is the model's monotonic improvement as document length increases. Unlike models that become "confused" or diluted by extra text, MORES+ consistently improved in ranking accuracy as more document chunks (up to 3,000 tokens) were provided. This confirms that the joint attention mechanism successfully aggregates relevant signals from across the entire document.
The Natural Language Advantage
The model's architecture is uniquely suited for complex queries. While keyword searches often rely on simple term matching, natural language questions require a deeper understanding of context. MORES+ provides the necessary granular interaction to capture these nuances, leading to its superior performance on description-based tasks.
Conclusion
MORES+ addresses the long-standing challenge of the 512-token barrier in neural re-ranking. By replacing rigid pooling methods with a flexible interaction module, the framework allows for comprehensive query-to-document attention. This approach not only improves retrieval effectiveness on long-form content but also maintains efficiency through decoupled document encoding. Future work will explore further optimizations for raw inference speed and the scalability of the interaction module for extremely long-form texts such as books.
Summary updated