TL;DR
Researchers at Carnegie Mellon University have developed MORES+, a modular re-ranking framework that overcomes the input length limits of standard Transformer models. By using a "joint query-to-all-chunk" attention mechanism, the model can look across an entire long document simultaneously to find relevant information, achieving state-of-the-art results on major Information Retrieval (IR) benchmarks.
What Problem Does This Paper Address?
Most modern search engines use a two-stage process: a fast "retriever" grabs a few hundred documents, and a powerful "re-ranker" (usually based on BERT) sorts them by relevance. However, BERT and its cousins have a hard limit on input length—typically 512 tokens.
To handle long documents, researchers usually "chunk" the text into smaller pieces, score them independently, and then average or max-pool the results. This creates an information bottleneck: by the time the model tries to calculate a final score, the rich, granular details of the document have been squashed into a single number or a low-dimensional vector. Other methods use "sparse attention" to handle longer sequences, but these often sacrifice the model's ability to understand complex relationships between words.
Methods
The authors introduce MORES+, which builds on the Modular Re-ranker (MORES) framework. Its architecture consists of two main parts:
- Independent Document Encoding: The document is split into chunks (e.g., 512 tokens each). Each chunk is processed by a Transformer encoder independently of the query. This is a huge win for efficiency because these representations can be pre-computed and stored offline.
- Joint Query-to-All-Chunk Attention: This is the core innovation. Instead of scoring chunks one by one, an "Interaction Module" takes the query and performs cross-attention across the representations of all document chunks at once.
Crucially, the authors initialize MORES+ using BART, a sequence-to-sequence model. They use BART’s encoder for the document chunks and its decoder (which is already designed for cross-attention) for the interaction module. This allows the query to "scan" the entire document to pick up relevant signals, much like a human reader refers back to an article to answer a specific question.
Datasets
The model was evaluated on three major benchmarks that feature documents significantly longer than the 512-token limit:
- Robust04: A classic IR collection (avg. 0.7K tokens/doc).
- ClueWeb09: A large web crawl dataset (avg. 3.3K tokens/doc).
- MS-MARCO Document Ranking: A massive, real-world dataset based on Bing searches (avg. 1.3K tokens/doc).
The researchers tested both "Title" queries (short keywords) and "Description" queries (full natural language sentences).
Key Insights and Ideas
- Soft Information Selection: Unlike "key-block" methods that try to pick the single best paragraph to represent a document, MORES+ uses "soft" selection. The attention mechanism can distribute its focus across multiple relevant snippets scattered throughout a long text.
- The Natural Language Advantage: The model shows its greatest strength on "Description" queries. While keyword searches don't always require deep understanding, complex natural language questions benefit immensely from the granular query-to-document interaction MORES+ provides.
- Efficiency via Decoupling: Because the document encoding is query-independent, the "online" part of the re-ranking (the interaction) is computationally light. The authors note this can speed up re-ranking by tens of times compared to standard "all-in-one" Transformer models.
Key Results
- State-of-the-Art Performance: MORES+ achieved the highest scores on Robust04 and ClueWeb09 for natural language queries, outperforming previous "encode-and-pool" and sparse-attention models.
- Leaderboard Success: On the highly competitive MS-MARCO leaderboard, MORES+ (combined with a dense retriever) achieved the 2nd best performance globally at the time of the paper's writing.
- Monotonic Improvement: Ablation studies showed that as more document chunks are fed into the model (up to 3,000 tokens), the ranking accuracy consistently improves, proving the model successfully aggregates information from the entire document rather than getting "confused" by extra text.
Limitations and Future Work
While MORES+ is more efficient than many competitors due to pre-computation, the authors acknowledge that they focused primarily on effectiveness (accuracy) in this paper. They leave a deeper exploration of raw inference speed and hardware optimization for future work. Additionally, while the model handles several thousand tokens well, it remains to be seen how it scales to extremely long documents (e.g., entire books) where even the interaction module might face memory constraints.
Summary updated