Authors: Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew Peters, Arie Cattan, Ido Dagan
Venue: Findings of the Association for Computational Linguistics: EMNLP 2021
Introduction
In the field of Natural Language Processing (NLP), most state-of-the-art models are trained as "monogamous" readers, processing a single document at a time. While models like BERT and RoBERTa excel at understanding internal text structures, they often struggle when information is fragmented across multiple sources. Whether tracking an event across news outlets or connecting legal evidence, "cross-document" reasoning remains a significant challenge.
Researchers from Bar-Ilan University and the Allen Institute for AI (AI2) address this gap with CDLM (Cross-Document Language Modeling), a pretraining strategy designed to turn language models into "polyglots" capable of synthesizing information clusters.
Methodology
CDLM is built upon the Longformer architecture, which utilizes a sparse attention mechanism to handle sequences up to 4,096 tokens. The researchers introduced two primary innovations to enable cross-document reasoning:
- Pretraining on Related Document Clusters: Instead of random text chunks, CDLM is pretrained on clusters of documents describing the same topic (sourced from the Multi-News dataset). This forces the model to identify and align overlapping information.
- Dynamic Global Attention: While standard Longformers use global attention sparingly (e.g., for the
[CLS]token), CDLM applies global attention to every masked token during pretraining.
This mechanism allows a masked token in one document to directly attend to relevant context in another document within the same cluster. For instance, if the word "alleged" is masked in Document A, the model can "peek" at related terms like "sued" in Document B to resolve the ambiguity.
Evaluation and Benchmarks
The model was evaluated across several diverse tasks:
- Cross-Document Coreference: Using the ECB+ dataset to resolve events and entities across texts.
- Document Matching: Benchmarks including AAN, OC, and S2ORC for citation recommendation, and PAN for plagiarism detection.
- Multi-hop QA: The HotpotQA benchmark.
Key Results
CDLM achieved significant performance gains, setting new state-of-the-art (SOTA) marks. On the ECB+ dataset, CDLM outperformed previous specialized models by substantial margins, particularly in event coreference.
Beyond coreference, CDLM surpassed task-specific architectures in citation recommendation and plagiarism detection. Ablation studies confirmed that the synergy between related-document training and dynamic global attention is crucial; neither component alone provides the same performance boost.
Conclusion
CDLM demonstrates that specialized pretraining objectives can be more effective than simply increasing model parameters. By explicitly teaching models to "read between documents," CDLM provides a versatile foundation for multi-document NLP tasks. Future work may involve scaling the model to "large" architectures and extending the framework to sequence-to-sequence models for tasks like multi-document summarization.
Summary updated