Abstract
The TREC 2024 NeuCLIR track marks the conclusion of a three-year cycle dedicated to advancing Cross-Language Information Retrieval (CLIR) through neural networks and Large Language Models (LLMs). This overview highlights the transition from traditional statistical methods to LLM-based reranking and the introduction of a novel Report Generation task. Evaluation across diverse datasets—including news in Chinese, Persian, and Russian, and Chinese technical documents—reveals that neural models are increasingly capable of "multilingual thinking," often bypassing the need for explicit translation to achieve state-of-the-art results.
Introduction
Cross-Language Information Retrieval (CLIR) involves searching in one language (e.g., English) to find relevant documents in another (e.g., Chinese). Historically, this relied on "translate-then-search" pipelines, which are prone to errors and representation misalignment. The TREC 2024 NeuCLIR track addresses these hurdles by leveraging dense retrieval and generative reranking, while also tackling the "last mile" problem: synthesizing retrieved foreign-language content into actionable English reports.
Methods and Datasets
The track evaluated a spectrum of approaches, from traditional benchmarks like BM25 and Probabilistic Structured Queries (PSQ) to cutting-edge neural models like ColBERT and PLAID. The most successful systems utilized a "retrieve-then-rerank" architecture, where a fast neural model identifies the top 100 documents, followed by a massive LLM (such as GPT-4 or Claude) that re-orders them based on deep semantic relevance.
The evaluation utilized several large-scale collections:
- NeuCLIR 1 News: Approximately 10 million news stories in Persian, Chinese, and Russian.
- Technical Documents: 396,209 Chinese academic abstracts, providing a high-difficulty test for specialized vocabulary.
- NeuMARCO: A multilingual version of the MS MARCO dataset used for training neural models.
Key Results and Insights
News Retrieval Performance
The integration of LLMs as rerankers has set a new performance standard. Top-performing systems achieved nDCG@20 scores between 0.6 and 0.7 across all three news languages. A significant insight from this year is that neural models are becoming proficient enough at cross-lingual alignment that the translation step can often be skipped; for instance, the most effective Russian retrieval system operated directly on original documents.
Technical Document Retrieval
The technical document task proved more challenging due to specialized scientific vocabulary. There is a massive performance gap (roughly 0.3 nDCG@20) between neural and statistical methods in this domain, suggesting that traditional keyword matching is no longer competitive against models that understand scientific context.
Report Generation and the ARGUE Framework
The pilot Report Generation task shifted the focus from finding documents to synthesizing them. Using the ARGUE framework—which employs "nuggets" (atomic facts) to check for comprehensiveness and accuracy—top systems scored near 0.9. However, "citation precision" remained low (around 0.3), indicating that while systems are fluent and factual, they often struggle to attribute information to the correct source document.
Conclusion and Future Directions
As the NeuCLIR track concludes, it leaves a legacy of high-performance neural retrieval standards. The track will be succeeded by TREC 2025 RAGTIME, which will focus on balanced multilingual collections, automated evaluation of report quality, and the synthesis of single reports from a simultaneous mix of Arabic, Chinese, English, and Russian sources.
Summary updated