Overview of the TREC 2024 NeuCLIR Track

Dawn J. Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldaini, Eugene Yang

Abstract

The TREC 2024 NeuCLIR track marks the conclusion of a three-year cycle dedicated to advancing Cross-Language Information Retrieval (CLIR) through neural networks and Large Language Models (LLMs). This overview highlights the transition from traditional statistical methods to LLM-based reranking and the introduction of a novel Report Generation task. Evaluation across diverse datasets—including news in Chinese, Persian, and Russian, and Chinese technical documents—reveals that neural models are increasingly capable of "multilingual thinking," often bypassing the need for explicit translation to achieve state-of-the-art results.

Introduction

Cross-Language Information Retrieval (CLIR) involves searching in one language (e.g., English) to find relevant documents in another (e.g., Chinese). Historically, this relied on "translate-then-search" pipelines, which are prone to errors and representation misalignment. The TREC 2024 NeuCLIR track addresses these hurdles by leveraging dense retrieval and generative reranking, while also tackling the "last mile" problem: synthesizing retrieved foreign-language content into actionable English reports.

Methods and Datasets

The track evaluated a spectrum of approaches, from traditional benchmarks like BM25 and Probabilistic Structured Queries (PSQ) to cutting-edge neural models like ColBERT and PLAID. The most successful systems utilized a "retrieve-then-rerank" architecture, where a fast neural model identifies the top 100 documents, followed by a massive LLM (such as GPT-4 or Claude) that re-orders them based on deep semantic relevance.

The evaluation utilized several large-scale collections:

  • NeuCLIR 1 News: Approximately 10 million news stories in Persian, Chinese, and Russian.
  • Technical Documents: 396,209 Chinese academic abstracts, providing a high-difficulty test for specialized vocabulary.
  • NeuMARCO: A multilingual version of the MS MARCO dataset used for training neural models.

Key Results and Insights

News Retrieval Performance

The integration of LLMs as rerankers has set a new performance standard. Top-performing systems achieved nDCG@20 scores between 0.6 and 0.7 across all three news languages. A significant insight from this year is that neural models are becoming proficient enough at cross-lingual alignment that the translation step can often be skipped; for instance, the most effective Russian retrieval system operated directly on original documents.

Figure 2: nDCG@20 performance for Chinese, Persian, and Russian News CLIR tasks, categorized by document language (Original, Translated, or Both).

Technical Document Retrieval

The technical document task proved more challenging due to specialized scientific vocabulary. There is a massive performance gap (roughly 0.3 nDCG@20) between neural and statistical methods in this domain, suggesting that traditional keyword matching is no longer competitive against models that understand scientific context.

Figure 5: System performance on the Chinese Technical Documents CLIR task, illustrating the effectiveness of neural models over statistical baselines (e.g., fast_psqtd).

Report Generation and the ARGUE Framework

The pilot Report Generation task shifted the focus from finding documents to synthesizing them. Using the ARGUE framework—which employs "nuggets" (atomic facts) to check for comprehensiveness and accuracy—top systems scored near 0.9. However, "citation precision" remained low (around 0.3), indicating that while systems are fluent and factual, they often struggle to attribute information to the correct source document.

Conclusion and Future Directions

As the NeuCLIR track concludes, it leaves a legacy of high-performance neural retrieval standards. The track will be succeeded by TREC 2025 RAGTIME, which will focus on balanced multilingual collections, automated evaluation of report quality, and the synthesis of single reports from a simultaneous mix of Arabic, Chinese, English, and Russian sources.

Summary updated

Abstract

The principal goal of the TREC Neural Cross-Language Information Retrieval (NeuCLIR) track is to study the effect of neural approaches on cross-language information access. The track has created test collections containing Chinese, Persian, and Russian news stories and Chinese academic abstracts. NeuCLIR includes four task types: Cross-Language Information Retrieval (CLIR) from news, Multilingual Information Retrieval (MLIR) from news, Report Generation from news, and CLIR from technical documents. A total of 274 runs were submitted by five participating teams (and as baselines by the track coordinators) for eight tasks across these four task types. Task descriptions and the available results are presented.
BibTeX
@inproceedings{lawrie2024overview,
  author = {Dawn J. Lawrie and Sean MacAvaney and James Mayfield and Paul McNamee and Douglas W. Oard and Luca Soldaini and Eugene Yang},
  title = {{Overview of the TREC 2024 NeuCLIR Track}},
  booktitle = {TREC},
  year = {2024},
  url = {https://openreview.net/forum?id=9vq91ag4Wv},
}
Figure