Overview of the TREC 2024 NeuCLIR Track

Dawn J. Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldaini, Eugene Yang

Introduction

The TREC 2024 NeuCLIR track marks the culmination of a three-year effort to advance neural cross-language information retrieval (CLIR). As global information needs transcend linguistic boundaries, the ability to retrieve and synthesize information from diverse languages becomes paramount. This year's track emphasized the integration of Large Language Models (LLMs) for reranking and introduced a pioneering pilot task in cross-lingual report generation.

Evaluation Framework

The 2024 track comprised four primary challenges:

  1. News CLIR: Retrieval of Chinese, Persian, or Russian news articles using English queries.
  2. News MLIR: A multilingual task requiring a unified ranked list from all three news collections.
  3. Technical Documents CLIR: Retrieval of Chinese academic abstracts (CSL dataset) based on English technical queries.
  4. Report Generation (Pilot): Synthesizing English reports from foreign-language sources, evaluated on fact recall and citation accuracy.

A defining trend of the 2024 track was the ubiquitous use of generative LLMs as final-stage rerankers. These models significantly boosted retrieval effectiveness across all languages.

nDCG@20 performance across (a) Chinese, (b) Persian, and (c) Russian News CLIR tasks, categorized by document language strategy.

As shown in the performance distributions across the three news collections, top systems achieved robust nDCG@20 scores, often exceeding 0.6 for Chinese and Persian. Notably, the Russian News CLIR task demonstrated the viability of "translation-free" pipelines. The most effective Russian system utilized original documents without machine translation during indexing, proving that modern multilingual embeddings can bridge language gaps natively.

Technical Document Retrieval

The Technical Documents task presented a unique challenge due to the specialized nature of academic abstracts. Evaluation revealed a stark contrast between neural approaches and traditional methods.

nDCG@20 performance for Technical Documents CLIR, showing a significant performance gap between neural rerankers and traditional BM25 baselines.

As illustrated in the Technical Documents results, there is a performance gap of approximately 0.3 nDCG@20 between top-tier neural systems (such as those utilizing plaid_distill or mlir-ATNESLH) and traditional statistical baselines like BM25. This underscores the necessity of sophisticated neural modeling for domain-specific cross-lingual retrieval.

Cross-Lingual Report Generation

The Report Generation pilot addressed the "last mile" problem: making retrieved foreign-language content accessible. While systems generated fluent English reports (scoring ~0.9 on structural metrics), accuracy remained a significant hurdle. Citation precision hovered around 30%, and fact (nugget) recall was below 50%. These results indicate that while LLMs are excellent at synthesis, grounding them in cross-lingual evidence requires further research in Retrieval-Augmented Generation (RAG).

Conclusion

The NeuCLIR track has successfully demonstrated the power of neural methods and LLMs in overcoming language barriers in information retrieval. As the track concludes, its legacy continues in the upcoming RAGTIME track for 2025, which will focus on RAG from mixed-language sources, including Arabic, Chinese, English, and Russian.

Summary updated

BibTeX
@inproceedings{lawrie2024overview,
  author = {Dawn J. Lawrie and Sean MacAvaney and James Mayfield and Paul McNamee and Douglas W. Oard and Luca Soldaini and Eugene Yang},
  title = {{Overview of the TREC 2024 NeuCLIR Track}},
  booktitle = {TREC},
  year = {2024},
  url = {https://openreview.net/forum?id=9vq91ag4Wv},
}
Figure