Introduction
The TREC 2024 NeuCLIR track marks the culmination of a three-year effort to advance neural cross-language information retrieval (CLIR). As global information needs transcend linguistic boundaries, the ability to retrieve and synthesize information from diverse languages becomes paramount. This year's track emphasized the integration of Large Language Models (LLMs) for reranking and introduced a pioneering pilot task in cross-lingual report generation.
Evaluation Framework
The 2024 track comprised four primary challenges:
- News CLIR: Retrieval of Chinese, Persian, or Russian news articles using English queries.
- News MLIR: A multilingual task requiring a unified ranked list from all three news collections.
- Technical Documents CLIR: Retrieval of Chinese academic abstracts (CSL dataset) based on English technical queries.
- Report Generation (Pilot): Synthesizing English reports from foreign-language sources, evaluated on fact recall and citation accuracy.
Performance Trends in News Retrieval
A defining trend of the 2024 track was the ubiquitous use of generative LLMs as final-stage rerankers. These models significantly boosted retrieval effectiveness across all languages.
As shown in the performance distributions across the three news collections, top systems achieved robust nDCG@20 scores, often exceeding 0.6 for Chinese and Persian. Notably, the Russian News CLIR task demonstrated the viability of "translation-free" pipelines. The most effective Russian system utilized original documents without machine translation during indexing, proving that modern multilingual embeddings can bridge language gaps natively.
Technical Document Retrieval
The Technical Documents task presented a unique challenge due to the specialized nature of academic abstracts. Evaluation revealed a stark contrast between neural approaches and traditional methods.
As illustrated in the Technical Documents results, there is a performance gap of approximately 0.3 nDCG@20 between top-tier neural systems (such as those utilizing plaid_distill or mlir-ATNESLH) and traditional statistical baselines like BM25. This underscores the necessity of sophisticated neural modeling for domain-specific cross-lingual retrieval.
Cross-Lingual Report Generation
The Report Generation pilot addressed the "last mile" problem: making retrieved foreign-language content accessible. While systems generated fluent English reports (scoring ~0.9 on structural metrics), accuracy remained a significant hurdle. Citation precision hovered around 30%, and fact (nugget) recall was below 50%. These results indicate that while LLMs are excellent at synthesis, grounding them in cross-lingual evidence requires further research in Retrieval-Augmented Generation (RAG).
Conclusion
The NeuCLIR track has successfully demonstrated the power of neural methods and LLMs in overcoming language barriers in information retrieval. As the track concludes, its legacy continues in the upcoming RAGTIME track for 2025, which will focus on RAG from mixed-language sources, including Arabic, Chinese, English, and Russian.
Summary updated