Introduction
Cross-Language Information Retrieval (CLIR) remains a pivotal challenge in information science, requiring systems to bridge the linguistic gap between queries and documents. The HLTCOE team at Johns Hopkins University addressed the TREC 2024 NeuCLIR track by investigating the intersection of efficient dense retrieval and high-parameter Large Language Model (LLM) reranking. Their approach focuses on overcoming the "domain and language gap" inherent in training data, which is predominantly English-centric, to improve search performance in Chinese, Persian, and Russian.
Methodology
Multi-Stage Retrieval Pipeline
The team utilized a sophisticated multi-stage pipeline centered on the PLAID engine and ColBERT-X. To adapt these models for cross-lingual tasks, two primary distillation strategies were employed:
- Translate-Distill (TD): A teacher model scores English query-document pairs, and a student model is trained to replicate these scores using translated document counterparts.
- Generate-Distill (GD): This method leverages LLMs to generate synthetic queries for documents within the target collection, creating domain-specific training data without human labeling.
LLM-Based Reranking
To maximize precision, the team implemented a two-tier reranking strategy. Initial candidates were processed by an mT5 Reranker, a 13-billion parameter model providing pointwise relevance predictions. For the final top-tier selection, they introduced a GPT-4o Heapsort algorithm. Rather than absolute scoring, GPT-4o acted as a comparator in a 4-ary heapsort, selecting the most relevant passage from small groups. This comparative approach leverages the reasoning capabilities of LLMs more effectively than traditional pointwise scoring.
Experimental Results
Cross-Language Information Retrieval (CLIR)
The integration of GPT-4o reranking yielded substantial performance gains across all target languages. In the Persian news task, the top-performing run achieved an nDCG@20 of 0.690, a significant improvement over the base PLAID retrieval score of 0.607.
As shown in the evaluation metrics, the kitchen_rankfuse.mt5rerank.gpt4rerank run consistently outperformed baseline models, demonstrating the efficacy of combining hybrid fusion with LLM-based tournament judging.
Technical Document Retrieval
The 2024 track introduced a Technical Documents task, representing a domain shift from general news. GPT-4o proved particularly dominant in this specialized area, where traditional models trained on news data often struggle with technical nomenclature and specific document structures.
In this domain, the plaid_distill_engzho.mt5rerank.gpt4rerank run achieved the highest nDCG@20 (0.455), confirming that LLM reranking provides a robust solution for domain adaptation in CLIR.
Key Insights and Discussion
- Comparative vs. Absolute Scoring: The success of the heapsort algorithm suggests that LLMs are more reliable as "tournament judges" (comparing passages) than as absolute scorers.
- Passage Optimization: Increasing passage size to 450 tokens without overlap provided a consistent performance boost, likely due to providing more context for the dense retrieval and reranking stages.
- Token Pooling: While the team experimented with k-means clustering to reduce index size (token pooling), the current implementation resulted in slower indexing and inconsistent performance, indicating a need for further optimization.
- Report Generation: In the pilot task for report generation, an "extractive" approach—grouping specific facts before synthesis—outperformed traditional abstractive summarization, highlighting the importance of preserving information "nuggets" in technical reporting.
Conclusion
The HLTCOE team's participation in TREC 2024 demonstrates that while specialized distillation techniques like GD and TD are essential for base retrieval, high-powered LLM reranking currently sets the state-of-the-art for precision in cross-language search. Future work will focus on refining Generate-Distill techniques to reduce reliance on expensive LLM reranking and optimizing storage-efficient indexing methods like token pooling.
Summary updated