HLTCOE at TREC 2024 NeuCLIR Track

Eugene Yang, Dawn J. Lawrie, Orion Weller, James Mayfield

Introduction

Cross-Language Information Retrieval (CLIR) remains a pivotal challenge in information science, requiring systems to bridge the linguistic gap between queries and documents. The HLTCOE team at Johns Hopkins University addressed the TREC 2024 NeuCLIR track by investigating the intersection of efficient dense retrieval and high-parameter Large Language Model (LLM) reranking. Their approach focuses on overcoming the "domain and language gap" inherent in training data, which is predominantly English-centric, to improve search performance in Chinese, Persian, and Russian.

Methodology

Multi-Stage Retrieval Pipeline

The team utilized a sophisticated multi-stage pipeline centered on the PLAID engine and ColBERT-X. To adapt these models for cross-lingual tasks, two primary distillation strategies were employed:

  • Translate-Distill (TD): A teacher model scores English query-document pairs, and a student model is trained to replicate these scores using translated document counterparts.
  • Generate-Distill (GD): This method leverages LLMs to generate synthetic queries for documents within the target collection, creating domain-specific training data without human labeling.

LLM-Based Reranking

To maximize precision, the team implemented a two-tier reranking strategy. Initial candidates were processed by an mT5 Reranker, a 13-billion parameter model providing pointwise relevance predictions. For the final top-tier selection, they introduced a GPT-4o Heapsort algorithm. Rather than absolute scoring, GPT-4o acted as a comparator in a 4-ary heapsort, selecting the most relevant passage from small groups. This comparative approach leverages the reasoning capabilities of LLMs more effectively than traditional pointwise scoring.

Experimental Results

Cross-Language Information Retrieval (CLIR)

The integration of GPT-4o reranking yielded substantial performance gains across all target languages. In the Persian news task, the top-performing run achieved an nDCG@20 of 0.690, a significant improvement over the base PLAID retrieval score of 0.607.

Performance metrics for CLIR across Persian, Russian, and Chinese languages.

As shown in the evaluation metrics, the kitchen_rankfuse.mt5rerank.gpt4rerank run consistently outperformed baseline models, demonstrating the efficacy of combining hybrid fusion with LLM-based tournament judging.

Technical Document Retrieval

The 2024 track introduced a Technical Documents task, representing a domain shift from general news. GPT-4o proved particularly dominant in this specialized area, where traditional models trained on news data often struggle with technical nomenclature and specific document structures.

Performance metrics for the Technical Documents task.

In this domain, the plaid_distill_engzho.mt5rerank.gpt4rerank run achieved the highest nDCG@20 (0.455), confirming that LLM reranking provides a robust solution for domain adaptation in CLIR.

Key Insights and Discussion

  • Comparative vs. Absolute Scoring: The success of the heapsort algorithm suggests that LLMs are more reliable as "tournament judges" (comparing passages) than as absolute scorers.
  • Passage Optimization: Increasing passage size to 450 tokens without overlap provided a consistent performance boost, likely due to providing more context for the dense retrieval and reranking stages.
  • Token Pooling: While the team experimented with k-means clustering to reduce index size (token pooling), the current implementation resulted in slower indexing and inconsistent performance, indicating a need for further optimization.
  • Report Generation: In the pilot task for report generation, an "extractive" approach—grouping specific facts before synthesis—outperformed traditional abstractive summarization, highlighting the importance of preserving information "nuggets" in technical reporting.

Conclusion

The HLTCOE team's participation in TREC 2024 demonstrates that while specialized distillation techniques like GD and TD are essential for base retrieval, high-powered LLM reranking currently sets the state-of-the-art for precision in cross-language search. Future work will focus on refining Generate-Distill techniques to reduce reliance on expensive LLM reranking and optimizing storage-efficient indexing methods like token pooling.

Summary updated

Abstract

The HLTCOE team applied PLAID, an mT5 reranker, GPT-4 reranker, score fusion, and document translation to the TREC 2024 NeuCLIR track. For PLAID we included a variety of models and training techniques -- Translate Distill (TD), Generate Distill (GD) and multi-lingual translate-distill (MTD). TD uses scores from the mT5 model over English MS MARCO query-document pairs to learn how to score query-document pairs where the documents are translated to match the CLIR setting. GD follows TD but uses passages from the collection and queries generated by an LLM for training examples. MTD uses MS MARCO translated into multiple languages, allowing experiments on how to batch the data during training. Finally, for report generation we experimented with system combination over different runs. One family of systems used either GPT-4o or Claude-3.5-Sonnet to summarize the retrieved results from a series of decomposed sub-questions. Another system took the output from those two models and verified/combined them with Claude-3.5-Sonnet. The other family used GPT4o and GPT3.5Turbo to extract and group relevant facts from the retrieved documents based on the decomposed queries. The resulting submissions directly concatenate the grouped facts to form the report and their documents of origin as the citations. The team submitted runs to all NeuCLIR tasks: CLIR and MLIR news tasks as well as the technical documents task and the report generation task.
BibTeX
@inproceedings{yang2024hltcoe,
  author = {Eugene Yang and Dawn J. Lawrie and Orion Weller and James Mayfield},
  title = {{HLTCOE at TREC 2024 NeuCLIR Track}},
  booktitle = {TREC},
  year = {2024},
  url = {https://openreview.net/forum?id=KhxWLOll0i},
}
Figure