Abstract
The HLTCOE team's participation in the TREC 2023 NeuCLIR track focused on bridging the gap between high-accuracy cross-encoders and efficient bi-encoders for Cross-Language Information Retrieval (CLIR). By leveraging a "Translate-Distill" approach, the team trained ColBERT-X models to emulate a massive 13-billion parameter mT5 reranker. Their results demonstrate that knowledge distillation allows leaner retrieval engines to achieve state-of-the-art performance across Persian, Russian, and Chinese news and technical documents while maintaining the speed required for production environments.
Introduction
Cross-Language Information Retrieval (CLIR) presents a significant challenge: how to enable users to search in one language (e.g., English) and find relevant documents in another (e.g., Chinese) without sacrificing either accuracy or efficiency. While large-scale cross-encoders like mT5 provide exceptional ranking accuracy by processing query-document pairs simultaneously, their computational cost makes them impractical for first-stage retrieval over millions of documents. Conversely, bi-encoders like ColBERT offer the necessary speed but often struggle to match the deep linguistic understanding of their larger counterparts. The HLTCOE team addressed this trade-off for the TREC 2023 NeuCLIR track by exploring knowledge distillation as a means to transfer expertise from massive multilingual models to efficient retrieval engines.
Methodology
The team's strategy integrated neural and lexical retrieval techniques:
- Translate-Distill (TD): This core innovation involved using a "Teacher" model—the 13B-parameter mT5-XXL reranker—to score query-document pairs. A smaller "Student" model, ColBERT-X, was then trained to replicate these scores, effectively internalizing the teacher's ranking logic within a more efficient architecture.
- PLAID & ColBERT-X: To further optimize efficiency, the team utilized the PLAID engine, which employs aggressive compression (down to 1 bit per dimension for residual vectors) to minimize the storage and computational footprint of the search index.
- Multilingual Translate-Train (MTT): Rather than developing language-specific models, the team trained a "universal" model using mixed-language batches. This approach allows a single model to handle Persian, Russian, and Chinese retrieval tasks simultaneously.
- Temporal Sharding: For event-specific topics, the team experimented with filtering the search index by document creation dates to reduce noise, though this yielded mixed results.
- Baselines: Traditional lexical methods, including Probabilistic Structured Queries (PSQ) and BM25, served as benchmarks for the neural approaches.
Experimental Setup
Models were primarily trained on the MS-MARCO v1 collection, with passages automatically translated into the target languages. Evaluation was conducted on the TREC 2023 NeuCLIR track, which features two main datasets: a News Collection (Persian, Russian, Chinese) and a Technical Document collection (Chinese technical abstracts).
Key Results
Cross-Language Retrieval (CLIR)
The Translate-Distill models emerged as the top performers in the single-language news tasks. As shown in the results for the News Collection, the distilled models (c1, c2) significantly outperformed standard Translate-Train baselines. For instance, in the Persian task, the distilled model achieved an nDCG of 0.547, compared to 0.495 for the standard ColBERT-X model.
Multilingual Retrieval (MLIR)
The multilingual MTT approach proved highly effective for searching across the entire collection simultaneously. The universal model achieved an nDCG@20 of 0.362, demonstrating that a single model can maintain high effectiveness across multiple languages.
Discussion and Insights
- Distillation as an Equalizer: The performance of the distilled ColBERT models suggests that massive parameter counts are not strictly necessary at inference time if the model is trained to mimic a high-capacity teacher.
- Domain Sensitivity: While neural models excelled in the news domain, the gap between neural and lexical models narrowed significantly on technical documents. This highlights the challenge of domain shift and the need for specialized distillation in technical fields.
- Temporal Challenges: Date-filtering, while theoretically sound for news, led to performance degradation. The authors hypothesize that temporal vocabulary shifts—changes in word usage over time—may negatively interact with the compressed vector representations used in PLAID.
Conclusion
The HLTCOE team's results at TREC 2023 NeuCLIR underscore the power of knowledge distillation for cross-language search. By successfully transferring the capabilities of mT5-XXL into the efficient ColBERT-X framework, they established a new benchmark for high-performance, scalable CLIR. Future work will focus on refining distillation configurations and addressing the unique challenges of domain-specific retrieval.
Summary updated