Abstract
The CIP team’s entry for the TREC 2022 Deep Learning Track demonstrates that passage retrieval performance can be significantly boosted by combining noise-resistant dense retrieval with advanced loss functions for re-ranking. By integrating query-noise alignment (RoDR), document expansion (docT5query), and Localized Contrastive Estimation (LCE), the team achieved state-of-the-art results on the MS MARCO v2 and TREC 2022 datasets.
Introduction
In Information Retrieval (IR), passage ranking is the critical task of identifying the most relevant text snippets for a given query. Modern systems typically employ a two-stage pipeline: a fast full-ranking (retrieval) stage to narrow down millions of documents, followed by a computationally intensive re-ranking stage using models like BERT.
However, two major hurdles persist. First, real-world user queries are often "noisy," containing typos or awkward phrasing that confuse standard Dense Retrieval (DR) models. Second, standard re-ranking models often rely on simple pointwise cross-entropy loss, which treats passages independently and fails to capture the subtle relative differences between highly relevant and partially relevant candidates. This paper details the CIP team's approach to overcoming these challenges through robust retrieval and advanced ranking objectives.
Methodology
Full Ranking: RoDR and Document Expansion
To address query noise, the authors utilized Robust Dense Retrieval (RoDR). This model is trained to align the embeddings of noisy queries with their clean counterparts using KL-divergence loss, ensuring semantic consistency despite textual errors. To further enhance retrieval, the team employed docT5query, where a T5 model generates potential questions for every passage. These questions are appended to the passages before indexing, facilitating better keyword matching via BM25 alongside neural dense retrieval.
Re-ranking: Advanced Loss Functions
For the re-ranking stage, a BERT-Large model was fine-tuned using specialized training objectives instead of standard classification loss:
- Localized Contrastive Estimation (LCE): This objective helps the model distinguish relevant passages from specific "hard negatives"—passages that appear relevant but are factually incorrect or less precise.
- Pairwise Hinge Loss: This focuses on the relative margin between positive and negative examples, which is often more effective for ranking tasks than absolute classification.
Experimental Setup
The models were trained and evaluated using the MS MARCO v1 and v2 datasets, with final testing conducted on the TREC 2022 Deep Learning Track test set, consisting of 76 difficult, human-judged queries. The team submitted several run configurations to test different combinations of retrievers and re-rankers.
Results and Discussion
The experimental results, summarized in the table below, highlight the efficacy of combining robust retrieval with contrastive re-ranking.
The top-performing configuration, cip_f2_r, which combined RoDR (with TAS-Balanced) and an LCE-based re-ranker, achieved an NDCG@10 of 0.5776 and a MAP of 0.1752 on the TREC 2022 test set. On the validation set (TREC 2021 queries), the cip_f3_r run achieved a high NDCG@10 of 0.7034.
Key insights from these results include:
- Noise Resistance is a Multiplier: Explicitly training the retriever to be invariant to textual noise makes the system significantly more reliable for real-world applications.
- Loss Functions Matter: The shift from pointwise cross-entropy to LCE and Hinge loss suggests that the training objective is as critical as the model architecture itself.
- Hybrid Synergy: The best results came from an ensemble approach—combining the strengths of dense vector search with traditional BM25 search on expanded documents.
Conclusion
The CIP team's performance at TREC 2022 underscores the importance of robustness and sophisticated ranking objectives in modern IR pipelines. While the results are strong, future work will investigate better alignment between the retrieval and re-ranking phases to ensure the retriever passes along candidates that the re-ranker is best equipped to handle. Additionally, further integration of sparse and dense retrieval methods remains a priority for maximizing both efficiency and accuracy.
Summary updated