Contrastive Fine-tuning Improves Robustness for Neural Rankers

Xiaofei Ma, Cicero Nogueira dos Santos, Andrew O. ArnoldACL 2021

TL;DR

This paper introduces a novel contrastive fine-tuning method for neural rankers that significantly improves their robustness to out-of-domain data and query perturbations. By combining a standard ranking loss with a contrastive loss that encourages similar query-document pairs to be close in the embedding space, the model learns more generalizable representations, leading to better performance on unseen domains and noisy inputs.

What Problem Does This Paper Address?

State-of-the-art neural rankers, while powerful, often struggle when deployed in real-world scenarios. They can exhibit poor performance on data from new domains (out-of-domain) or when queries are slightly altered (perturbed). This fragility limits their practical applicability in information retrieval systems where query variations and domain shifts are common. Existing methods like domain adaptation require labeled target data, which is often unavailable, and data augmentation can be sensitive to the quality of generated perturbations. This paper aims to address this robustness gap by proposing a more general and effective fine-tuning strategy.

Methods

The core of this work is a new fine-tuning approach that augments the standard ranking loss with a contrastive loss.

  • Standard Ranking Loss: The paper utilizes common ranking losses like the standard hinge loss (SHL) or a modified hinge loss (MHL). These losses aim to ensure that a positive query-document pair receives a higher score than a negative one for the same query.
  • Contrastive Loss: The novel contribution is the addition of a contrastive loss, specifically the triplet margin loss (TML). Instead of just comparing positive and negative pairs for the same query, this contrastive loss compares representations of pairs from different queries. The goal is to pull together representations of positive query-document pairs (even if from different queries) and push apart representations of positive pairs from negative pairs. This encourages the model to learn underlying semantic similarities across a broader range of examples.
  • Combined Loss: The final loss function is a weighted average of the ranking loss and the contrastive loss. The authors found that equal weighting often works well.

The method is applied to fine-tune pre-trained language models like BERT and BART, which are common architectures for neural rankers.

Datasets

The paper uses four publicly available passage ranking datasets for training and evaluating in-domain performance:

  • WikiQA: Question-sentence pairs from Bing query logs, with answers from Wikipedia summaries.
  • WikiPassageQA: A benchmark for non-factoid answer passage retrieval, using Wikipedia documents.
  • InsuranceQA: Question-answer pairs collected from the insurance domain.
  • YahooQA: Questions and answers from Yahoo! Answers.

To specifically evaluate robustness, the authors created new test datasets based on the WikiQA test set. These datasets include:

  • Query Perturbations: Versions of WikiQA queries with added/removed punctuation, typos, and changed contractions.
  • Query Reformulations: Versions of WikiQA queries that are paraphrased, presented as headlines, or have a changed voice. These were semi-automatically generated using a fine-tuned T5 model.

Key Insights and Ideas

  • Contrastive learning for ranking is more than just hard negative mining: While hard negative mining focuses on finding difficult negative examples for a single query, this approach leverages similarities across different queries to build more robust representations.
  • Learning semantic similarity across queries improves generalization: By forcing the model to recognize that positive pairs from different queries are semantically related, it learns more general features that are less tied to specific query phrasing or domain nuances.
  • Contrastive fine-tuning is more efficient than data augmentation: The proposed method achieves robustness improvements with significantly less training time compared to traditional data augmentation techniques, even when using augmented data within the contrastive framework.
  • Visualizing representations shows better separation: t-SNE plots demonstrate that the contrastive loss leads to a clearer separation between positive and negative query-passage representations in the embedding space.

Key Results

  • Improved Zero-Shot Transfer: The contrastive fine-tuning method significantly boosts zero-shot transfer performance across domains, with improvements of up to 9 absolute points in Mean Average Precision (MAP) on some target datasets.
  • Robustness to Perturbations: The method consistently improves robustness against various query perturbations (typos, punctuation changes) and reformulations (paraphrases, headlines, voice changes), outperforming models trained with ranking loss alone.
  • Competitive In-Domain Performance: While the primary focus is robustness, the proposed method achieves state-of-the-art or competitive in-domain performance on the tested datasets.
  • Data Augmentation Comparison: Contrastive fine-tuning with paraphrased queries was more effective than traditional data augmentation for improving robustness against various reformulation types.

Limitations and Future Work

  • Hyperparameter Tuning: The paper notes that the performance of hard negative mining methods could be further improved with proper hyperparameter tuning.
  • Future Work: The authors plan to explore other state-of-the-art contrastive loss functions and novel methods for combining multiple loss terms. They also suggest evaluating the approach on a wider range of tasks and datasets.

Summary updated

BibTeX
@inproceedings{ma2021contrastive,
  author = {Xiaofei Ma and Cicero Nogueira dos Santos and Andrew O. Arnold},
  title = {{Contrastive Fine-tuning Improves Robustness for Neural Rankers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure