Policy-Gradient Training of Language Models for Ranking

Ge Gao, Jonathan D. Chang, Claire Cardie, Kianté Brantley, Thorsten Joachims

TL;DR

Neural PG-RANK is a new training algorithm that allows Large Language Models (LLMs) to optimize ranking metrics (like nDCG) directly using policy gradients. By treating the retriever as a stochastic ranking policy, it eliminates the need for complex heuristics like "hard negative mining" or knowledge distillation, leading to state-of-the-art performance in second-stage reranking.

What Problem Does This Paper Address?

In modern AI, retrieval is the backbone of "grounding"—ensuring models like ChatGPT use factual sources rather than hallucinating. However, training these retrievers is notoriously finicky.

Standard methods use contrastive loss, which essentially teaches a model to push "correct" documents closer to a query and "incorrect" ones further away. This approach has two major flaws:

  1. Heuristic Overload: To make it work, researchers have to use complex tricks like mining "hard negatives" (irrelevant documents that look relevant) or distilling knowledge from even larger "teacher" models.
  2. Metric Mismatch: Contrastive loss is a proxy. It doesn't directly optimize for what we actually care about: the final ranking quality (e.g., is the best document at the very top?).

Ranking is a "combinatorial" problem—there are a staggering number of ways to order a list of documents. Directly optimizing this list has historically been mathematically difficult because ranking is not naturally differentiable.

Methods

The authors introduce Neural PG-RANK, which bridges the gap between reinforcement learning and information retrieval.

  • Plackett-Luce Ranking Policy: Instead of just outputting a score, the model is treated as a policy that defines a probability distribution over all possible rankings. It uses the Plackett-Luce model, which breaks down the probability of a full ranking into a sequence of document selections.
  • Policy Gradient (REINFORCE): The model "samples" different rankings, sees how well they perform (their "utility," such as nDCG), and then uses policy gradients to update its weights. If a sampled ranking has high utility, the model is encouraged to produce similar rankings in the future.
  • Gumbel-Softmax Trick: To avoid the $O(n^2)$ computational cost of sampling rankings one by one, they use the Gumbel-Softmax trick to sample an entire ranking in $O(n \log n)$ time.
  • Variance Reduction: Policy gradients are notoriously noisy. The authors implement a "leave-one-out" baseline, which compares the utility of a specific sampled ranking against the average utility of other samples to stabilize training.
Figure 1: Illustration of our Neural PG-RANK. Given a query and a collection of documents, a Placket-Luce ranking policy samples ranking, receives utility, and gets updated using policy gradient and the received utility. Our method can directly optimize any ranking metric of interest as utility, and allows end-to-end training of any differential policy. Query and document examples are from MS MARCO dataset (Campos et al., 2017).

Datasets

The researchers trained their models on MS MARCO, the industry-standard large-scale dataset for web search retrieval.

For evaluation, they tested on:

  • MS MARCO dev set: For in-domain performance.
  • BEIR Benchmark: A collection of 15 diverse datasets (including NaturalQuestions, HotpotQA, and TREC-COVID) to test how well the model generalizes to new domains like science, finance, and Wikipedia-based QA.

Key Insights and Ideas

  • Direct Optimization Wins: The most significant "aha" moment is that when the training objective (optimizing nDCG) aligns perfectly with the evaluation setup, performance skyrockets.
  • Ditching the Crutches: As shown in the figure below, Neural PG-RANK achieves top-tier results without the "additional supervision" (distillation) or complex "negative mining" required by models like SBERT or TAS-B.
Table 1: Reliance of state-of-the-art comparison systems and our Neural PG-RANK on negative document mining and additional supervision. Each check denotes a heuristics used during training. Our method minimizes the reliance on the type of negative documents, and does not require any additional supervision from other models to improve retrieval performance.
  • QA Generalization: The model shows a surprising "transfer" ability. Even though it was trained on general web search, it became exceptionally good at retrieving documents for complex question-answering tasks (NQ and HotpotQA).

Key Results

The results for second-stage reranking (taking a top-1000 list and re-ordering it) were particularly impressive:

  • In-Domain Mastery: On MS MARCO, Neural PG-RANK achieved an nDCG@10 of 0.987, nearly a perfect score, significantly outperforming the SBERT baseline of 0.892.
  • Out-of-Domain Success: On the NaturalQuestions (NQ) dataset, it reached 0.878 nDCG@10, beating all comparison systems. On HotpotQA, it hit 0.902.
  • Top-Heavy Gains: The improvements were even more dramatic at the very top of the list (nDCG@1), showing the model is excellent at picking the single best answer.
Table 3: Second-stage reranking: nDCG@10 results on out-of-domain datasets. * marks evaluations run by us using the publicly available checkpoint. Bold font represents the highest number per row, and underline shows the second highest. Light green color highlights the experiments where our Neural PG-RANK yields performance gain.

Limitations and Future Work

  • First-Stage Retrieval: The model currently struggles when asked to search the entire web (millions of documents) from scratch. The authors believe this is because training on small candidate sets creates "blind spots." They suggest "cutting-plane methods" as a future fix.
  • Domain Specificity: While it excelled at QA, it showed weaker generalization on specialized science and finance datasets.
  • Computational Cost: Sampling multiple rankings during training is more computationally expensive than standard contrastive training, though the authors argue the performance gains justify the cost.

Summary updated

Abstract

Text retrieval plays a crucial role in incorporating factual knowledge for decision making into language processing pipelines, ranging from chat-based web search to question answering systems. Current state-of-the-art text retrieval models leverage pre-trained large language models (LLMs) to achieve competitive performance, but training LLM-based retrievers via typical contrastive losses requires intricate heuristics, including selecting hard negatives and using additional supervision as learning signals. This reliance on heuristics stems from the fact that the contrastive loss itself is heuristic and does not directly optimize the downstream metrics of decision quality at the end of the processing pipeline. To address this issue, we introduce Neural PG-RANK, a novel training algorithm that learns to rank by instantiating a LLM as a Plackett-Luce ranking policy. Neural PG-RANK provides a principled method for end-to-end training of retrieval models as part of larger decision systems via policy gradient, with little reliance on complex heuristics, and it effectively unifies the training objective with downstream decision-making quality. We conduct extensive experiments on various text retrieval benchmarks. The results demonstrate that when the training objective aligns with the evaluation setup, Neural PG-RANK yields remarkable in-domain performance improvement, with substantial out-of-domain generalization to some critical datasets employed in downstream question answering tasks.
BibTeX
@misc{gao2023policy,
  author = {Ge Gao and Jonathan D. Chang and Claire Cardie and Kianté Brantley and Thorsten Joachims},
  title = {{Policy-Gradient Training of Language Models for Ranking}},
  year = {2023},
  eprint = {2310.04407},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.04407},
}
Figure