A Study of Latent Structured Prediction Approaches to Passage Reranking

Iryna Haponchyk, Alessandro MoschittiNAACL 2019 · DOI

Abstract

Passage reranking is a critical component of Question Answering (QA) systems. Traditional methods often rely on pointwise or pairwise signals, ignoring the global structure of the ranked list. This paper proposes a structured prediction framework that treats rankings as latent variables and optimizes for Mean Average Precision (MAP) directly. By introducing a novel approximate inference method, the authors demonstrate significant performance improvements—exceeding 7 absolute points in MAP—on standard benchmarks like WikiQA.

Introduction

In the context of Question Answering, passage reranking aims to order candidate answers such that relevant passages appear at the top. Current state-of-the-art systems typically employ "pointwise" binary classification or "pairwise" comparison models. However, these approaches fail to capture the global context of the entire ranked list.

Transitioning to a "list-wise" or structured approach presents two primary challenges:

  1. Metric Complexity: Non-factorizable metrics like Mean Average Precision (MAP) cannot be easily decomposed into independent parts for direct optimization.
  2. Multiplicity of Ground Truths: Multiple valid rankings exist (any permutation where all relevant passages precede irrelevant ones), making it difficult to define a single "gold standard" for training.

Methodology: Latent Structured Prediction

To address these challenges, the authors adapt Latent Structured Perceptron (LSP) and Latent Structural SVM (LSSVM) algorithms for the reranking task.

Latent Variables for Rankings

Instead of a fixed gold ranking, the model treats the ranking as a latent variable. During training, the model "imputes" the most consistent ranking with ground truth labels, effectively picking the "easiest" perfect ranking to learn from.

MAP Optimization and Greedy Inference

The authors derive a strict decomposition of the Average Precision (AP) loss to optimize MAP directly. To handle the computational complexity of finding the "max-violating" ranking, they introduce a greedy, bottom-up inference procedure. This method reduces complexity from $O(N^2)$ to $O(N \log N)$ while maintaining high performance.

Experimental Results

The models were evaluated on the WikiQA and TREC13 datasets using both textual similarity features and CNN-generated embeddings.

Performance comparison on WikiQA dataset (DEV and TEST sets) showing significant gains for structured approaches over baselines.

As shown in the results table, structured approaches like LSP-AP significantly outperform the standard SVM baseline. On the WikiQA test set, LSP-AP achieved a MAP of 64.50, compared to 54.67 for the SVM baseline—a gain of nearly 10 absolute points, well exceeding the 7-point threshold mentioned in the study.

Analysis and Insights

The "Latent" Advantage

By treating the ranking as latent, the model avoids the bias introduced by choosing an arbitrary gold ranking. This flexibility allows the model to focus on the core task: placing relevant answers at the top of the list.

Training Dynamics and Stability

The study found that while the Latent Structured Perceptron (LSP) can achieve high performance, it is sensitive to the number of training epochs.

MAP performance across training epochs for different LSP variants. Note the higher stability and performance of the AP-optimized versions.

As illustrated in the training curves, the AP-optimized variants (LSP-AP and LSP-AP*) generally reach higher MAP plateaus than the standard LSP, though they still exhibit some fluctuations that require careful tuning to avoid performance degradation.

Structural vs. Neural Features

While Deep Learning (CNNs) provides superior feature representations, the structured learning framework remains complementary. Feeding CNN embeddings into these structured models yields performance that matches or exceeds state-of-the-art neural architectures, suggesting that global list structure is a signal that neural models alone may not fully exploit.

Conclusion

This work demonstrates that structured prediction, combined with latent variable modeling and efficient MAP optimization, provides a powerful framework for passage reranking. The proposed greedy inference method offers a practical balance between computational efficiency and ranking quality. Future work will explore the end-to-end integration of these structured loss functions directly into deep neural network architectures.

Summary updated

BibTeX
@inproceedings{haponchyk2019study,
  author = {Iryna Haponchyk and Alessandro Moschitti},
  title = {{A Study of Latent Structured Prediction Approaches to Passage Reranking}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1183},
}
Figure