Choppy: Cut Transformer For Ranked List Truncation

Dara Bahri, Yi Tay, Che Zheng, Donald Metzler, Andrew Tomkins

Abstract

Work in information retrieval has traditionally focused on ranking and relevance: given a query, return some number of results ordered by relevance to the user. However, the problem of determining how many results to return, i.e. how to optimally truncate the ranked result list, has received less attention despite being of critical importance in a range of applications. Such truncation is a balancing act between the overall relevance, or usefulness of the results, with the user cost of processing more results. In this work, we propose Choppy, an assumption-free model based on the widely successful Transformer architecture, to the ranked list truncation problem. Needing nothing more than the relevance scores of the results, the model uses a powerful multi-head attention mechanism to directly optimize any user-defined IR metric. We show Choppy improves upon recent state-of-the-art methods.

Introduction

In Information Retrieval (IR), the primary goal is typically to rank documents by relevance. However, a critical secondary challenge is determining the optimal number of results ($k$) to present to a user. This "ranked list truncation" problem is a delicate balancing act: providing too many results can lead to information overload or high processing costs (e.g., in legal e-discovery), while providing too few may exclude vital information.

Traditional approaches often rely on sequential decision-making, such as LSTMs that decide whether to "keep" or "stop" at each rank. These methods frequently suffer from exposure bias—where errors accumulate during inference—and struggle to capture long-range dependencies within the list. Furthermore, their loss functions often do not align directly with the evaluation metrics that matter most to users.

The Choppy Model

We introduce Choppy, a Transformer-based architecture that treats truncation as a global optimization problem rather than a sequential one. By leveraging multi-head self-attention, Choppy captures "dyadic interactions" between relevance scores across the entire candidate list, allowing it to evaluate the "big picture" before making a cut.

Architecture and Global Distribution

Unlike autoregressive models, Choppy outputs a softmax distribution over all possible cut positions $n$ simultaneously. This global approach eliminates the train/test mismatch inherent in sequential models. Because Transformers are permutation-invariant, we incorporate learnable positional embeddings to preserve the crucial rank-order information of the input scores.

Expected Metric Loss

A core contribution of Choppy is the Expected Metric Loss. Instead of standard classification or regression losses, we train the model to directly maximize the expected value of a user-defined IR metric (e.g., F1, Precision, or DCG) over the predicted probability distribution. This metric-agnostic approach allows the model to be tailored to specific application needs without architectural changes.

F1 score and predicted probability distribution across candidate cut positions for three sample queries, illustrating how Choppy adapts to varying query characteristics.

Experimental Setup

We evaluated Choppy using the TREC Robust04 benchmark, which contains 250 queries and approximately 528,000 documents. To demonstrate versatility, we tested the model on two distinct initial rankings:

  1. BM25: A classic probabilistic retrieval model.
  2. DRMM: A deep relevance matching model representing modern neural ranking.

The truncation task was limited to the top 300 results, a range sufficient for most general search applications.

Key Results

Choppy demonstrates significant improvements over previous state-of-the-art models like BiCut.

Performance and Efficiency

On the Robust04 dataset using BM25 rankings, Choppy achieved an F1 score of 0.272, an 11.5% relative improvement over BiCut. It also showed strong performance on DCG (0.295 on DRMM), a metric that has historically been difficult for truncation models to optimize. Beyond accuracy, Choppy's parallelizable Transformer architecture makes it over three times faster during inference than recurrent alternatives.

Hyperparameter Stability

Ablation studies reveal that Choppy is remarkably robust to hyperparameter choices. Performance remains consistent across a wide range of model dimensions ($d$) and numbers of attention heads ($h$). This stability suggests that Choppy requires minimal tuning to achieve high performance on new datasets.

Sensitivity analysis of Choppy's performance (F1 and DCG) relative to model dimension and the number of attention heads, demonstrating high stability.

Conclusion

Choppy represents a shift in ranked list truncation by moving from sequential heuristics to global, attention-based optimization. By directly optimizing for expected IR metrics and leveraging the efficiency of the Transformer architecture, it provides a faster and more accurate solution for balancing relevance and user effort. Future work will explore incorporating document content and extending the model to handle deeper result lists in specialized domains like medical reviews.

Summary updated

Abstract

Work in information retrieval has traditionally focused on ranking and relevance: given a query, return some number of results ordered by relevance to the user. However, the problem of determining how many results to return, i.e. how to optimally truncate the ranked result list, has received less attention despite being of critical importance in a range of applications. Such truncation is a balancing act between the overall relevance, or usefulness of the results, with the user cost of processing more results. In this work, we propose Choppy, an assumption-free model based on the widely successful Transformer architecture, to the ranked list truncation problem. Needing nothing more than the relevance scores of the results, the model uses a powerful multi-head attention mechanism to directly optimize any user-defined IR metric. We show Choppy improves upon recent state-of-the-art methods.
BibTeX
@inproceedings{bahri2020choppy,
  author = {Dara Bahri and Yi Tay and Che Zheng and Donald Metzler and Andrew Tomkins},
  title = {{Choppy: Cut Transformer For Ranked List Truncation}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2020},
  doi = {10.1145/3397271.3401188},
}
Figure