Choppy: Cut Transformer for Ranked List Truncation

Dara Bahri, Yi Tay, Che Zheng, Donald Metzler, Andrew Tomkins

TL;DR

Choppy is a Transformer-based model designed to solve the "ranked list truncation" problem—deciding exactly where to cut off a list of search results to balance relevance against user effort. By treating the cutoff point as a joint probability distribution and directly optimizing for standard IR metrics (like F1 or DCG), Choppy achieves an 11.5% performance boost over previous state-of-the-art models while running three times faster.

What Problem Does This Paper Address?

In Information Retrieval (IR), most research focuses on ranking: putting the best results at the top. However, a secondary but critical problem is truncation: determining the optimal number of results ($k$) to actually show the user.

Showing too many results leads to "ad blindness" or high processing costs for the user (e.g., in legal e-discovery), while showing too few might omit the information the user actually needs. Previous state-of-the-art methods like BiCut used Recurrent Neural Networks (LSTMs) to make sequential "keep/stop" decisions. These suffered from "exposure bias" (errors compounding during inference), struggled with long-range dependencies, and used loss functions that didn't perfectly align with the metrics users actually care about.

Methods

The authors introduce Choppy, which moves away from sequential decision-making in favor of a global, attention-based approach:

  1. Transformer Architecture: Choppy uses a multi-head self-attention mechanism to capture "dyadic interactions" between relevance scores across the entire list. This allows the model to see the "big picture" of the result set before deciding where to cut.
  2. Learnable Positional Embeddings: Since Transformers are inherently permutation-invariant and the input is just a 1D list of scalar scores, the authors add a learnable positional embedding to help the model understand the rank order.
  3. Expected Metric Loss: This is the paper's most significant technical contribution. Instead of a standard classification loss, they define a loss function that calculates the expected value of a user-defined metric (like F1 or Precision) over all possible cut positions. This ensures the model is trained to directly maximize the final evaluation score.
  4. Joint Distribution: Unlike models that make a binary choice at every step, Choppy outputs a softmax distribution over all $n$ possible cut positions, treating the list length as a single global prediction.

Datasets

The model was evaluated using the TREC Robust04 collection, a standard benchmark containing 250 queries and roughly 528,000 news articles. To test Choppy's versatility, the authors applied it to two different types of initial rankings:

  • BM25: A traditional probabilistic "bag-of-words" retrieval model.
  • DRMM: A modern deep relevance matching model (neural ranking).

Key Insights and Ideas

  • Self-Attention on Scalars: This is one of the first successful applications of self-attention to a sequence of simple scalar ranking scores rather than complex word embeddings.
  • Metric-Agnostic Training: Because the loss function is framed as an expectation of a metric, you can swap in almost any IR metric (F1, Precision, DCG) without changing the model architecture.
  • Solving Exposure Bias: By predicting the cut position all at once (non-autoregressively), Choppy avoids the "train/test mismatch" that plagues sequential models like BiLSTMs.
  • Efficiency Matters: Because Transformers allow for parallel processing of the sequence, Choppy is significantly faster (>3x) during inference than its recurrent predecessors.

Key Results

  • State-of-the-Art Performance: On the Robust04 dataset (using BM25), Choppy achieved a 0.272 F1 score, representing an 11.5% relative improvement over the BiCut model.
  • DCG Support: Unlike previous models that struggled to optimize for Discounted Cumulative Gain (DCG), Choppy showed strong performance on this metric (0.295 on DRMM).
  • Hyperparameter Stability: Ablation studies showed that the model is remarkably robust; performance remained stable across various model dimensions ($d$) and numbers of attention heads ($h$), suggesting it requires very little "tuning" to work well on new data.

Limitations and Future Work

  • Candidate Limit: The study limited the truncation task to the top 300 results. While sufficient for most web searches, some specialized fields (like systematic medical reviews) might require looking deeper into the tail.
  • Feature Set: Choppy relies solely on relevance scores. The authors suggest that incorporating actual document content or more diverse features could further improve accuracy.
  • Generalization: While the authors expect Choppy to work "out-of-the-box" on other datasets, the paper focuses its empirical evidence primarily on the Robust04 benchmark. Future work could explore its performance in more diverse production environments like sponsored search or legal discovery.

Summary updated

Abstract

Work in information retrieval has traditionally focused on ranking and relevance: given a query, return some number of results ordered by relevance to the user. However, the problem of determining how many results to return, i.e. how to optimally truncate the ranked result list, has received less attention despite being of critical importance in a range of applications. Such truncation is a balancing act between the overall relevance, or usefulness of the results, with the user cost of processing more results. In this work, we propose Choppy, an assumption-free model based on the widely successful Transformer architecture, to the ranked list truncation problem. Needing nothing more than the relevance scores of the results, the model uses a powerful multi-head attention mechanism to directly optimize any user-defined IR metric. We show Choppy improves upon recent state-of-the-art methods.
BibTeX
@inproceedings{bahri2020choppy,
  author = {Dara Bahri and Yi Tay and Che Zheng and Donald Metzler and Andrew Tomkins},
  title = {{Choppy: Cut Transformer for Ranked List Truncation}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2020},
  doi = {10.1145/3397271.3401188},
}

Loading related content...

Figure