Overview of the TREC 2020 Fair Ranking Track

Fernando Diaz, Asia J. Biega, Fernando Diaz 0001, Michael D. Ekstrand, Sergey Feldman, Sebastian KohlmeierarXiv 2021 · arXiv

Abstract

The TREC 2020 Fair Ranking Track addresses the "winner-take-all" phenomenon in search engines, where a small number of top-ranked items capture the vast majority of user attention. This track introduces a framework for evaluating "expected exposure," aiming to balance consumer relevance with producer fairness. This paper summarizes the track's methodology, datasets, and key findings from participant submissions, highlighting the trade-offs between ranking relevance and group equity.

Introduction: The Challenge of Group Fairness

Traditional Information Retrieval (IR) systems prioritize relevance above all else, typically placing the most relevant document at the top of the list. While effective for information consumers, this often results in significant visibility disparities for content producers. In academic search, for instance, researchers from prestigious institutions or well-known authors may dominate search results, overshadowing equally relevant work from underrepresented groups or developing regions.

The TREC 2020 Fair Ranking Track focuses on Group Fairness, ensuring that groups of authors (categorized by demographics or geography) receive exposure proportional to the relevance of their contributions. Because ranking is inherently a zero-sum game—only one document can occupy the first position—systems must distribute exposure across multiple search sessions or "impressions" over time.

Methodology: The Expected Exposure Framework

The track evaluated systems on two primary tasks: Reranking (reordering a provided list of documents) and Retrieval (finding the top 100 documents from a massive corpus). The technical backbone of the evaluation is the Expected Exposure metric, which utilizes a probabilistic browsing model to quantify attention.

The evaluation protocol (Algorithm 1) and the state transition diagram for the browsing model.

As shown in the evaluation protocol, the model assumes:

  1. Browsing Behavior: Users start at the top of a list and have a specific probability of "stopping" at each document based on its relevance.
  2. Individual and Group Exposure: Exposure is calculated as the cumulative probability of a user examining an author's documents across a sequence of rankings.
  3. Target Exposure: A "target" exposure is defined based on a policy that perfectly reflects document relevance.
  4. Disparity ($\Delta_G$): Systems are evaluated by the difference between actual group exposure and the target exposure. Lower $\Delta_G$ values indicate higher fairness.

Participants employed various novel strategies to optimize these metrics, including stochastic ranking to "spread" exposure and fairness-aware objective functions that optimize for KL-divergence between output and target distributions.

Datasets and Group Definitions

The track utilized the Semantic Scholar (S2) Open Corpus, a massive dataset of 186 GB covering millions of academic papers. Queries were sampled from real-world search logs, specifically excluding "known-item" searches to focus on broader topical discovery.

Authors were grouped based on the economic development level of their home country (Advanced vs. Developing economies), using IMF classifications. NIST assessors manually annotated papers to provide ground truth for author affiliations and institution types. This process highlighted significant challenges in data completeness, as many papers lacked sufficient affiliation information in their metadata.

Key Results and Insights

The evaluation revealed a clear "Pareto frontier" between relevance and fairness. Systems that achieved the lowest disparity often saw a decrease in relevance, though top performers managed to balance both effectively.

Scatter plot showing the trade-off between relevance and disparity across different participant runs.

Key findings from the track include:

  • Top Performers: The NLE team (Allen Institute for AI) achieved the best reranking results, with $\Delta_G$ scores as low as 0.428.
  • Effectiveness of Randomization: Approaches that used stochastic ranking and multiple permutations per query generally outperformed static, deterministic models in achieving fairness.
  • The Power of External Data: Successful teams leveraged external resources, such as Google Scholar metadata and pre-trained word embeddings, to better estimate author stature and document relevance.

Conclusion and Future Directions

The TREC 2020 Fair Ranking Track highlights that fairness in IR is best viewed as a distribution over time rather than a static property of a single list. By moving beyond simple relevance, the track challenges the IR community to consider the rights of content producers alongside the needs of consumers.

However, limitations remain. The difficulty of manual annotation meant that papers with more than 10 authors were excluded, potentially biasing results. Furthermore, the reliance on a single group definition (economic status) leaves room for future work to explore intersectional fairness and more robust methods for handling missing data.

Summary updated

Abstract

This paper provides an overview of the NIST TREC 2020 Fair Ranking track. For 2020, we again adopted an academic search task, where we have a corpus of academic article abstracts and queries submitted to a production academic search engine. The central goal of the Fair Ranking track is to provide fair exposure to different groups of authors (a group fairness framing). We recognize that there may be multiple group definitions (e.g. based on demographics, stature, topic) and hoped for the systems to be robust to these. We expected participants to develop systems that optimize for fairness and relevance for arbitrary group definitions, and did not reveal the exact group definitions until after the evaluation runs were submitted.The track contains two tasks,reranking and retrieval, with a shared evaluation.
BibTeX
@misc{diaz2021overview,
  author = {Fernando Diaz and Asia J. Biega and Fernando Diaz and Michael D. Ekstrand and Sergey Feldman and Sebastian Kohlmeier},
  title = {{Overview of the TREC 2020 Fair Ranking Track}},
  year = {2021},
  eprint = {2108.05135},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.05135},
}
Figure