Abstract
The TREC 2020 Fair Ranking Track addresses the "winner-take-all" phenomenon in search engines, where a small number of top-ranked items capture the vast majority of user attention. This track introduces a framework for evaluating "expected exposure," aiming to balance consumer relevance with producer fairness. This paper summarizes the track's methodology, datasets, and key findings from participant submissions, highlighting the trade-offs between ranking relevance and group equity.
Introduction: The Challenge of Group Fairness
Traditional Information Retrieval (IR) systems prioritize relevance above all else, typically placing the most relevant document at the top of the list. While effective for information consumers, this often results in significant visibility disparities for content producers. In academic search, for instance, researchers from prestigious institutions or well-known authors may dominate search results, overshadowing equally relevant work from underrepresented groups or developing regions.
The TREC 2020 Fair Ranking Track focuses on Group Fairness, ensuring that groups of authors (categorized by demographics or geography) receive exposure proportional to the relevance of their contributions. Because ranking is inherently a zero-sum game—only one document can occupy the first position—systems must distribute exposure across multiple search sessions or "impressions" over time.
Methodology: The Expected Exposure Framework
The track evaluated systems on two primary tasks: Reranking (reordering a provided list of documents) and Retrieval (finding the top 100 documents from a massive corpus). The technical backbone of the evaluation is the Expected Exposure metric, which utilizes a probabilistic browsing model to quantify attention.
As shown in the evaluation protocol, the model assumes:
- Browsing Behavior: Users start at the top of a list and have a specific probability of "stopping" at each document based on its relevance.
- Individual and Group Exposure: Exposure is calculated as the cumulative probability of a user examining an author's documents across a sequence of rankings.
- Target Exposure: A "target" exposure is defined based on a policy that perfectly reflects document relevance.
- Disparity ($\Delta_G$): Systems are evaluated by the difference between actual group exposure and the target exposure. Lower $\Delta_G$ values indicate higher fairness.
Participants employed various novel strategies to optimize these metrics, including stochastic ranking to "spread" exposure and fairness-aware objective functions that optimize for KL-divergence between output and target distributions.
Datasets and Group Definitions
The track utilized the Semantic Scholar (S2) Open Corpus, a massive dataset of 186 GB covering millions of academic papers. Queries were sampled from real-world search logs, specifically excluding "known-item" searches to focus on broader topical discovery.
Authors were grouped based on the economic development level of their home country (Advanced vs. Developing economies), using IMF classifications. NIST assessors manually annotated papers to provide ground truth for author affiliations and institution types. This process highlighted significant challenges in data completeness, as many papers lacked sufficient affiliation information in their metadata.
Key Results and Insights
The evaluation revealed a clear "Pareto frontier" between relevance and fairness. Systems that achieved the lowest disparity often saw a decrease in relevance, though top performers managed to balance both effectively.
Key findings from the track include:
- Top Performers: The NLE team (Allen Institute for AI) achieved the best reranking results, with $\Delta_G$ scores as low as 0.428.
- Effectiveness of Randomization: Approaches that used stochastic ranking and multiple permutations per query generally outperformed static, deterministic models in achieving fairness.
- The Power of External Data: Successful teams leveraged external resources, such as Google Scholar metadata and pre-trained word embeddings, to better estimate author stature and document relevance.
Conclusion and Future Directions
The TREC 2020 Fair Ranking Track highlights that fairness in IR is best viewed as a distribution over time rather than a static property of a single list. By moving beyond simple relevance, the track challenges the IR community to consider the rights of content producers alongside the needs of consumers.
However, limitations remain. The difficulty of manual annotation meant that papers with more than 10 authors were excluded, potentially biasing results. Furthermore, the reliance on a single group definition (economic status) leaves room for future work to explore intersectional fairness and more robust methods for handling missing data.
Summary updated