Cost-Sensitive Learning to Rank

Ryan McBride, Ke Wang, Zhouyang Ren, Wenyuan LiAAAI 2019 · DOI

Abstract

Standard Learning to Rank (LTR) algorithms, originally developed for web search, treat all queries as equally important. However, in high-stakes domains like emergency response or infrastructure management, the cost of misranking a catastrophic event is far greater than misranking a minor one. This paper introduces Cost-Sensitive Learning to Rank (CS-LTR), a framework that prioritizes high-impact scenarios—such as massive power outages or forest fires—over routine events. By adapting standard LTR algorithms to optimize for actual cost-savings rather than normalized relevance scores, the authors demonstrate significant improvements in resource allocation efficiency across multiple real-world datasets.

Introduction

In Information Retrieval (IR), ranking models are typically evaluated based on their ability to find relevant documents for a given query. When these models are applied to risk management, two critical "mismatches" emerge:

  1. List Importance: Standard metrics like NDCG (Normalized Discounted Cumulative Gain) normalize scores for every list. In a search engine, every query is roughly equal. In risk management, a storm that threatens 10,000 power outages is orders of magnitude more important than a storm threatening 10. Standard LTR treats them as equally important "queries."
  2. Gain Functions: IR metrics often use an exponential gain function ($2^y - 1$) to prioritize "highly relevant" results. In the physical world, costs are often linear. Ranking an event with 51 impacted customers ahead of one with 50 isn't twice as valuable; it’s just slightly better.
Table showing how different storms have vastly different impacts, illustrating the need for cost-sensitive ranking.

The authors propose bridging this gap by creating models that maximize actual cost-savings. This approach ensures that organizations with limited resources can prioritize actions where they will prevent the most damage.

Methodology

The researchers introduced two primary evaluators and several algorithmic adaptations to incorporate cost sensitivity:

  • New Metrics ($R_{CS}@k$): They defined $R@k$ as a cost-sensitive evaluator for a single list, incorporating domain-specific cost functions and the probability of taking action at a specific rank. This is scaled to $R_{CS}@k$, which weights different lists by their total potential cost-saving impact.
  • Cost-Sensitive (CS) Model Search: Standard LTR algorithms—including LambdaMART, Coordinate Ascent, and AdaRank—were adapted to directly optimize for the $R_{CS}$ metric. For instance, CS-MART replaces the standard IR-based gradient with one driven by potential cost-savings.
  • Cost-Reweighted (CR) Model Search: For gradient-based methods like RankNet and ListNet, the authors introduced a simpler adaptation: reweighting the loss function of each list by its total potential cost. This forces the model to prioritize high-impact events during the training phase.

Experimental Setup

The framework was evaluated on five diverse datasets to test its generalizability:

  • Proprietary Outage Data: Two datasets from BC Hydro (High-Risk Outages and Customers Outages) using weather and infrastructure features to predict power failures.
  • UCI Crime: Ranking geographical districts by crime rates.
  • UCI Forest Fire: Ranking areas by the acreage of fire damage.
  • UCI Concrete: Ranking concrete batches by compressive strength.

Results and Discussion

The proposed CS and CR methods consistently outperformed standard LTR baselines and traditional regression models. In 81 out of 84 experimental tests, the cost-sensitive versions achieved superior results.

Comprehensive comparison of cost-sensitive algorithms against standard baselines across multiple datasets.

On the "Customers Outages" dataset, CS-MART achieved a cost-saving score of 19.2% (at k=10), a dramatic improvement over the 6.0% managed by standard LambdaMART. The paper also highlights that while a regression model might have a low Root Mean Square Error (RMSE), it can still fail as a ranker by misordering the most critical items. CS-MART successfully prioritized high-cost failure features, such as vegetation density, over more frequent but lower-impact features like wind speed.

Limitations and Future Work

The authors noted that gradient-based methods like ListNet and RankNet faced numerical stability issues when dealing with very high cost values due to exponential terms in their loss functions. Additionally, the framework's effectiveness depends on accurate cost functions and action probabilities provided by domain experts. Future research may explore more complex resource allocation policies that extend beyond simple rank-based probabilities.

Conclusion

Cost-Sensitive Learning to Rank represents a significant shift from traditional IR-focused ranking. By aligning model optimization with real-world economic or social costs, this framework provides a robust tool for decision-makers in fields where resource allocation is both limited and high-stakes.

Summary updated

BibTeX
@inproceedings{mcbride2019cost,
  author = {Ryan McBride and Ke Wang and Zhouyang Ren and Wenyuan Li},
  title = {{Cost-Sensitive Learning to Rank}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33014570},
}

Loading related content...

Figure