Rank-Learner: Orthogonal Ranking of Treatment Effects

Henri Arno, Dennis Frauen, Emil Javurek, Thomas Demeester, Stefan Feuerriegel

TL;DR

This paper introduces Rank-Learner, a novel method for directly learning the ranking of treatment effects from observational data. Unlike existing approaches that first estimate precise treatment effects (a harder problem), Rank-Learner uses a specialized orthogonal learning objective to focus solely on the ranking, leading to improved robustness and performance.

What Problem Does This Paper Address?

Many real-world decision-making scenarios require prioritizing individuals based on their expected benefit from a treatment, rather than knowing the exact magnitude of that benefit. Examples include triaging patients for medical care or targeting marketing campaigns. While estimating individual treatment effects (ITE) or conditional average treatment effects (CATE) has been extensively studied, directly learning the ranking of these effects from observational data has been largely overlooked. Existing methods often rely on estimating CATEs first, which is computationally more demanding and can be sensitive to errors in estimating intermediate "nuisance" functions (like propensity scores or outcome models). This paper aims to bridge this gap by proposing a method that directly learns rankings in a robust and efficient manner.

Methods

The core of Rank-Learner is a novel two-stage learning approach:

  1. Nuisance Estimation: In the first stage, standard machine learning models (e.g., neural networks) are used to estimate "nuisance functions." These include the propensity score (probability of receiving treatment) and the conditional outcome models (expected outcome with and without treatment). Crucially, this stage employs cross-fitting to ensure robust estimates.
  2. Orthogonal Ranking Learning: In the second stage, Rank-Learner uses a Neyman-orthogonal pairwise ranking objective. This is the key innovation. Instead of trying to recover precise CATEs, it optimizes a loss function that is specifically designed to learn the relative ordering of treatment effects. The "Neyman-orthogonality" ensures that this learning process is robust to estimation errors in the nuisance functions from the first stage. This is achieved by constructing "pseudo labels" that incorporate an orthogonal correction term. Rank-Learner is also model-agnostic, meaning any machine learning model can be plugged into its framework.

Datasets

The authors evaluate Rank-Learner on:

  • Synthetic datasets: Generated to have controlled properties, allowing for a clear understanding of ranking performance.
  • Semi-synthetic datasets: Real-world covariates from MOVIELENS, MIMIC-III, and CURRENT POPULATION SURVEY (CPS) are combined with simulated treatments and outcomes.
  • Real-world uplift benchmark: The CRITEO uplift dataset, used for evaluating ranking quality in a practical advertising scenario.

Key Insights and Ideas

  • Ranking is Easier than Estimation: Estimating the precise magnitude of treatment effects is a harder problem than simply determining their relative order. Rank-Learner exploits this by focusing directly on the ranking objective.
  • Orthogonality is Key for Robustness: By making the ranking objective Neyman-orthogonal, Rank-Learner becomes robust to errors in estimating nuisance functions. This is crucial in observational studies where these estimates are never perfect.
  • Direct Ranking Outperforms CATE-First Approaches: Empirically, directly optimizing for ranking yields better results than first estimating CATEs and then ranking based on those estimates.
  • Model Agnosticism Offers Flexibility: The ability to use arbitrary machine learning models makes Rank-Learner adaptable to various data types and complexities.

Key Results

  • Rank-Learner consistently outperforms standard CATE estimators (like T-learner and DR-learner) and non-orthogonal ranking methods across synthetic, semi-synthetic, and real-world benchmarks.
  • On synthetic benchmarks, Rank-Learner achieves higher AUTOC scores (a metric for ranking quality) than baselines, with improvements being most pronounced in small-sample regimes where nuisance estimation error is higher.
  • The method demonstrates robustness to limited overlap in treatment assignment and is relatively insensitive to the choice of its smoothness parameter (κ).
  • Even with a small fraction of training pairs used for the second-stage learning, Rank-Learner achieves strong performance, indicating that computational efficiency can be achieved without significant performance loss.

Limitations and Future Work

  • Sensitivity to Nuisance Misspecification: While orthogonal, the method still degrades when nuisance functions are severely misspecified (e.g., using linear models for non-linear data). However, orthogonal methods show less degradation than non-orthogonal ones.
  • Choice of Smoothness Parameter (κ): While Rank-Learner is not highly sensitive to κ, its selection involves a bias-variance trade-off. The paper proposes a practical strategy using AUTOC on a validation set.
  • Future Work: The paper suggests exploring more sophisticated architectures and training procedures for Rank-Learner, as well as comparing it against a wider range of advanced CATE estimation and ranking methods. The authors also highlight the potential for Rank-Learner to be used downstream for instantiating different treatment policies for varying budget constraints.

Summary updated

Abstract

Many decision-making problems require ranking individuals by their treatment effects rather than estimating the exact effect magnitudes. Examples include prioritizing patients for preventive care interventions, or ranking customers by the expected incremental impact of an advertisement. Surprisingly, while causal effect estimation has received substantial attention in the literature, the problem of directly learning *rankings of treatment effects* has largely remained unexplored. In this paper, we introduce *Rank-Learner*, a novel two-stage learner that directly learns the ranking of treatment effects from observational data. We first show that naive approaches based on precise treatment effect estimation solve a harder problem than necessary for ranking, while our *Rank-Learner* optimizes a pairwise learning objective that recovers the true treatment effect ordering, without explicit CATE estimation. We further show that our *Rank-Learner* is Neyman-orthogonal and thus comes with strong theoretical guarantees, including robustness to estimation errors in the nuisance functions. In addition, our *Rank-Learner* is model-agnostic, and can be instantiated with arbitrary machine learning models (e.g., neural networks). We demonstrate the effectiveness of our method through extensive experiments where *Rank-Learner* consistently outperforms standard CATE estimators and non-orthogonal ranking methods. Overall, we provide practitioners with a new, orthogonal two-stage learner for ranking individuals by their treatment effects.
BibTeX
@inproceedings{arno2026rank,
  author = {Henri Arno and Dennis Frauen and Emil Javurek and Thomas Demeester and Stefan Feuerriegel},
  title = {{Rank-Learner: Orthogonal Ranking of Treatment Effects}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=xM56OFSUaN},
}
Figure