Beyond classification and ranking: constrained optimization of the ROI

Lian Yan, Patrick BaldasareKDD 2006 · DOI

TL;DR

This paper introduces a novel algorithm that directly optimizes Return on Investment (ROI) for financial applications by framing it as a constrained optimization problem. Unlike traditional classification or ranking methods, this approach directly targets the monetary value of successful outcomes under a fixed budget, leading to substantial improvements in financial impact for clients.

What Problem Does This Paper Address?

In many financial industry applications, like predicting account collectability or customer defection, the goal is not just to accurately classify an outcome (e.g., will a customer pay or defect) but to maximize the financial return. Traditional classification methods, even those with cost-sensitive learning, often fall short because optimizing for true positive rates doesn't directly translate to maximizing the actual monetary value collected or retained. This is especially true when there's a budget constraint that limits the number of accounts that can be targeted. For instance, identifying a high percentage of potential defectors is less valuable than identifying the few defectors who would have the largest financial impact. The paper addresses this gap by proposing a method that directly optimizes the monetary measure of success (like collection amount or net redemption) within a defined budget.

Methods

The core of the paper is a new algorithm that formulates the problem as a constrained optimization task. Instead of predicting a binary class or a rank, the algorithm aims to find a model output (a score) such that the accounts with the highest scores, up to a certain "pull rate" (determined by the budget), correspond to the accounts with the highest monetary value.

The authors achieve this by:

  • Directly Optimizing Monetary Value: The objective function is designed to maximize the sum of the target monetary measure (e.g., collection amount, net redemption) for the top r% of accounts, where r is the pull rate dictated by the budget.
  • Constrained Optimization: This maximization is performed under the constraint that the number of selected accounts (those above a certain threshold) must be exactly r% of the total.
  • Differentiable Approximation: Since the constraint and the indicator function for selecting accounts are non-differentiable, the paper proposes differentiable approximations. This allows for the use of gradient-based optimization methods.
  • Lagrangian Formulation: The constrained problem is converted into an unconstrained one using a Lagrangian, where a parameter (μ) is gradually decreased during training to enforce the constraint.
  • Parametric Models: The approach is designed to work with any parametric model where gradients can be computed, such as Multi-Layer Perceptrons (MLPs).

Datasets

The paper evaluates its proposed algorithm on two real-world datasets:

  • Accounts Receivable Data: Collected from several US financial institutions, focusing on delinquent consumer loan accounts. The goal was to predict collectability and maximize the collection amount from the top 11% of accounts.
  • Mutual Fund Defection Data: From a major US mutual fund company, aiming to predict account defection and maximize net redemption from the top 10% of accounts.

The datasets consist of numerous features including account history, credit information, demographic data, and transaction details. The authors mention using a training/test split for evaluation.

Key Insights and Ideas

  • ROI Optimization is Not Classification/Ranking: The most significant insight is that optimizing for classification accuracy or ranking alone does not guarantee optimal financial returns. The direct optimization of the monetary outcome is crucial.
  • Budget Constraint is Key: The paper emphasizes that real-world financial applications operate under budget constraints, which directly translate to a "pull rate" or the number of accounts that can be targeted. The algorithm explicitly incorporates this.
  • Differentiable Approximations Enable Gradient Descent: The clever use of differentiable approximations for non-differentiable functions is what makes gradient-based optimization feasible for this direct ROI maximization problem.
  • "Valuable False Positives": The algorithm can identify accounts that might not meet a strict definition of "defector" (or "delinquent") but still represent high potential monetary value. This is a key advantage over traditional classification.
  • Generalizability: The authors suggest the algorithm's applicability extends beyond the two demonstrated cases to other areas like stock selection, tax auditing, and fundraising.

Key Results

The paper demonstrates substantial improvements in financial impact compared to several baseline methods:

  • Accounts Receivable: The proposed model achieved an average collection amount of $157** per account in the top 11% of accounts, significantly outperforming classification ($126), weighted classification ($106), ranking ($61), and regression ($116) models. This represents a 25% improvement** over the classification model.
  • Mutual Fund Defection: While specific numbers are presented in Table 4, the paper highlights that the classification model, despite a higher true positive rate (39% vs. 14% for the new model), was less effective at identifying accounts with the highest redemption amounts. The new model's focus on direct monetary value led to better financial outcomes.

The optimization process itself showed convergence in terms of pull rate, with the test set pull rate closely matching the target pull rate, validating the effectiveness of the differentiable approximations.

Limitations and Future Work

  • Savability: The authors acknowledge that "savability" (the likelihood of successfully retaining an account once identified) is a crucial factor for ROI but is difficult to model directly. They mention an empirical approach of increasing the pull rate or excluding accounts with low savability scores. Future work could involve combining savability models more formally.
  • Valuable False Positives vs. Domain Expert Definitions: Companies might be hesitant to adopt a model with a low true positive rate according to their existing definitions, even if it maximizes ROI. The paper notes that an intuitive attempt to incorporate the true positive rate into the objective function didn't work well, suggesting further research is needed to balance these objectives.
  • No Budget Constraint Scenario: The paper briefly touches on scenarios without a fixed budget, suggesting that optimizing for marginal collection cost equaling marginal revenue could be a direction.
  • Algorithmic Extensions: The authors mention exploring other parametric models and optimizing the threshold parameter β alongside other model parameters. They also suggest potential applications in customer churn prediction where revenue loss is the primary concern.

Summary updated

BibTeX
@inproceedings{yan2006beyond,
  author = {Lian Yan and Patrick Baldasare},
  title = {{Beyond classification and ranking: constrained optimization of the ROI}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2006},
  doi = {10.1145/1150402.1150533},
}
Figure