Abstract
We propose a method that enables practitioners to conveniently incorporate custom non-decomposable performance metrics into differentiable learning pipelines, notably those based upon neural network architectures. Our approach is based on the recently developed adversarial prediction framework, a distributionally robust approach that optimizes a metric in the worst case given the statistical summary of the empirical distribution. We formulate a marginal distribution technique to reduce the complexity of optimizing the adversarial prediction formulation over a vast range of non-decomposable metrics. We demonstrate how easy it is to write and incorporate complex custom metrics using our provided tool. Finally, we show the effectiveness of our approach various classification tasks on tabular datasets from the UCI repository and benchmark datasets, as well as image classification tasks.
Introduction
In modern machine learning, there is often a significant disconnect between the training objective and the evaluation metric. Most deep learning models are trained to minimize surrogate losses, such as binary cross-entropy (BCE), because these functions are smooth and differentiable. However, the ultimate performance of these models is typically measured using metrics like the F1-score, sensitivity, or the Matthews correlation coefficient (MCC).
The primary challenge lies in the fact that many of these metrics are non-decomposable. Unlike accuracy, which can be calculated as an average of individual sample losses, non-decomposable metrics depend on the entire set of predictions simultaneously, often derived from the components of a confusion matrix.
This "objective-metric mismatch" is particularly problematic in imbalanced datasets. A model might achieve high accuracy by simply predicting the majority class, yet fail completely on the specific metric (like recall or F-beta) that is critical for the application.
Methodology: The AP-Perf Framework
To bridge this gap, we introduce AP-Perf, a framework that allows for the direct optimization of complex performance metrics within standard differentiable learning pipelines. AP-Perf is built upon the Adversarial Prediction framework, which treats training as a distributionally robust minimax game. In this setup, the predictor (the neural network) aims to maximize the expected performance metric, while a constrained adversary attempts to find a distribution that minimizes it.
Marginalization and Efficiency
A naive implementation of this adversarial game is computationally prohibitive, as it involves an exponential number of possible label assignments. We address this by introducing a marginalization technique that reduces the optimization complexity to quadratic. This is achieved by optimizing over the marginal probabilities of the confusion matrix components rather than the full joint distribution.
Practical Implementation
The resulting objective is concave, allowing us to compute sub-gradients that integrate seamlessly with standard backpropagation in frameworks like PyTorch or Flux. To ensure training remains practical, we developed a specialized solver using the Alternating Direction Method of Multipliers (ADMM). This solver is highly efficient, typically requiring only 10–30 milliseconds per batch on a standard CPU. Furthermore, AP-Perf supports optimizing one metric subject to constraints on another (e.g., maximizing precision while ensuring recall stays above a certain threshold).
Experimental Evaluation
We evaluated AP-Perf across a diverse set of tasks, including 20 tabular datasets from the UCI repository (e.g., adult, bankmarketing, californiahousing) and standard image datasets like MNIST and Fashion MNIST. For multiclass datasets, we employed binary transformations to specifically test performance in imbalanced-class scenarios.
Key Results
Our experiments demonstrate that direct optimization consistently yields better results than standard surrogate losses. In a head-to-head comparison across 22 datasets, AP-Perf matched or outperformed BCE in nearly every scenario.
The benefits were most significant for "hard" problems and metrics that prioritize specific aspects of performance, such as the F2-score (which weights recall more heavily than precision). The framework also proved highly effective at handling complex constraints, such as optimizing precision at specific recall thresholds—a common requirement in high-stakes fields like medical diagnosis or fraud detection.
Conclusion and Future Work
AP-Perf provides a theoretically grounded, Fisher-consistent, and computationally efficient method for incorporating custom performance metrics into deep learning. By providing a simple Julia-based interface, we make it easy for practitioners to optimize what they actually care about.
Currently, the framework is limited to metrics that can be expressed as functions of the confusion matrix. It does not yet support area-based metrics (like AUC-ROC) or ranking-based metrics (like Precision@k). Extending the marginalization techniques to these classes of metrics is a key direction for future research.
Summary updated