Abstract: We propose a method that enables practitioners to conveniently incorporate custom non-decomposable performance metrics into differentiable learning pipelines, notably those based upon neural networks. By framing the problem as an adversarial game and using a clever marginalization technique, the authors provide a "plug-and-play" tool that outperforms standard cross-entropy training on specialized evaluation tasks.
The Challenge of Non-Decomposable Metrics
Most neural networks are trained to minimize Binary Cross-Entropy (BCE). While BCE is an effective proxy for accuracy, it often fails in real-world scenarios where accuracy is not the primary goal. For instance, in medical diagnosis or fraud detection, practitioners might prioritize the F-score, Sensitivity, or Cohen’s Kappa over raw accuracy.
These metrics are "non-decomposable," meaning the total score cannot be calculated by simply summing individual losses for each sample; the entire dataset or batch must be evaluated at once. Historically, optimizing these metrics has been difficult because they are often non-convex and non-continuous, making them "gradient-unfriendly." Existing solutions were typically limited to specific metrics and difficult to integrate into modern deep learning frameworks.
Methods: The Adversarial Prediction Framework
The authors leverage the Adversarial Prediction Framework to address these challenges. Instead of standard Empirical Risk Minimization, they treat learning as a zero-sum game between a Predictor, which tries to maximize the expected performance metric, and an Adversary, which tries to minimize it by choosing a distribution that matches the training data's feature statistics.
Key Innovations
- Marginalization: Solving this adversarial game naively is exponentially complex. The authors developed a technique to reduce this to quadratic complexity by focusing on the marginal probabilities of confusion matrix entities (True Positives, False Positives, etc.).
- Differentiable Wrapper: They provide a programming interface where a user defines a metric using confusion matrix terms. The framework automatically handles the underlying optimization and provides gradients to the neural network.
- Custom Solver: To maintain training efficiency, they implemented a solver using the Alternating Direction Method of Multipliers (ADMM), allowing the inner optimization to be solved in roughly 10–30 milliseconds per batch.
Experimental Results
The authors evaluated AP-Perf across 22 datasets (including UCI benchmarks, MNIST, and Fashion MNIST) and 8 different metrics. The results demonstrate that AP-Perf consistently yields better results for the target metric compared to standard BCE training.
As shown in the scatter plot, nearly all experimental points lie above the $y=x$ line. The benefit was most pronounced on "hard" problems where BCE achieved good accuracy but poor scores on imbalanced metrics like the F2-score. AP-Perf showed significant gains on the F2-score specifically because it can capture the nuance of weighting recall more heavily than precision—a detail standard cross-entropy cannot easily model.
Conclusion and Future Work
AP-Perf provides a robust, Fisher-consistent method for optimizing complex metrics directly within differentiable pipelines. While the current implementation focuses on binary classification and metrics derivable from the confusion matrix, it represents a significant step toward more flexible machine learning objectives. Future work aims to extend the framework to support area-based metrics like AUC-ROC and multi-class classification problems.
Summary updated