AP-Perf: Incorporating Generic Performance Metrics in Differentiable Learning

Rizal Fathony, J. Zico Kolter

Abstract: We propose a method that enables practitioners to conveniently incorporate custom non-decomposable performance metrics into differentiable learning pipelines, notably those based upon neural networks. By framing the problem as an adversarial game and using a clever marginalization technique, the authors provide a "plug-and-play" tool that outperforms standard cross-entropy training on specialized evaluation tasks.

The Challenge of Non-Decomposable Metrics

Most neural networks are trained to minimize Binary Cross-Entropy (BCE). While BCE is an effective proxy for accuracy, it often fails in real-world scenarios where accuracy is not the primary goal. For instance, in medical diagnosis or fraud detection, practitioners might prioritize the F-score, Sensitivity, or Cohen’s Kappa over raw accuracy.

These metrics are "non-decomposable," meaning the total score cannot be calculated by simply summing individual losses for each sample; the entire dataset or batch must be evaluated at once. Historically, optimizing these metrics has been difficult because they are often non-convex and non-continuous, making them "gradient-unfriendly." Existing solutions were typically limited to specific metrics and difficult to integrate into modern deep learning frameworks.

Methods: The Adversarial Prediction Framework

The authors leverage the Adversarial Prediction Framework to address these challenges. Instead of standard Empirical Risk Minimization, they treat learning as a zero-sum game between a Predictor, which tries to maximize the expected performance metric, and an Adversary, which tries to minimize it by choosing a distribution that matches the training data's feature statistics.

Common non-decomposable performance metrics that can be optimized using the AP-Perf framework.

Key Innovations

  • Marginalization: Solving this adversarial game naively is exponentially complex. The authors developed a technique to reduce this to quadratic complexity by focusing on the marginal probabilities of confusion matrix entities (True Positives, False Positives, etc.).
  • Differentiable Wrapper: They provide a programming interface where a user defines a metric using confusion matrix terms. The framework automatically handles the underlying optimization and provides gradients to the neural network.
  • Custom Solver: To maintain training efficiency, they implemented a solver using the Alternating Direction Method of Multipliers (ADMM), allowing the inner optimization to be solved in roughly 10–30 milliseconds per batch.

Experimental Results

The authors evaluated AP-Perf across 22 datasets (including UCI benchmarks, MNIST, and Fashion MNIST) and 8 different metrics. The results demonstrate that AP-Perf consistently yields better results for the target metric compared to standard BCE training.

Performance comparison between AP-Perf and standard BCE. Points above the diagonal line indicate superior performance by the AP-Perf framework.

As shown in the scatter plot, nearly all experimental points lie above the $y=x$ line. The benefit was most pronounced on "hard" problems where BCE achieved good accuracy but poor scores on imbalanced metrics like the F2-score. AP-Perf showed significant gains on the F2-score specifically because it can capture the nuance of weighting recall more heavily than precision—a detail standard cross-entropy cannot easily model.

Conclusion and Future Work

AP-Perf provides a robust, Fisher-consistent method for optimizing complex metrics directly within differentiable pipelines. While the current implementation focuses on binary classification and metrics derivable from the confusion matrix, it represents a significant step toward more flexible machine learning objectives. Future work aims to extend the framework to support area-based metrics like AUC-ROC and multi-class classification problems.

Summary updated

Abstract

We propose a method that enables practitioners to conveniently incorporate custom non-decomposable performance metrics into differentiable learning pipelines, notably those based upon neural networ...
BibTeX
@inproceedings{fathony2020ap,
  author = {Rizal Fathony and J. Zico Kolter},
  title = {{AP-Perf: Incorporating Generic Performance Metrics in Differentiable Learning}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=iatN2e20qbQ},
}
Figure