In machine learning, we rarely optimize for what we actually care about. We want high accuracy, but we optimize cross-entropy. We want a high F-measure, but we minimize mean squared error. We want a short traveling salesman path, but we train on classification labels for edges.
This discrepancy is known as the loss-metric mismatch. Because most evaluation metrics (like accuracy or ranking order) are discrete and non-differentiable, we use "surrogate" losses that are smooth and easy for gradient descent to navigate. However, as research across diverse fields shows, minimizing a proxy does not guarantee maximizing the goal.
The Proxy Problem
The fundamental issue is that surrogate losses are often just loose upper bounds on the true error. Hang Li, Tie-Yan Liu, and their colleagues demonstrated this in Ranking Measures and Loss Functions in Learning to Rank, showing that popular ranking losses are essentially mathematical surrogates for 1-NDCG. When these bounds are loose, the model can "improve" its loss while its actual ranking performance stagnates or even declines.
This misalignment can lead to "bad" learning curves. Marco Loog and Tom J. Viering highlight a paradoxical phenomenon in A Survey of Learning Curves with Bad Behavior: adding more data can sometimes increase generalization error. This happens because more data allows the model to fit the surrogate objective more perfectly, driving it further away from the optimal solution for the actual target metric.
Differentiable Relaxations of Discrete Metrics
One way to bridge this gap is to create "soft" versions of hard metrics. Instead of abandoning the metric during training, researchers develop differentiable approximations that allow for direct optimization.
- Segmentation: Jinzheng Cai and Lin Yang replaced standard pixel-wise cross-entropy with a differentiable "Jaccard Loss" in Improving Deep Pancreas Segmentation to directly target the overlap metrics used in medical imaging.
- Saliency: In Optimizing the F-Measure for Threshold-Free Salient Object Detection, Kai Zhao and Wenguan Wang introduced FLoss, a relaxed version of the F-measure that forces the model to produce high-contrast maps that are naturally optimized for the final evaluation.
- Verification: Victoria Mingote and her team developed aDCF in Optimization of False Acceptance/Rejection Rates, allowing speaker verification systems to optimize directly for the Detection Cost Function rather than generic triplet losses.
Performance-Guided Optimization
When a metric is too complex to relax into a simple formula, we can use a second model to "predict" the performance of the first. This is the intuition behind NPPM (Network Pruning via Performance Maximization) by Shangqian Gao and Heng Huang. Rather than pruning neurons based on low loss (a poor proxy for accuracy), they train a dedicated predictor to estimate sub-network accuracy directly.
Similarly, Mahdi Milani Fard proposed Metric-Optimized Example Weights (MOEW). This framework learns to reweight training instances so that the resulting model maximizes a specific validation metric, whether that be fairness, precision at a certain recall, or a proprietary business KPI. This effectively turns the training process into a bilevel optimization where the inner loop minimizes loss and the outer loop maximizes the true metric.
The Search-Training Disconnect
The mismatch is particularly acute in combinatorial optimization. When we use neural networks to guide solvers for problems like the Traveling Salesman Problem (TSP), we often train them to predict "optimal edges." However, Yifan Xia and Xianliang Yang identify a search-training disconnect in Rethinking Post-Hoc Search-Based Neural Approaches.
The neural model is optimized in isolation, but at inference time, it is plugged into a non-differentiable search algorithm (like Monte Carlo Tree Search). If the training loss doesn't account for how the search algorithm explores the space, the "priors" the model provides will be suboptimal. As Anton Milan and Seyed Hamid Rezatofighi argue in Data-Driven Approximations to NP-Hard Problems, the goal shouldn't just be to mimic a heuristic, but to learn a representation that actually facilitates finding the discrete global optimum.
Why It Works
Aligning the loss with the metric works because it reduces the "hypothesis space" the model must search. When we use a generic surrogate like cross-entropy, the model might find many solutions that satisfy the loss but fail the metric (e.g., achieving high accuracy by ignoring a rare but critical class). By tightening the mathematical relationship between the gradient signal and the evaluation criteria—whether through direct loss minimization as explored by Alex Schwing and Raquel Urtasun or through differentiable surrogates like Houdini (Moustapha Cissé)—we ensure that every step of gradient descent is a step toward the actual goal.