Aligning Surrogate Losses with Task Evaluation Metrics

This direction is split into other directions: surrogate loss misalignment. Its last version stays readable and citable here.

Version 1 of 1300 papers334 conceptsfrozen 2026-10-04

In machine learning, we rarely optimize for what we actually care about. We want high accuracy, but we optimize cross-entropy. We want a high F-measure, but we minimize mean squared error. We want a short traveling salesman path, but we train on classification labels for edges.

This discrepancy is known as the loss-metric mismatch. Because most evaluation metrics (like accuracy or ranking order) are discrete and non-differentiable, we use "surrogate" losses that are smooth and easy for gradient descent to navigate. However, as research across diverse fields shows, minimizing a proxy does not guarantee maximizing the goal.

The Proxy Problem

The fundamental issue is that surrogate losses are often just loose upper bounds on the true error. Hang Li, Tie-Yan Liu, and their colleagues demonstrated this in Ranking Measures and Loss Functions in Learning to Rank, showing that popular ranking losses are essentially mathematical surrogates for 1-NDCG. When these bounds are loose, the model can "improve" its loss while its actual ranking performance stagnates or even declines.

This misalignment can lead to "bad" learning curves. Marco Loog and Tom J. Viering highlight a paradoxical phenomenon in A Survey of Learning Curves with Bad Behavior: adding more data can sometimes increase generalization error. This happens because more data allows the model to fit the surrogate objective more perfectly, driving it further away from the optimal solution for the actual target metric.

Differentiable Relaxations of Discrete Metrics

One way to bridge this gap is to create "soft" versions of hard metrics. Instead of abandoning the metric during training, researchers develop differentiable approximations that allow for direct optimization.

Performance-Guided Optimization

When a metric is too complex to relax into a simple formula, we can use a second model to "predict" the performance of the first. This is the intuition behind NPPM (Network Pruning via Performance Maximization) by Shangqian Gao and Heng Huang. Rather than pruning neurons based on low loss (a poor proxy for accuracy), they train a dedicated predictor to estimate sub-network accuracy directly.

Similarly, Mahdi Milani Fard proposed Metric-Optimized Example Weights (MOEW). This framework learns to reweight training instances so that the resulting model maximizes a specific validation metric, whether that be fairness, precision at a certain recall, or a proprietary business KPI. This effectively turns the training process into a bilevel optimization where the inner loop minimizes loss and the outer loop maximizes the true metric.

The Search-Training Disconnect

The mismatch is particularly acute in combinatorial optimization. When we use neural networks to guide solvers for problems like the Traveling Salesman Problem (TSP), we often train them to predict "optimal edges." However, Yifan Xia and Xianliang Yang identify a search-training disconnect in Rethinking Post-Hoc Search-Based Neural Approaches.

The neural model is optimized in isolation, but at inference time, it is plugged into a non-differentiable search algorithm (like Monte Carlo Tree Search). If the training loss doesn't account for how the search algorithm explores the space, the "priors" the model provides will be suboptimal. As Anton Milan and Seyed Hamid Rezatofighi argue in Data-Driven Approximations to NP-Hard Problems, the goal shouldn't just be to mimic a heuristic, but to learn a representation that actually facilitates finding the discrete global optimum.

Why It Works

Aligning the loss with the metric works because it reduces the "hypothesis space" the model must search. When we use a generic surrogate like cross-entropy, the model might find many solutions that satisfy the loss but fail the metric (e.g., achieving high accuracy by ignoring a rare but critical class). By tightening the mathematical relationship between the gradient signal and the evaluation criteria—whether through direct loss minimization as explored by Alex Schwing and Raquel Urtasun or through differentiable surrogates like Houdini (Moustapha Cissé)—we ensure that every step of gradient descent is a step toward the actual goal.

Papers in this version

Showing 60 of 300 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_aligning_surrogate_losses_with_task_eval_758bbc1e,
  title = {Aligning Surrogate Losses with Task Evaluation Metrics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/aligning-surrogate-losses-with-task-evaluation-metrics/rdr_758bbc1e396a41b3bcf63d3ab695d91e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_aligning_surrogate_losses_with_task_eval_758bbc1e_v1,
  title = {Aligning Surrogate Losses with Task Evaluation Metrics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/aligning-surrogate-losses-with-task-evaluation-metrics/rdr_758bbc1e396a41b3bcf63d3ab695d91e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}