Optimizing Models for Arbitrary Performance Metrics

24 concepts13 papers1 proposal

Machine learning models are typically trained using smooth, mathematically convenient loss functions like cross-entropy or mean squared error. These functions are popular because they are easy to differentiate, making gradient descent straightforward. However, real-world applications evaluate success using highly complex, non-smooth, and non-decomposable metrics. In multi-label classification, practitioners care about Macro-F1, Jaccard distance, or Hamming loss. In information retrieval, they care about Mean Average Precision (MAP) or Normalized Discounted Cumulative Gain (NDCG).

Traditionally, optimizing for a new metric meant inventing a new algorithm or manually deriving a complex, metric-specific loss approximation. The core idea behind metric-agnostic optimization is to break this bottleneck by decoupling the evaluation metric from the underlying learning algorithm. By treating the performance metric as a modular component—or a "black box"—a single optimization framework can adapt to virtually any arbitrary evaluation criteria.

The Core Intuition: Decoupling Metric from Solver

Why does metric-agnostic optimization work? Instead of hard-coding a specific loss function into the model's parameters, these frameworks establish an abstraction layer between the model's predictions and how those predictions are scored.

As Claudio Gentile and Francesco Orabona show in On Multilabel Classification and Ranking with Partial Feedback, separating the specific loss function from the underlying generative model allows a system to remain flexible across different performance metrics without requiring fundamental changes to its representational strategy.

This principle applies across diverse domains:

To achieve this decoupling in practice, researchers rely on three main architectural mechanisms: cost-sensitive reductions, structural separation oracles, and localized surrogate learning.

Mechanism 1: Cost-Sensitive Reductions

One of the most direct ways to optimize for an arbitrary metric is to translate that metric into a set of misclassification penalties, reducing the problem to cost-sensitive learning.

In Condensed Filter Tree for Cost-Sensitive Multi-Label Classification, Chun-Liang Li and Hsuan-Tien Lin treat the evaluation metric as a black box during training. Instead of manually deriving inference rules for complex composite scores, their framework calculates the costs of different label combinations based on the target metric and uses a tree structure to focus training on the most critical prediction paths.

[Arbitrary Metric (F1, Accuracy, etc.)]
                 │
                 ▼ (Cost Transformation)
[Misclassification Penalties / Cost Matrix]
                 │
                 ▼
[Standard Cost-Sensitive Learning Solver]

Building on this reduction strategy, Yuping Wu and Hsuan-Tien Lin developed a framework in Progressive random k-labelsets for cost-sensitive multi-label classification that utilizes a cost transformation mechanism to map arbitrary evaluation metrics into specific penalties within label subsets. This allows the underlying classifier architecture to remain completely agnostic to the metric being optimized.

To bridge the gap between cost-sensitivity and representation learning, Kuan-Hao Huang and Hsuan-Tien Lin proposed CLEMS in Cost-sensitive label embedding for multi-label classification. CLEMS treats evaluation metrics as black-box dissimilarity measures and projects labels into a low-dimensional geometric space where the Euclidean distances between points directly correspond to the costs imposed by the target metric.

Mechanism 2: Structural Optimization and Separation Oracles

When predictions involve complex dependencies—such as rankings or multi-label sets—the problem can be framed as structured output learning. Here, the metric-agnostic property is achieved by isolating the metric within a component called a separation oracle.

In Metric Learning to Rank, Brian McFee and Gert R. G. Lanckriet showed that a structural learning framework (like a Structural SVM) can optimize for entirely different ranking metrics (such as AUC, MAP, or NDCG) simply by substituting the separation oracle. The core optimization solver remains identical; the oracle's only job is to inspect the current model and find the most violated ranking constraints with respect to the chosen metric.

This modularity is similarly exploited by James Petterson and Tibério S. Caetano in Reverse Multi-Label Learning. By inverting the standard multi-label paradigm—predicting the set of instances that possess a given label, rather than the set of labels for a given instance—they treat instance sets as structured outputs. This mathematical shift allows target metrics like Macro-F1 or Hamming loss to be swapped directly into the objective function without altering the underlying structured learning machinery.

Mechanism 3: Search Spaces and Local Surrogates

For deep learning models or non-linear architectures where exact structural inference is computationally prohibitive, researchers instead frame prediction as a search problem or dynamically approximate the metric's surface.

In HC-Search for Multi-Label Prediction: An Empirical Study, Janardhan Rao Doppa et al. frame multi-label prediction as a state-space search. Instead of deriving complex probabilistic inference rules for each metric, they use the target evaluation metric directly as an oracle during training to guide the search components, allowing the model to automatically adapt when switching between F1-score and accuracy.

[Current Model Predictions] ──► [Local Loss Surface] ──► [Smooth Surrogate Approximation] ──► [Gradient Step]
                                         ▲
                                         │ (Sample & Score)
                                [Black-Box Metric]

When exact gradients are needed for deep neural networks, Cheng-Yu Hsieh et al. introduced the Locally-Learned Surrogate Loss (LLSL) framework in A Deep Model With Local Surrogate Loss for General Cost-Sensitive Multi-Label Learning. Because arbitrary metrics are often non-smooth and step-like, LLSL iteratively samples the black-box metric around the current prediction to learn a smooth, local surrogate approximation of the cost surface, successfully guiding gradient descent.

Other deep architectures embed this metric-awareness directly into their internal training logic:

Theoretical Foundations: Consistent Global Surrogates

While empirical frameworks focus on algorithmic flexibility, recent theoretical work explores whether a single mathematical structure can inherently guarantee optimization across multiple distinct metrics simultaneously.

In Multi-Label Learning with Stronger Consistency Guarantees, Anqi Mao et al. designed a generalized multi-label surrogate framework employing comp-sum and constrained losses. They proved that a single, unified surrogate framework can provide theoretical consistency guarantees for a broad family of distinct evaluation metrics—including Hamming loss, F-beta, and Jaccard distance. This demonstrates that metric-agnostic optimization is not just an algorithmic convenience, but a theoretically sound paradigm capable of unifying diverse performance criteria under a single mathematical objective.

Key Foundational Work

Reverse Multi-Label Learning (2010) — contributed 2 concept elements to this direction.
Metric Learning to Rank (2010) — contributed 2 concept elements to this direction.
On Multilabel Classification and Ranking with Partial Feedback (2012) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_metric_agnostic_optimization_98d945e1,
  title = {metric-agnostic optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/metric-agnostic-optimization/rdr_98d945e1cb0e458b998d4f1d4e4600ef},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_metric_agnostic_optimization_98d945e1_v1,
  title = {metric-agnostic optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/metric-agnostic-optimization/rdr_98d945e1cb0e458b998d4f1d4e4600ef/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake