Automated Discovery of Differentiable Loss Functions

29 concepts24 papers1 proposal

Deep learning models are almost always evaluated on discrete, non-smooth, or combinatorial metrics. Whether it is Average Precision (AP) for object detection, Mean Intersection over Union (mIoU) for semantic segmentation, or downstream financial profit in predictive pipelines, the real-world performance of a model is judged by criteria that cannot be directly optimized using gradient descent.

To bypass this limitation, researchers traditionally rely on hand-crafted surrogate losses—such as Cross-Entropy or Mean Squared Error—to act as smooth proxies during training. However, as noted by Lanlan Liu et al. in A Unified Framework of Surrogate Loss by Refactoring and Interpolation, this manual engineering process is a "black art" that relies heavily on trial-and-error, expert intuition, and task-specific heuristics.

A growing body of research addresses this bottleneck by treating loss function design as an optimization problem itself. Instead of relying on static, human-designed objectives, these frameworks automatically search for, generate, or learn differentiable loss functions that are directly aligned with downstream evaluation metrics.


The Limitations of Human Intuition

Why should we automate loss function design? The core issue is that human-designed loss functions are constrained by mathematical simplicity and static assumptions. Humans naturally prefer clean, interpretable equations. However, the high-dimensional optimization landscapes of deep neural networks are rarely simple.

As argued by Lijun Wu et al. in Learning to Teach with Dynamic Loss Functions, a single fixed loss function might not be optimal across an entire training trajectory. A model early in training may require a completely different optimization gradient than a model nearing convergence. Manually engineering these time-varying, complex objectives is practically impossible to scale.

Furthermore, when translating high-level domain constraints or logical rules into deep networks, manual formulation introduces significant friction. Francesco Giannini et al. point out in Constraint-Based Visual Generation that translating first-order logic constraints into differentiable terms typically requires ad-hoc tuning that fails to generalize across different tasks. While neural architecture search has successfully automated the design of network layers, Risheng Liu et al. emphasize in Retinex-Inspired Unrolling With Cooperative Prior Architecture Search for Low-Light Image Enhancement that human bias remains heavily present in the heuristics used to select the accompanying loss formulations.


Parameterizing the Space of Objectives

To automate the discovery of loss functions, we must first define a searchable mathematical space. Current literature approaches this parameterization through three primary lenses: symbolic operator graphs, parameterized metric approximations, and neural network surrogates.

┌───────────────────────────────────────────────────────────────────┐
│                     Loss Parameterization Spaces                  │
├───────────────────────────────────────────────────────────────────┤
│ 1. Symbolic Graphs:   [Input] ──► (log) ──► (+) ──► (exp) ──► Loss│
│                                                                   │
│ 2. Parameterized:     Loss = Softmax(Features) + Margin(α, β)     │
│                                                                   │
│ 3. Neural Surrogate:  [Predictions] ──► [ Neural Network ] ──► Loss│
└───────────────────────────────────────────────────────────────────┘

Symbolic Operator Graphs

One paradigm treats a loss function as a computational graph constructed from primitive mathematical operations (such as $\log$, $\exp$, $+$, $-$, $\times$). In AutoLossGen: Automatic Loss Function Generation for Recommender Systems, Zelong Li et al. construct custom loss functions from scratch using basic operators, discovering non-intuitive mathematical formulations that natively handle the extreme data sparsity typical of recommendation datasets. Similarly, Hongyang Gu et al. demonstrate in AutoLoss-GMS: Searching Generalized Margin-based Softmax Loss Function for Person Re-identification that decomposing margin-based softmax losses into symbolic graphs allows evolutionary algorithms to discover novel feature-learning objectives that outperform standard hand-crafted alternatives like ArcFace.

Parameterized Metric Approximations

Instead of searching from scratch, this approach unifies existing domain-specific loss functions into a single parameterized mathematical space. For instance, in Loss Function Search for Face Recognition, Xiaobo Wang et al. identify that the core mechanism of discriminative face recognition losses is simply modulating the softmax probability of the ground-truth class. By parameterizing this probability modulation, they create a unified search space that encompasses and expands upon existing heuristics. For complex evaluation metrics like Average Precision, Chenxin Tao et al. show in Searching Parameterized AP Loss for Object Detection that parameterizing the differentiable approximations of ranking functions allows a search algorithm to perfectly balance the delicate trade-offs between classification and localization sub-tasks.

Neural Surrogates

A third paradigm replaces analytic equations entirely with neural networks that act as learned loss functions. In Learning Surrogates via Deep Embedding, Yash Patel et al. train deep embedding networks to approximate non-differentiable metrics like Edit Distance, providing smooth gradients directly to the task network. To ensure these surrogates remain accurate locally during optimization, Cheng-Yu Hsieh et al. introduce in A Deep Model With Local Surrogate Loss for General Cost-Sensitive Multi-Label Learning a method that iteratively learns a smooth, local approximation of the cost surface to guide gradient descent. This concept can also be applied hierarchically; for instance, Lijun Wang et al. implement this in CLIFFNet for Monocular Depth Estimation with Hierarchical Embedding Loss by replacing pixel-wise losses with a hierarchical embedding loss learned directly from data.


Finding the optimal loss function within these parameterized spaces is typically framed as a bi-level optimization problem. The inner loop optimizes the model's weights using a candidate loss function, while the outer loop optimizes the loss function's parameters or structure based on validation performance.

┌────────────────────────────────────────────────────────┐
│               Bi-Level Optimization Loop               │
├────────────────────────────────────────────────────────┤
│                                                        │
│   ┌────────────────────────────────────────────────┐   │
│   │ Outer Loop: Update Loss Function Parameters    │   │
│   └───────────────────────┬────────────────────────┘   │
│                           │ Provides Candidate Loss    │
│                           ▼                            │
│   ┌────────────────────────────────────────────────┐   │
│   │ Inner Loop: Train Model Weights via Gradients  │   │
│   └───────────────────────┬────────────────────────┘   │
│                           │ Evaluates Performance      │
│                           ▼                            │
│   ┌────────────────────────────────────────────────┐   │
│   │ Validation: Compute Target Metric (mIoU, AP)   │   │
│   └────────────────────────────────────────────────┘   │
└────────────────────────────────────────────────────────┘

Evolutionary Algorithms

Evolutionary and genetic algorithms are highly effective at navigating discrete symbolic spaces where gradient information for the loss function itself does not exist. In Improved Training Speed, Accuracy, and Data Utilization Through Loss Function Optimization, Risto Miikkulainen et al. utilize genetic algorithms to evolve mathematical loss expressions from scratch, discovering novel loss functions that offer faster training speeds and implicit regularization. This evolutionary approach is also highly effective for complex multi-model setups; Lujun Li et al. demonstrate in KD-Zero: Evolving Knowledge Distiller for Any Teacher-Student Pairs that evolutionary algorithms can discover custom feature transformations and distance metrics for knowledge distillation, bypassing the need for human trial-and-error when pairing disparate architectures.

Reinforcement Learning

When loss generation is framed as a sequential decision-making process, reinforcement learning agents can be trained to construct mathematical expressions. Hao Li et al. apply this in Auto Seg-Loss: Searching Metric Surrogates for Semantic Segmentation, using an RL controller to generate differentiable surrogate losses that maximize non-smooth target metrics like mIoU or Boundary F1-score on validation data.

To avoid the massive computational overhead of training models to convergence in an inner loop, some methods make the loss selection process itself differentiable. In Stochastic Loss Function, Qingliang Liu et al. utilize a "Decision Network" combined with the Gumbel-Softmax trick. This allows the framework to dynamically select and combine the best loss functions from a pool, optimizing both the model weights and the loss selection parameters simultaneously in a single backpropagation round. Similarly, Myeongjang Pyeon et al. leverage differentiable search in SEDONA: Search for Decoupled Neural Networks toward Greedy Block-wise Learning to automatically design local, block-wise objective functions that eliminate the sequential bottlenecks of standard end-to-end backpropagation.


Why Automated Losses Outperform Human Designs

The core reason automated loss functions consistently outperform human-engineered alternatives comes down to gradient alignment.

Human-designed surrogates focus on absolute error minimization. However, in many real-world applications, the absolute value of a prediction matters far less than the relative order or structural relationships between predictions.

Consider "Predict-Then-Optimize" (PTO) problems, where machine learning models predict parameters that are subsequently fed into downstream optimization algorithms (such as portfolio optimization or supply chain scheduling). In Automatic Loss Function Search for Predict-Then-Optimize Problems with Strong Ranking Property, Boshi Wang et al. demonstrate that standard losses like MSE fail because they treat all prediction errors equally. Their automated framework, APOC, discovers loss functions that inherently enforce a "Strong Ranking Property"—ensuring the model prioritizes learning the relative order of critical parameters, which directly translates to better downstream decision quality.

A similar phenomenon occurs in specialized business domains. Di Chen et al. show in Task-Based Learning via Task-Oriented Prediction Network with Applications in Finance that manually designing surrogate functions to mimic complex financial logic is inherently suboptimal. By automating the generation of the surrogate objective, the optimization landscape is smoothed in a way that directly aligns the model's gradients with real-world profit maximization rather than abstract error reduction.

By treating the loss function as a dynamic, learnable component of the machine learning pipeline rather than a fixed mathematical axiom, automated loss discovery eliminates human bias, bridges the gap between training and evaluation, and uncovers highly effective, non-intuitive mathematical paths to convergence.

Key Foundational Work

AM-LFS: AutoML for Loss Function Search (2019) — contributed 1 concept elements to this direction.
Constraint-Based Visual Generation (2018) — contributed 1 concept elements to this direction.
Learning to Teach with Dynamic Loss Functions (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_manual_loss_function_design_e1d1562a,
  title = {manual loss function design},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manual-loss-function-design/rdr_e1d1562aea7444ffa73eab1087cc7df1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_manual_loss_function_design_e1d1562a_v1,
  title = {manual loss function design},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manual-loss-function-design/rdr_e1d1562aea7444ffa73eab1087cc7df1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake