Learning Symbolic Model-Agnostic Loss Functions via Meta-Learning

Christian Raymond, Qi Chen, Bing Xue, Mengjie Zhang

Introduction

In modern deep learning, the choice of loss function is typically limited to a few handcrafted options such as Cross-Entropy or Mean Squared Error. While these functions are robust, the No Free Lunch Theorem implies that a loss function tailored to a specific task or dataset could yield superior performance. Recent efforts in "loss function learning" aim to automate this discovery, but they often struggle with a trade-off between the interpretability of symbolic expressions and the optimizability of neural networks.

The EvoMAL Framework

The researchers propose EvoMAL (Evolved Model-Agnostic Loss), a hybrid neuro-symbolic framework designed to evolve specialized loss functions from scratch. EvoMAL utilizes a two-stage search process:

  1. Symbolic Search: Genetic Programming (GP) explores a space of mathematical primitives (e.g., addition, logarithms, square roots) to define the structural "skeleton" of a loss function.
  2. Gradient-Based Optimization: To refine the numerical constants within these symbolic structures, the framework converts the GP expression tree into a differentiable "meta-loss network." This allows for the use of unrolled differentiation to fine-tune coefficients via meta-learning.
The EvoMAL evolutionary process, incorporating symbolic filters and rejection protocols to ensure computational efficiency.

To maintain efficiency, EvoMAL employs several filtering mechanisms. A Symbolic Equivalence Filter prevents the evaluation of mathematically identical formulas, while Rejection Protocols discard candidate functions that exhibit poor training dynamics early in the process.

Neuro-Symbolic Integration

A key innovation of EvoMAL is its ability to bridge the gap between discrete symbolic structures and continuous optimization. By representing a symbolic formula as a computational graph, the system can treat the loss function as a learnable component of the end-to-end training pipeline.

The integration of a learned symbolic loss function within a standard neural network architecture.

This approach ensures that the resulting loss functions remain interpretable. Unlike "black-box" parametric losses, EvoMAL produces explicit mathematical formulas that researchers can analyze to understand the underlying learning dynamics.

Key Insights and Results

Validation across tabular regression and image classification tasks (MNIST, CIFAR-10/100, SVHN) demonstrated that EvoMAL consistently outperforms standard baselines. For instance, on CIFAR-10 using a ResNet-18, EvoMAL achieved a 5.28% error rate, surpassing the 6.58% baseline of Cross-Entropy.

Beyond raw performance, the symbolic nature of EvoMAL led to significant theoretical insights:

  • Discovery of Label Smoothing: The framework independently "re-discovered" Label Smoothing Regularization (LSR) through a learned loss called Absolute Cross-Entropy (ACE).
  • SparseLSR: Analysis of evolved formulas inspired the derivation of SparseLSR, a regularization technique that provides the benefits of label smoothing with $O(1)$ complexity, significantly reducing training time for large-scale classification problems.
  • Transferability: Loss functions evolved on smaller datasets (CIFAR-10) generalized effectively to more complex tasks (CIFAR-100), highlighting the robustness of symbolic representations.

Conclusion

EvoMAL represents a significant step forward in automated machine learning by combining the exploratory power of Genetic Programming with the efficiency of gradient-based meta-learning. By producing interpretable, high-performing loss functions, it not only improves model accuracy but also provides a tool for discovering new algorithmic principles. Future work may involve co-evolving architectures and loss functions or extending the framework to unsupervised and reinforcement learning domains.

Summary updated

BibTeX
@article{raymond2023learning,
  author = {Christian Raymond and Qi Chen and Bing Xue and Mengjie Zhang},
  title = {{Learning Symbolic Model-Agnostic Loss Functions via Meta-Learning}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  url = {https://openreview.net/forum?id=Bk2Fdca7p0},
}
Figure