Abstract
The Stochastic Loss Function (SLF) framework introduces a dynamic approach to model training by replacing fixed loss functions with a Decision Network. This auxiliary network selects and combines optimal loss functions from a candidate pool in real-time. Utilizing the Gumbel-Softmax trick, SLF enables joint optimization of model weights and loss selection via differentiable backpropagation. Empirical results across classification, clustering, and detection tasks demonstrate that SLF consistently outperforms static loss functions, providing a more adaptive "teaching signal" throughout the learning process.
Introduction
In standard deep learning, practitioners typically select a fixed loss function, such as Cross-Entropy or Mean Squared Error, prior to training. However, a static loss may not provide the most instructive gradients at every stage of the learning process. While adaptive or learned loss functions have been proposed, they often suffer from "violent decoupling," where network weights and loss parameters are optimized separately or iteratively. This inefficiency can prevent models from reaching a global optimum. The Stochastic Loss Function (SLF) addresses this by making the choice of loss function a differentiable part of the architecture, allowing the model to dynamically determine which gradients to follow.
Methodology
The SLF framework consists of two primary components: a Main Network and a Decision Network. The Main Network performs the target task (e.g., image classification), while the Decision Network—a small auxiliary module—analyzes the main network's output to select the most appropriate loss functions from a predefined candidate pool (e.g., Hinge, Cross-Entropy, L1).
Joint Optimization via Gumbel-Softmax
The core challenge in selecting a discrete loss function is its non-differentiable nature. SLF overcomes this using the Gumbel-Softmax trick, which allows the Decision Network to sample from the loss pool while maintaining a differentiable path for backpropagation. By sampling $K$ times and averaging the results, SLF creates a sparse, weighted combination of loss functions. This mechanism filters out unhelpful or "noisy" gradients and enables the simultaneous update of main network weights ($w$) and decision network parameters ($v$).
Experimental Results
The authors evaluated SLF across a diverse set of benchmarks, including image classification (MNIST, CIFAR), clustering (ImageNet-10), regression (DPP4), neural machine translation (IWSLT-14), and object detection (PASCAL VOC).
Performance in Classification and Clustering
SLF consistently outperformed fixed loss functions across all tested architectures. As shown in the table below, SLF achieved superior accuracy on CIFAR-10 and CIFAR-100 compared to standard Cross-Entropy (C-E) and other adaptive methods.
In clustering tasks, the improvements were even more pronounced. On ImageNet-10, SLF improved the accuracy of the Deep Adaptive Clustering (DAC) baseline from 51.22% to 73.45%.
Insights into Training Dynamics
The experiments suggest that SLF acts as a "teaching strategy," adjusting the gradient signal based on the model's current state. The Gumbel-Softmax approach promotes sparsity, effectively "shutting off" unhelpful loss functions at specific training stages. This leads to more stable training and allows the model to discover optimal transitions between different loss regimes without manual scheduling.
Conclusion
The Stochastic Loss Function framework demonstrates that treating the loss function as a learnable, dynamic component significantly enhances model performance across a wide array of tasks. While the current implementation requires a predefined candidate pool and introduces the number of samples ($K$) as a hyperparameter, it represents a major step toward fully autonomous learning systems. Future work may focus on automatically generating the candidate loss functions themselves, further reducing the need for human intervention in loss design.
Summary updated