Tilting the playing field: Dynamical loss functions for machine learning

Miguel Ruiz-Garcia, Ge Zhang, Samuel S. Schoenholz, Andrea J. LiuICML 2021

Instead of training neural networks on a fixed loss landscape, what if we continuously shifted the ground beneath them? Recent research proposes the use of "dynamical loss functions" that cyclically shift focus between different classes during training. This "peristaltic" movement of the landscape forces neural networks out of narrow, poorly-generalizing valleys and into wider, deeper minima, significantly improving accuracy in both small and large-scale models.

The Problem with Static Landscapes

In standard supervised learning, we define a static loss function (like cross-entropy) and use gradient descent to find its minimum. However, the "loss landscape" of a neural network is notoriously jagged. Models can easily get stuck in narrow valleys that lead to poor generalization or, in underparameterized cases, fail to find a viable solution entirely.

While "curriculum learning" attempts to fix this by presenting easier examples first, it usually requires human-labeled "difficulty" scores for every data point. This paper asks a different question: Can we improve learning by continuously transforming the loss landscape itself, without needing extra labels?

The Method: Oscillating Weights

The authors introduce a time-dependent weight for each class in the loss function to dynamically alter the training environment:

  1. Oscillating Weights: They modify the standard cross-entropy loss by multiplying each class's contribution by a time-dependent factor, $\Gamma_i(t)$.
  2. The Cycle: These factors oscillate. During a specific period, the weight for one class increases (emphasizing it) while others decrease. The system then cycles through every class in the dataset.
  3. The "Tilt": This effectively "tilts" the playing field. As the weights change, valleys in the loss landscape widen, deepen, narrow, or rise.

To understand why this works, the authors monitor the Hessian (which measures the local curvature and width of the landscape) and the Neural Tangent Kernel (NTK) (which describes how the network's outputs evolve during training).

Experimental Setup

The researchers evaluated their approach on two primary setups:

  • The Spiral Dataset: A synthetic 2D dataset consisting of three intertwined classes. This served as a "toy model" to visualize how the network navigates the landscape.
The Spiral Dataset
  • CIFAR-10: A standard image recognition benchmark. They used a Myrtle5 architecture (a small but efficient CNN) and a Wide Residual Network (WideResNet) to test the method in more realistic, overparameterized regimes.

Key Insights: Peristalsis and Bifurcation

  • Peristaltic Progress: The authors liken the training process to "peristaltic movement." By constantly shifting which class is emphasized, the landscape "squeezes" the model parameters. If a model is in a valley that becomes too narrow for the current learning rate, it becomes unstable and "bounces" out.
  • Bifurcation Cascades: As the loss landscape narrows, the system undergoes "bifurcation instabilities." The training dynamics split into branches where the model bounces between the walls of a valley. This instability acts as a natural selection mechanism that pushes the model toward wider regions of the landscape.
  • The Curvature Threshold: There is a fundamental link between the learning rate ($\eta$) and the maximum eigenvalue of the Hessian ($\lambda_{max}$). Instabilities occur when $\lambda_{max}$ exceeds a threshold proportional to $1/\eta$. Dynamical loss functions exploit this by intentionally narrowing valleys to trigger these "escapes."
  • Task Switching as a Resource: Usually, switching tasks leads to "catastrophic forgetting." Here, a controlled, cyclical emphasis on different classes actually improves the final model's ability to handle all classes simultaneously.

Results and Hyperparameter Tuning

The dynamical loss function yielded several impressive results:

  • Improved Generalization: On CIFAR-10 using the Myrtle5 network, the dynamical loss function increased validation accuracy by ~6% (from ~73% to ~79%) compared to standard static cross-entropy.
  • Solving the "Unsolvable": In underparameterized models (where the network is "too small" for the task), the dynamical loss function allowed the network to find deep minima where standard training failed completely.
  • Flatter Minima: The method consistently resulted in models with lower Hessian eigenvalues at the end of training. In deep learning, "flatter" minima are widely associated with better generalization on unseen data.
Training and Validation Accuracy vs. Amplitude and Period

The method introduces two new hyperparameters: the amplitude ($A$) and the period ($T$) of the oscillations. As seen in the heatmaps above, finding the "sweet spot" for these values currently requires empirical testing to maximize validation accuracy.

Limitations and Future Work

While promising, the authors noted that their method did not show a statistically significant improvement on the WideResNet architecture. They hypothesize that the dynamical loss might interact poorly with Batch Normalization, or that the specific training schedule used was too short to realize the benefits.

Future work will likely investigate how the number of local minima and saddle points changes as the oscillation amplitude increases, potentially leading to a more rigorous theoretical framework for "designing" optimal loss landscapes.

Summary updated

BibTeX
@inproceedings{ruizgarcia2021tilting,
  author = {Miguel Ruiz-Garcia and Ge Zhang and Samuel S. Schoenholz and Andrea J. Liu},
  title = {{Tilting the playing field: Dynamical loss functions for machine learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
}
Figure