Sharpness-Aware Minimization Activates the Interactive Teaching's Understanding and Optimization

Mingwei Xu, Xiaofeng Cao, Ivor Tsang

Introduction

In real-world machine learning, datasets are rarely perfect; they are often riddled with "noisy" or incorrectly labeled samples. A common and highly effective strategy to handle this is Co-teaching, an interactive teaching paradigm where two identical neural networks train simultaneously. Each network identifies "clean" samples (those yielding low loss) and passes them to its peer for training.

While Co-teaching works remarkably well in practice, it has long been treated as a heuristic. The underlying optimization principles and convergence properties of interactive teaching have lacked rigorous theoretical analysis. Furthermore, standard interactive teaching can still get trapped in sharp local optima, leading to poor generalization on unseen data. This paper provides a long-awaited theoretical foundation for interactive teaching by framing it as an Expectation-Maximization (EM) algorithm and introduces SRIT (Sharpness Reduction Interactive Teaching), a method that integrates Sharpness-Aware Minimization (SAM) to significantly improve model generalization and robustness against corrupted labels.

Theoretical Framework and Methodology

The authors tackle the theoretical gaps of interactive teaching through three main contributions:

1. The Expectation-Maximization (EM) Framework The paper formally proves that interactive teaching is a specific type of Expectation-Maximization process. In this perspective, the "cleanliness" of a data point is treated as a latent variable. The "E-step" corresponds to selecting low-loss samples, while the "M-step" corresponds to updating the model parameters. This realization provides a robust mathematical framework for proving convergence, moving the field from empirical observation to theoretical certainty.

2. Loss Landscape Analysis Geometrically, the authors demonstrate that by discarding high-loss samples, interactive teaching effectively "clips the peaks" of the loss landscape. This data filtering acts as a form of optimization surgery, creating a smoother, lower-loss surface that is easier for the optimizer to navigate.

3. Sharpness Reduction Interactive Teaching (SRIT) To prevent models from settling into "sharp" minima—which are known to generalize poorly—the authors incorporate Sharpness-Aware Minimization (SAM). In the proposed SRIT method, the two networks do not merely exchange clean samples; they exchange "sharpness knowledge."

SRIT Optimization Process

One network identifies a nearby point in the parameter space where the loss is highest (the sharpest point) and guides the other network to avoid that direction. This dual-level optimization forces both models toward flatter regions of the landscape. By making SAM interactive, the models act as checks and balances for each other, finding regions that are consistently low-loss across a neighborhood.

Experiments and Key Results

The researchers evaluated SRIT on five standard image classification benchmarks (MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and SVHN), injecting various types of synthetic noise (Symmetric, Pairflip, Tridiagonal, and Instance noise) at ratios of 20% and 40%.

Performance Comparison of Co-teaching vs SRIT

The proposed SRIT method consistently outperformed standard Co-teaching and other state-of-the-art baselines across all datasets and noise types:

  • Accuracy Boosts: On the challenging CIFAR-100 dataset with 20% symmetric noise, SRIT achieved 59.66% accuracy, a nearly 10% absolute improvement over standard Co-teaching (50.21%).
  • Robustness to High Noise: At higher noise levels (40%), the performance gap often widened. For example, on MNIST with Pairflip noise, SRIT maintained a 98.14% accuracy compared to Co-teaching’s 91.54%.
  • Training Stability: Training curves indicated that SRIT is much less prone to the "overfitting dip" typically seen in noisy-label training, where a model initially learns correct patterns but eventually memorizes the noise.

Limitations and Future Directions

While SRIT offers substantial improvements, it introduces computational overhead. SAM requires two gradient computations per update (one to find the sharp point, and one to perform the actual parameter update), making training slower than standard SGD. Additionally, the method relies on hyperparameters like perturbation radius and sampling fraction, which may require tuning for new domains.

Future work suggested by the authors includes exploring ways to reduce the computational complexity of the SAM component and investigating how these interactive, sharpness-aware strategies might be applied to more complex multi-agent systems and Large Language Models (LLMs).

Summary updated

Abstract

Teaching is a potentially effective approach for understanding interactions among multiple intelligences. Previous explorations have convincingly shown that teaching presents additional opportunities for observation and demonstration within the learning model, such as data distillation and selection. However, the underlying optimization principles and convergence of interactive teaching lack theoretical analysis, and in this regard co-teaching serves as a notable prototype. In this paper, we discuss its role as a reduction of the larger loss landscape derived from Sharpness-Aware Minimization (SAM). Then, we classify it as an iterative parameter estimation process using Expectation-Maximization. The convergence of this typical interactive teaching is achieved by continuously optimizing a variational lower bound on the log marginal likelihood. This lower bound represents the expected value of the log posterior distribution of the latent variables under a scaled, factorized variational distribution. To further enhance interactive teaching's performance, we incorporate SAM's strong generalization information into interactive teaching, referred as Sharpness Reduction Interactive Teaching (SRIT). This integration can be viewed as a novel sequential optimization process. Finally, we validate the performance of our approach through multiple experiments.
BibTeX
@inproceedings{xu2024sharpness,
  author = {Mingwei Xu and Xiaofeng Cao and Ivor Tsang},
  title = {{Sharpness-Aware Minimization Activates the Interactive Teaching's Understanding and Optimization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=Prw98p1nV0},
}
Figure