Joint Asymmetric Loss for Learning with Noisy Labels

Jialiang Wang, Xianming Liu, Xiong Zhou, Gangfeng Hu, Deming Zhai, Junjun Jiang, Xiangyang Ji

Abstract

This paper introduces Joint Asymmetric Loss (JAL), a novel framework designed to enhance the training of deep neural networks on datasets containing noisy or incorrect labels. By proposing Asymmetric Mean Square Error (AMSE) as a new passive loss function, the authors bridge a critical theoretical gap, allowing asymmetric losses to be integrated into complex optimization frameworks. JAL significantly outperforms traditional symmetric loss functions by balancing robustness to label noise with the ability to fit complex data patterns.

Introduction

Deep learning models are highly sensitive to label noise, often "memorizing" incorrect labels during training, which leads to poor generalization. While symmetric loss functions (e.g., Mean Absolute Error) were developed to provide noise robustness, they frequently suffer from underfitting due to overly restrictive mathematical constraints. The Active Passive Loss (APL) framework attempted to mitigate this by combining an "active" loss (maximizing the probability of the labeled class) with a "passive" loss (minimizing the probabilities of other classes). However, until now, the superior properties of asymmetric losses had not been successfully adapted for the passive component of APL. This research introduces the JAL framework to leverage asymmetric properties for both active and passive learning components.

Methodology

The core of the JAL framework lies in two primary innovations:

  1. Asymmetric Mean Square Error (AMSE): The authors extend standard Mean Square Error into a passive asymmetric loss. A key hyperparameter, $a \ge 1$, is introduced to control the "strictness" of the loss, allowing for a tunable balance between robustness and fitting capability.
  2. The JAL Framework: AMSE is integrated into the APL framework as a passive loss. The paper proposes two main variants: JAL-CE (Normalized Cross Entropy + AMSE) and JAL-FL (Normalized Focal Loss + AMSE).

Theoretical analysis (Theorem 4.1) establishes the conditions under which AMSE satisfies the "asymmetric condition," ensuring noise tolerance even under complex, class-dependent noise scenarios.

Experimental Evaluation

The JAL framework was evaluated across diverse benchmarks, including synthetic noise (CIFAR-10/100), human-annotated noise (CIFAR-10N/100N), and large-scale real-world noise (WebVision, ImageNet, and Clothing1M).

t-SNE visualizations of learned representations. JAL variants (b, d) show significantly clearer class separation and wider margins compared to standard CE (a) and FL (c).

Key Insights

  • Improved Feature Representation: As shown in the t-SNE visualizations above, JAL creates wider and cleaner margins between classes compared to standard methods. This indicates that the model learns a more robust representation of the underlying data structure rather than simply ignoring noise.
  • Flexibility of Asymmetry: Unlike symmetric losses, the relaxed conditions of asymmetric losses allow models to fit clean data effectively, solving the persistent underfitting problem associated with previous robust methods.
  • The Role of Hyperparameter $a$: In AMSE, $a$ serves as a dial for robustness. Larger values of $a$ impose tighter constraints suitable for extreme noise (e.g., 80%), while moderate values optimize general performance.

Results

JAL consistently established new state-of-the-art (SOTA) performance across various noise levels:

Test accuracy vs. epoch under symmetric (80%) and asymmetric (40%) noise. AMSE variants consistently outperform the baseline NNCE.
  • Extreme Noise Robustness: On CIFAR-10 with 80% symmetric noise, JAL-CE achieved 65.43% accuracy, nearly 25% higher than standard NCE (41.20%).
  • Complex Noise Handling: On CIFAR-100 with 40% asymmetric noise, JAL-CE reached 56.17% accuracy, outperforming the previous best method (ANL-CE) by over 10%.
  • Real-World Application: On the Clothing1M dataset (approx. 38% noise), JAL-CE achieved 70.31% accuracy, surpassing all other tested robust loss functions.

Conclusion and Future Work

The JAL framework demonstrates that asymmetric passive losses can effectively overcome the limitations of symmetric functions in noisy label learning. While the current implementation requires some tuning of the hyperparameter $a$, future research may focus on automatically adapting these parameters during training. Furthermore, the principles of JAL could be extended beyond image classification to domains such as Natural Language Processing and object detection.

Summary updated

BibTeX
@inproceedings{wang2025joint,
  author = {Jialiang Wang and Xianming Liu and Xiong Zhou and Gangfeng Hu and Deming Zhai and Junjun Jiang and Xiangyang Ji},
  title = {{Joint Asymmetric Loss for Learning with Noisy Labels}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure