Joint Class-Affinity Loss Correction for Robust Medical Image Segmentation with Noisy Labels

Xiaoqing Guo, Yixuan Yuan

Introduction

Deep learning has revolutionized medical image segmentation, yet its success remains heavily dependent on the availability of large-scale, pixel-perfect annotations. In the medical domain, obtaining such high-quality labels is a significant bottleneck, as it requires extensive time and expertise from clinical specialists. Consequently, researchers often resort to "noisy" labels—annotations that are imprecise, misaligned, or generated through automated but imperfect processes.

Standard approaches to learning with noisy labels (LNL) typically focus on correcting individual pixel-wise predictions. However, these methods often overlook the inherent spatial dependencies and structural context within medical images. This paper introduces Joint Class-Affinity Segmentation (JCAS), a novel framework that leverages the relationship between neighboring pixels to enhance robustness against label noise.

The JCAS Framework

The core philosophy of JCAS is that while individual pixel labels may be incorrect, the affinity—the relationship indicating whether two pixels belong to the same category—is often more resilient to noise. The framework consists of two primary components: Differentiated Affinity Reasoning (DAR) and Class-Affinity Loss Correction (CALC).

Differentiated Affinity Reasoning (DAR)

Traditional segmentation models treat pixels as independent entities during the loss calculation. The DAR module shifts this focus by generating an affinity map that predicts the relationships between pairs of pixels. By incorporating this contextual information, the model can refine its initial pixel-wise predictions. Effectively, the DAR module uses the "wisdom of the neighborhood" to filter out irrelevant noise and strengthen the signal of correct structural patterns, preventing the "blurring" effect common in simpler affinity-based models.

Class-Affinity Loss Correction (CALC)

To formally address the noise present in the training data, the authors utilize Noise Transition Matrices (NTMs). These matrices mathematically describe the probability of a clean label being flipped into a noisy one. The breakthrough of the CALC module lies in Theorem 1, a theoretical derivation provided by the authors that links pixel-level noise to affinity-level noise. This bridge allows the model to simultaneously correct the loss functions for both pixel-wise classification and pair-wise affinity prediction, ensuring consistency across both tasks even when the ground truth is corrupted.

Experimental Validation

The JCAS framework was rigorously tested using the Endovis18 surgical instrument dataset. To simulate various real-world challenges, the authors evaluated the model against four distinct types of noise:

  1. Ellipse Noise: Simulates "weak" annotations where objects are roughly outlined with ellipses rather than precise contours.
  2. Symmetric & Asymmetric Noise: Standard synthetic patterns where labels are randomly or systematically flipped to incorrect classes.
  3. Source-Free Domain Adaptation (SFDA): Uses real-world noisy pseudo-labels generated when a model trained on one hospital's data (Endovis17) is applied to a different environment (Endovis18).

Key Results

The experimental results demonstrate that JCAS consistently outperforms state-of-the-art methods. Notably, under symmetric and asymmetric noise conditions, the framework achieved performance within ~1% of the "Upper Bound" (the performance of a model trained on perfectly clean data).

Ablation studies further highlighted the importance of the affinity-based approach: the inclusion of the DAR module alone improved the Jaccard (Jac) score by over 9% compared to a standard backbone. Furthermore, while competing methods often exhibited "overfitting" to noise—where performance peaks early and then degrades—JCAS demonstrated a stable, upward learning curve, converging to a high-performance steady state.

Discussion and Insights

The most significant insight of this research is the inherent robustness of pair-wise affinity. In many cases, even if two pixels are both mislabeled as the same incorrect class, their affinity label (indicating they belong to the same category) remains correct. This mathematical property significantly reduces the effective noise rate; for instance, a 44% noise rate at the pixel level can translate to only a 23% noise rate at the affinity level.

Limitations and Future Directions

Despite its success, the authors identify several areas for future work:

  • Computational Overhead: While the memory increase is currently manageable (~4MB), calculating pair-wise affinities for extremely high-resolution 3D volumes could increase computational demands.
  • NTM Estimation: The effectiveness of CALC depends on the accurate estimation of the Noise Transition Matrix, which may be more challenging in complex scenarios where noise is not class-dependent.
  • Generalization: While proven effective for surgical instruments, further research is needed to determine how well this affinity-first approach applies to organic, diffuse structures such as tumors or lesions in MRI and CT scans.

Conclusion

JCAS represents a significant step forward in robust medical image segmentation. By mathematically bridging pixel-wise and affinity-level noise, the framework allows models to learn effectively from imperfect data, bringing the performance of noisy-label training remarkably close to that of expert-level gold standards.

Summary updated

Abstract

Noisy labels collected with limited annotation cost prevent medical image segmentation algorithms from learning precise semantic correlations. Previous segmentation arts of learning with noisy labels merely perform a pixel-wise manner to preserve semantics, such as pixel-wise label correction, but neglect the pair-wise manner. In fact, we observe that the pair-wise manner capturing affinity relations between pixels can greatly reduce the label noise rate. Motivated by this observation, we present a novel perspective for noisy mitigation by incorporating both pixel-wise and pair-wise manners, where supervisions are derived from noisy class and affinity labels, respectively. Unifying the pixel-wise and pair-wise manners, we propose a robust Joint Class-Affinity Segmentation (JCAS) framework to combat label noise issues in medical image segmentation. Considering the affinity in pair-wise manner incorporates contextual dependencies, a differentiated affinity reasoning (DAR) module is devised to rectify the pixel-wise segmentation prediction by reasoning about intra-class and inter-class affinity relations. To further enhance the noise resistance, a class-affinity loss correction (CALC) strategy is designed to correct supervision signals via the modeled noise label distributions in class and affinity labels. Meanwhile, CALC strategy interacts the pixel-wise and pair-wise manners through the theoretically derived consistency regularization. Extensive experiments under both synthetic and real-world noisy labels corroborate the efficacy of the proposed JCAS framework with a minimum gap towards the upper bound performance. The source code is available at \url{https://github.com/CityU-AIM-Group/JCAS}.
BibTeX
@inproceedings{guo2022joint,
  author = {Xiaoqing Guo and Yixuan Yuan},
  title = {{Joint Class-Affinity Loss Correction for Robust Medical Image Segmentation with Noisy Labels}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  url = {https://openreview.net/forum?id=LW3DCntt8Mo},
}
Figure