Human gaze-based dual teacher guidance learning for semi-supervised medical image segmentation

Rongjun Ge, Chong Wang, Yuxin Liu, Chunqiang Lu, Cong Xia, Yehui Jiang, Fangyi Xu, Yinsu Zhu, Daoqiang Zhang, Chengyu Liu, Yang Chen, Shuo Li, Yuting HeNeural Networks 2025 · DOI

TL;DR

This paper introduces HG-DTGL, a novel semi-supervised learning framework for medical image segmentation that leverages human gaze data to guide the learning process. By incorporating gaze as a "teacher," the model learns to focus on clinically relevant regions, significantly improving segmentation accuracy, especially when labeled data is scarce.

What Problem Does This Paper Address?

Accurate medical image segmentation is crucial for diagnosis and treatment planning, but it's severely hampered by the scarcity of labeled data. Manual annotation is time-consuming and expensive, making it a bottleneck for training robust segmentation models. While semi-supervised learning methods aim to alleviate this by using unlabeled data, they often struggle to effectively capture the subtle details and critical regions that human experts (like radiologists) intuitively focus on. Existing methods lack a mechanism to explicitly guide the model's attention towards these important areas, leading to suboptimal performance, particularly for complex lesions or early-stage diseases.

Methods

The core of this work is the Human Gaze-based Dual Teacher Guidance Learning (HG-DTGL) model. It builds upon the mean-teacher framework, which uses a student and a teacher network (the teacher being an exponential moving average of the student's weights) to generate pseudo-labels for unlabeled data. The novelty lies in introducing human gaze data as an additional "teacher."

Key components include:

  • GazeMix: A new data augmentation technique that uses radiologist gaze heatmaps to guide the "copy-paste" of foreground regions onto background images. This creates more realistic and informative mixed images, expanding the dataset's diversity and scale while preserving crucial contextual information.
  • Multi-scale Gaze Perception (MGP) module: This module extracts multi-scale features from the network's perception, allowing it to capture both fine-grained details and holistic structures. It helps align the network's attention with human gaze.
  • Gaze Loss ($L_{gaze}$): This loss function directly compares the network's attention map with the human gaze heatmap, encouraging the model to focus on the regions that radiologists actually look at. This addresses the "where should be gazed" aspect.
  • Segmentation Loss ($L_{seg}$): This standard loss function, combined with pseudo-labels from the teacher network and ground truth from labeled data, guides the model to learn "what content is gazed on."

The dual-teacher approach (gaze teacher and network teacher) allows the model to learn both where to look and what to segment, leading to more accurate and robust segmentation.

Datasets

The authors evaluated their method on several public datasets:

  • ACDC (Automated Cardiac Diagnosis Challenge): MRI data for segmenting cardiac structures.
  • CAMUS (Cardiac Acquisition for Multi-view Segmentation): Ultrasound images for cardiac segmentation.
  • Synapse: CT scans for abdominal organ segmentation.
  • SCR (Segmentation in Chest Radiographs): X-ray images for chest anatomical structure segmentation.

The gaze data was collected from four experienced radiologists using a custom-built system.

Key Insights and Ideas

  • Gaze as a "Teacher": The most significant insight is that human gaze data can be effectively leveraged as an explicit supervisory signal to guide deep learning models, particularly in tasks where human attention is critical, like medical image interpretation.
  • Beyond Random Augmentation: GazeMix demonstrates that guided data augmentation, informed by human perception, is more effective than random methods like MixUp or CutMix for improving segmentation robustness and accuracy.
  • Bridging the Perception Gap: The MGP module and Gaze Loss work synergistically to reduce the discrepancy between how a network "sees" an image and how a human expert perceives it, leading to more clinically relevant segmentations.
  • Robustness to Noise: The study shows that even with some noise in the gaze data (e.g., transient eye movements), the proposed method remains robust and achieves acceptable performance, highlighting the value of filtering and the overall strength of the approach.

Key Results

The HG-DTGL model achieved state-of-the-art performance across all tested datasets (ACDC, CAMUS, Synapse, SCR) at very low labeling ratios (5% and 10%). For instance, on the ACDC dataset with 5% labeling, it achieved a Dice score of 89.37% and a 95HD of 1.45%, significantly outperforming other semi-supervised methods. Qualitative results (visualizations) showed that HG-DTGL produced segmentations that were more consistent with the ground truth in terms of detail, boundaries, and overall structure compared to other methods. Ablation studies confirmed the effectiveness of each proposed component (GazeMix, MGP, Gaze Loss).

Limitations and Future Work

The authors acknowledge that the complex visual environment in medical imaging can pose challenges for gaze-based models. Specifically, the original gaze data can contain transient eye movements (saccades) that are irrelevant to semantic segmentation, potentially introducing noise. While their method shows robustness, they suggest future work could focus on better addressing this noise issue. They also propose further enhancing the performance and applicability of their method by exploring more advanced gaze noise filtering techniques.

Summary updated

BibTeX
@article{ge2025human,
  author = {Rongjun Ge and Chong Wang and Yuxin Liu and Chunqiang Lu and Cong Xia and Yehui Jiang and Fangyi Xu and Yinsu Zhu and Daoqiang Zhang and Chengyu Liu and Yang Chen and Shuo Li and Yuting He},
  title = {{Human gaze-based dual teacher guidance learning for semi-supervised medical image segmentation}},
  journal = {Neural Networks},
  year = {2025},
  doi = {10.1016/j.neunet.2025.107865},
}
Figure