TeachMe: a web-based teaching system for annotating abdominal lymph nodes

Shuaihua Chen, Hao Huang, Xuyang Yang, Han Wang, Mingtian Wei, Haixian Zhang, Ziqiang Wang, Yi ZhangScientific 2022 · DOI

TL;DR

TeachMe is a web-based platform that streamlines the annotation of abdominal lymph nodes while simultaneously training junior doctors. By combining a three-level expert review workflow with an AI-assisted feedback loop, the system creates high-quality medical datasets and measurably improves clinical interpretation skills.

What Problem Does This Paper Address?

Detecting lymph nodes (LNs) in abdominal CT scans is critical for diagnosing and treating colorectal cancer (CRC) recurrence. However, manual interpretation is a "needle in a haystack" problem—it is labor-intensive, time-consuming, and prone to human error.

While AI models can help, they require massive, precisely labeled datasets that are difficult to produce. Furthermore, there is a significant gap in radiology education: junior doctors need extensive practice to reach expert-level proficiency, but senior doctors are often too "time-poor" to provide consistent, detailed feedback. Existing e-learning tools are often passive (theory-based) rather than active (practice-based).

Methods

The authors developed TeachMe, a Browser/Server architecture system (Node.js, MongoDB, Python) designed to create a virtuous cycle between data collection and education. The core of the system is its three-level annotation-review workflow:

  1. Primary Annotation: Two junior doctors independently annotate a CT study.
  2. AI Supplement: A pre-trained automated detection model analyzes the scan to identify potential lymph nodes that the human annotators might have missed.
  3. Superior Review: A senior expert reviews the merged human and AI results to confirm the "gold standard" expert knowledge.

For the educational component, the system features a feedback mechanism. When a junior doctor completes a practice case, TeachMe compares their work against the expert database and generates a report highlighting omitted nodes, incorrect locations, or misclassifications.

Datasets

The study introduced the PLNDataset (Pelvic Lymph Node Dataset):

  • Scale: 236 CRC patients' contrast-enhanced CT (CE-CT) scans.
  • Content: 6,880 individual abdominal lymph nodes.
  • Categorization: Nodes are divided into seven anatomical regions (e.g., Colorectal mesenteries, Aortic bifurcation, Obturator cranial LNs).
  • Uniqueness: Unlike standard datasets, this was "calibrated" through the three-level review process, ensuring a higher degree of ground-truth accuracy.

Key Insights and Ideas

  • AI as a "Safety Net": Instead of replacing the human, the AI acts as a second set of eyes during the data collection phase, specifically looking for "candidates omitted" by the primary annotators.
  • Closing the Loop: The system treats the expert database not just as a training set for machines, but as an automated "tutor" for humans. This allows junior doctors to practice repeatedly without requiring a senior doctor to be present for every session.
  • Standardized Sorting: The authors implemented a specific sorting rule for annotations (ascending left-to-right and descending top-to-bottom). This seemingly small UI choice allows doctors to track specific nodes across different studies and consultations more reliably.

Key Results

  • Improved AI Performance: Training a detection model on the "calibrated" expert data (reviewed by superior annotators) significantly outperformed training on raw primary annotations. Sensitivity increased from 79% to 83%, and the mean Free-response Receiver Operating Characteristic (mFROC) improved from 0.351 to 0.391.
  • Human Skill Acquisition: Junior doctors showed marked improvement over three rounds of practice. For example, "User A" improved their detection accuracy from approximately 0.84 to 0.92.
  • Efficiency: The web-based nature of the tool removed the need for doctors to be physically present at a hospital workstation (PACS), allowing for more flexible data collection.

Limitations and Future Work

  • Image Format: Currently, TeachMe does not support raw DICOM files directly. Images are displayed with fixed window widths and levels, meaning users cannot adjust brightness or contrast to better see "tricky" regions.
  • Annotation Tools: The system is currently limited to rectangular bounding boxes. The authors acknowledge that for tasks like segmentation, they need to implement free-hand or bidirectional cross-shaped tools.
  • Label Flexibility: The current label set is hard-coded for gastrointestinal surgery. Future versions aim to allow experts to formulate customized labels for other medical specialties.

Summary updated

Abstract

The detection and characterization of lymph nodes through interpreting abdominal medical images are significant for diagnosing and treating colorectal cancer recurrence. However, interpreting abdominal medical images manually is labor-intensive and time-consuming. The related radiology education has many limitations as well. In this context, we seek to build an extensive collection of abdominal medical images with ground truth labels for lymph nodes recognition research and help junior doctors to train their interpretation skills. Therefore, we develop TeachMe, which is a web-based teaching system for annotating abdominal lymph nodes. The system has a three-level annotation-review workflow to construct an expert database of abdominal lymph nodes and a feedback mechanism helping junior doctors to learn the tricks of interpreting abdominal medical images. TeachMe's functionalities make itself stand out against other platforms. To validate these functionalities, we invite a medical team from Gastrointestinal Surgery Center, West China Hospital, to participate in the data collection workflow and experience the feedback mechanism. With the help of TeachMe, an expert dataset of abdominal lymph nodes has been created and an automated detection model for abdominal lymph nodes with incredible performances has been proposed. Moreover, through three rounds of practicing via TeachMe, our junior doctors' interpretation skills have been improved.
BibTeX
@inproceedings{chen2022teachme,
  author = {Shuaihua Chen and Hao Huang and Xuyang Yang and Han Wang and Mingtian Wei and Haixian Zhang and Ziqiang Wang and Yi Zhang},
  title = {{TeachMe: a web-based teaching system for annotating abdominal lymph nodes}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41598-022-08958-8},
}
Figure