Benchmarking a foundation LLM on its ability to re-label structure names in accordance with the AAPM TG-263 report

Jason Holmes, Lian Zhang, Yuzhen Ding, Hongying Feng, Zhengliang Liu, Tianming Liu, William W. Wong, Sujay A. Vora, Jonathan B. Ashman, Wei LiuarXiv 2023 · DOI · arXiv

Abstract

Standardizing anatomical structure names in radiation oncology is critical for multi-center research and data mining, yet manual adherence to the AAPM TG-263 report remains challenging. This study evaluates the efficacy of GPT-4 as an automated "translator" to map inconsistent, institution-specific structure names to TG-263 standards. Using a dataset of 600 patients across three disease sites (Prostate, Head and Neck, and Thorax), the researchers implemented a system utilizing zero-shot learning and Chain-of-Thought prompting. The model achieved high overall accuracy (96.0%–98.5%), demonstrating that large language models (LLMs) can effectively interpret complex clinical nomenclature rules without requiring 3D image data.

Introduction: The Challenge of Clinical Nomenclature

In radiation oncology, "structure sets" define the digital boundaries of tumors and organs at risk used for treatment planning. Ideally, every clinic would follow identical naming conventions (e.g., "Heart" or "Lung_L"). In practice, naming is often inconsistent, filled with abbreviations, typos, and institutional quirks like "Hrt," "Lt_Lung," or "Lung_Left."

While the American Association of Physicists in Medicine (AAPM) released the TG-263 report in 2018 to standardize these names, the report is a complex, text-heavy document that is difficult to implement manually. This inconsistency creates a massive barrier for large-scale data mining and multi-center research. Previous attempts to automate re-labeling used rigid look-up tables or machine learning models requiring binary masks (image data), but these often struggled with the sheer variety of human-entered text.

Methods and System Architecture

The researchers developed a system that integrates the GPT-4 API into a standard medical imaging workflow. They implemented a Digital Imaging and Communications in Medicine (DICOM) storage server that intercepts structure set files. Instead of simple pattern matching, the system utilizes GPT-4's reasoning capabilities through two key techniques:

  1. Zero-shot learning: Providing the model with the TG-263 guiding principles and standard nomenclature worksheets directly in the prompt.
  2. Chain-of-Thought (CoT) prompting: Instructing the LLM to explain its reasoning step-by-step before providing the final re-labeled name.

To ensure high performance, the team employed an iterative refinement process for the instructions provided to the model.

Iterative prompt tuning process for refining GPT-4 instructions.

The prompt included institutional-specific instructions to handle local quirks and the specific TG-263 rules for the relevant disease site (Prostate, Head and Neck, or Thorax).

Datasets and Experimental Setup

The study utilized data from 600 patients randomly selected from the Mayo Clinic’s Aria database (2014–2023).

  • Prompt Tuning Set: 450 patients (150 per disease site) were used to iteratively refine the instructions given to GPT-4.
  • Evaluation Set: 150 patients (50 per disease site) were used for the final benchmark.
  • Disease Sites: Prostate, Head and Neck (H&N), and Thorax.

The dataset was purely textual; only the original structure names were sent to the LLM, ensuring no protected health information (PHI) was transmitted.

Key Insights and Results

GPT-4 demonstrated a sophisticated ability to act as a rule interpreter. Unlike traditional "fuzzy matching" algorithms, the model could follow the logic of the TG-263 report, understanding, for example, that an optimization volume should have a "z" prefix or that "fem_head_l" refers to the left femur.

A significant finding was that the structure name itself contains the most critical information for re-labeling; 3D image data is often unnecessary for accurate translation. The model achieved exceptional accuracy across all tested disease sites, with the primary difficulties occurring in Optimization Target Volumes (OTVs) due to ambiguities within the TG-263 standard itself.

Summary of re-labeling accuracy across disease sites and target volumes.

Final performance metrics included:

  • Overall Accuracy: Prostate (96.0%), Head and Neck (98.5%), and Thorax (96.9%).
  • Efficiency: The cost was remarkably low, ranging from $0.15 to $0.25 per patient case.
  • Reliability: Hallucinations were virtually non-existent, likely due to the highly constrained nature of the task and the use of Chain-of-Thought reasoning.

Limitations and Future Directions

At the time of the study, GPT-4’s 8,000-token limit required splitting prompts by disease site. Furthermore, while text is powerful, some names remain ambiguous without spatial context. Future work may explore multi-modal LLMs that can integrate 3D contours with text. Additionally, the authors suggest that fine-tuning smaller, open-source LLMs on institutional data could provide similar performance locally, further enhancing privacy and reducing costs.

Summary updated

Abstract

Purpose: To introduce the concept of using large language models (LLMs) to re-label structure names in accordance with the American Association of Physicists in Medicine (AAPM) Task Group (TG)-263 standard, and to establish a benchmark for future studies to reference. Methods and Materials: The Generative Pre-trained Transformer (GPT)-4 application programming interface (API) was implemented as a Digital Imaging and Communications in Medicine (DICOM) storage server, which upon receiving a structure set DICOM file, prompts GPT-4 to re-label the structure names of both target volumes and normal tissues according to the AAPM TG-263. Three disease sites, prostate, head and neck, and thorax were selected for evaluation. For each disease site category, 150 patients were randomly selected for manually tuning the instructions prompt (in batches of 50) and 50 patients were randomly selected for evaluation. Structure names that were considered were those that were most likely to be relevant for studies utilizing structure contours for many patients. Results: The overall re-labeling accuracy of both target volumes and normal tissues for prostate, head and neck, and thorax cases was 96.0%, 98.5%, and 96.9% respectively. Re-labeling of target volumes was less accurate on average except for prostate - 100%, 93.1%, and 91.1% respectively. Conclusions: Given the accuracy of GPT-4 in re-labeling structure names of both target volumes and normal tissues as presented in this work, LLMs are poised to be the preferred method for standardizing structure names in radiation oncology, especially considering the rapid advancements in LLM capabilities that are likely to continue.
BibTeX
@misc{holmes2023benchmarking,
  author = {Jason Holmes and Lian Zhang and Yuzhen Ding and Hongying Feng and Zhengliang Liu and Tianming Liu and William W. Wong and Sujay A. Vora and Jonathan B. Ashman and Wei Liu},
  title = {{Benchmarking a foundation LLM on its ability to re-label structure names in accordance with the AAPM TG-263 report}},
  year = {2023},
  eprint = {2310.03874},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.03874},
}
Figure