Improving Neural Models for Radiology Report Retrieval with Lexicon-based Automated Annotation

Luyao Shi, Tanveer F. Syeda-Mahmood, Tyler BaldwinNAACL 2022 · OpenReview

Researchers from IBM Research, in a study presented at NAACL 2022, have introduced a novel framework to bridge the gap between general-purpose language models and the specialized task of radiology report retrieval. By leveraging lexicon-based automated annotation, the team developed a method to train neural models that significantly outperform traditional keyword-based and standard clinical language models.

The Challenge: Clinical Context and Data Scarcity

Clinicians and researchers frequently need to identify specific patient cohorts by searching through unstructured radiology reports. However, traditional keyword-based search methods like BM25 often fail to capture clinical context, synonyms, or the critical distinction between the presence and absence of a condition (e.g., "pneumothorax" vs. "no pneumothorax").

While modern Transformer-based models like BERT offer better semantic understanding, they face two primary hurdles in the medical domain:

  1. The Negation Problem: Standard models struggle to distinguish between positive findings and ruled-out conditions.
  2. The Data Bottleneck: Training models to recognize thousands of clinical findings requires massive amounts of expert-labeled data, which is prohibitively expensive to produce.

Methodology: Weakly Supervised Learning with Triplet Loss

The authors propose a hybrid approach that combines lexicon-driven concept detection with supervised query-match learning. Instead of relying on human labels, they utilized a specialized Chest X-Ray (CXR) lexicon and string-matching algorithms to identify findings in sentences, paired with a negation detection tool.

These findings were converted into "queries" to train a Sentence-BERT (SBERT) model using a triplet loss function. As shown in the architecture below, this function forces the model to minimize the distance in embedding space between a query ($e_q$) and a matching sentence ($e_m$), while maximizing the distance to a non-matching sentence ($e_u$).

Figure 1: SBERT architecture utilizing triplet loss for query-sentence matching.

To enhance training, the researchers employed hard-sampling (selecting the most confusing incorrect sentences) and mega-batching (increasing the pool of potential sentences to find harder negative examples).

Experimental Results

The study evaluated the model using two major public collections: the Indiana University (IND) dataset for training and the NIH dataset for testing generalization. The results, summarized in the table below, demonstrate that the proposed SBERT model significantly outperforms both traditional baselines and specialized models like BioClinicalBERT.

Figure 4: Performance comparison (mAP and mR) across IND and NIH datasets for various retrieval models.

Key findings include:

  • Superior Accuracy: On the IND dataset, the SBERT model achieved a Mean Average Precision (mAP) of 0.46, compared to 0.17 for BioClinicalBERT and 0.38 for BM25.
  • Negation Awareness: By explicitly training on "no [finding]" queries, the model learned to create a clear separation in the embedding space between positive and negative instances, a task where standard BERT models typically fail.
  • Zero-Shot Generalization: Despite being trained primarily on X-ray reports, the model performed remarkably well on the NIH dataset and even on different modalities like CT scans (achieving an mAP of 0.59–0.66 on human-annotated gold standards). This suggests the model learns underlying clinical concepts rather than just memorizing phrasing.

Conclusion and Future Work

This research demonstrates that lexicon-based weak supervision is a powerful tool for domain-specific NLP tasks where labeled data is scarce. Future work will aim to expand this approach to handle complex boolean queries (e.g., "Finding A and B, but not C") and move from sentence-level to full report-level retrieval. The authors suggest this methodology could be adapted to other specialized fields, such as finance or law, provided a robust domain lexicon is available.

Summary updated

BibTeX
@inproceedings{shi2022improving,
  author = {Luyao Shi and Tanveer F. Syeda-Mahmood and Tyler Baldwin},
  title = {{Improving Neural Models for Radiology Report Retrieval with Lexicon-based Automated Annotation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=A0p4KtvkFvZn},
}

Loading related content...

Figure